112 Commits

Author SHA1 Message Date
jpark37
7772a56a97 win-capture: Fix 32-bit Vulkan capture
Vulkan loader needs pure name, which needs manual fix-up on 32-bit.
2020-07-09 22:10:34 -07:00
jpark37
cdd912b162 win-capture: Reset command pool rather than buffer
Reorganize data to avoid best practices layer warning. Sort of a false
positive in our case because we only have one buffer per pool now, but
implicit layers should be clean citizens.
2020-06-21 06:47:22 -07:00
Jim
9c107738f6
Merge pull request #3015 from jpark37/vulkan-external-fixes
win-capture: Improve Vulkan synchronization
2020-06-21 05:02:20 -07:00
jpark37
a7813c9f87 win-capture: Remove dead VkResult values 2020-06-07 15:22:13 -07:00
jpark37
83dd9c8540 win-capture: Improve Vulkan synchronization
Use general layout and 0 access masks for external synchronization as
specified in the spec.

Also set pipeline stages for maximum synchronization just in case
because it doesn't seem like the pipeline stages are specified.

"Vulkan-incompatible APIs will require the image to be in the GENERAL
layout whenever they are accessing them."

"Whilst it is not invalid to provide destination or source access masks
for memory barriers used for release or acquire operations,
respectively, they have no practical effect. Access after a release
operation has undefined results, and so visibility for those accesses
has no practical effect. Similarly, write access before an acquire
operation will produce undefined results for future access, so
availability of those writes has no practical use. In an earlier version
of the specification, these were required to match on both sides - but
this was subsequently relaxed. These masks should be set to 0."
2020-05-31 22:31:39 -07:00
Jim
ad919b47ea
Merge pull request #2921 from jpark37/vulkan-format-fail
win-capture: Fail on unsupported Vulkan formats
2020-05-21 17:29:57 -07:00
Jim
cb98f456eb
Merge pull request #2595 from jpark37/vulkan-cleanup
Vulkan hook COM clean up, and extension check
2020-05-20 18:20:05 -07:00
Jim
23ee7aebc6
Merge pull request #2589 from jpark37/vulkan-alloc
win-capture: Use VkAllocationCallbacks
2020-05-20 18:17:20 -07:00
jp9000
7993179466 cmake: Add cmake folders 2020-05-13 06:52:37 -07:00
jpark37
30f6870b23 win-capture: Fail on unsupported Vulkan formats
Don't allow unsupported Vulkan formats to fall back to B8G8R8A8.
Probably better to fail completely than do an illegal copy.

Also remove bad conversion for VK_FORMAT_A2R10G10B10_UNORM_PACK32.
Red and blue channels were reversed, and there's no DXGI equivalent.

Addresses #2796. We can do more later if justified.
2020-05-12 12:06:03 -07:00
jpark37
8987fc6c81 win-capture: Verify VK_KHR_external_memory_win32 support
Good practice to verify necessary extensions exist. We'll probably need
keyed mutex extension in the future to support Intel.
2020-03-29 10:08:03 -07:00
jpark37
21dc1f573b win-capture: Cleaner COM usage
Remove hard-coded GUIDs, call correct functions, and clean up
unnecessary casts.
2020-03-29 10:00:36 -07:00
jpark37
348fcd5b00 win-capture: Fix crash if GPU can't Vulkan capture 2020-03-28 09:01:26 -07:00
jpark37
e1739707d0 win-capture: Use VkAllocationCallbacks 2020-03-27 23:59:37 -07:00
jp9000
59351b27ae win-capture/graphics-hook: Separate some debugging stuff
Reduces debugging spam when debugging Vulkan capture is needed in the
hook.
2020-03-13 06:57:16 -07:00
jpark37
9f15514c1a win-capture: Vulkan surface refactor
Make sure HWND tracking is cleaned up when Vulkan surfaces are
destroyed. Also use unbounded linked list to fix games that leak
surfaces on Alt+Tab like Doom.

Also replace CRITICAL_SECTION with SRWLOCK, both for claimed speed
benefit, and to remove initialization code.
2020-03-13 04:36:19 -07:00
jp9000
98c1ac2f80 win-capture/graphics-hook: Add a bit of logging
Just makes sure to be explicit when a device isn't valid.
2020-03-11 02:00:42 -07:00
jpark37
421f97e68d win-capture: Handle vkCreateSwapchainKHR errors
Fixes behavior for an application reported by AMD.
2020-03-10 21:31:40 -07:00
jp9000
428a19edd6 win-capture/graphics-hook: Make vulkan layer all capital 2020-03-09 10:44:04 -07:00
jp9000
229f8d37cb win-capture/graphics-hook: Fix reacquire bug
Sometimes the width/height would initially be at 0x0 when the swap chain
was alt-tabbing, causing the capture to fail full-stop when trying to
reacquire.
2020-03-09 02:50:34 -07:00
jp9000
30e3ef3b49 win-capture/graphics-hook: Log vulkan capture dimensions 2020-03-09 02:34:33 -07:00
jpark37
4f4fc3bfe6 win-capture: Restore Vulkan 1.1 version hack 2020-03-07 20:07:56 -08:00
jpark37
3439be57fe win-capture: Support VK_FORMAT_A8B8G8R8_UNORM_PACK32 2020-03-07 09:58:07 -08:00
jp9000
87f5bd6e9f win-capture/graphics-hook: Fix wrong HWND
Game capture assumes the top-level ancestor HWND, not a child HWND.
This fixes bugs such as the VLC window not capturing anymore due to the
fact that the capture HWND of VLC is a child window of the main VLC
window and not the main VLC window itself.
2020-03-07 07:35:34 -08:00
jpark37
9da6a4d6f6 win-capture: Vulkan capture clean-up
Remove unnecessary DXGI swap chain creation, and use VK_NULL_HANDLE.
2020-03-06 17:04:31 -08:00
jpark37
fc4f3c0934 win-capture: Clean up various VC++ warnings 2020-03-06 17:02:38 -08:00
craftwar
3821a0344e win-capture: Fix format string warning 2020-03-06 19:52:02 +08:00
jpark37
c706a7bed5 win-capture: Handle NULL VkApplicationInfo 2020-03-05 21:20:00 -08:00
jp9000
7e78c17ace win-capture: Check hook version before capture init
Checks the hook version to ensure compatibility with hook DLL.  It's
unlikely it'll ever be necessary to increment the hook version, but this
is just a precautionary thing that allows a hook DLL to make sure it's
rejected by an older OBS version if needed.  Again however, very
unlikely that the major version will ever be incremented.
2020-02-29 04:53:29 -08:00
Matthieu Cunzi
74acbd7ed0 win-capture: Add Vulkan capture
Co-authored-by: jp9000 <obs.jim@gmail.com>
Co-authored-by: jpark37 <jpark37@users.noreply.github.com>
2020-02-29 04:53:28 -08:00
jp9000
a20f1168a7 win-capture/graphics-hook: Don't allow multiple hooks
Prevents multiple separate hook DLLs from being loaded in to a target at
once.  This is done just in the off-chance that someone might add
another hook to the Vulkan layer registry.
2020-02-29 00:15:54 -08:00
jpark37
39dc64f0f8 win-capture/graphics-hook: Refactor mutex check
OpenMutex doesn't return INVALID_HANDLE_VALUE, so there's no need to
check for that.  Also only call CloseHandle if the call was successful.
2020-02-29 00:15:54 -08:00
jp9000
3e3521b9e1 win-capture/graphics-hook: Fix OBS detection
Fixes where the graphics hook would try to detect if OBS is currently
active or not.
2020-02-28 23:48:11 -08:00
jp9000
ef6a6827f4 win-capture/graphics-hook: Add flog and flog_hr
Allows automatically outputting the function name as part of the hook
logging.  This really doesn't need to be a manual process.  Makes code a
bit cleaner when used.
2020-02-28 23:48:11 -08:00
VodBox
d2e115881d win-capture: Fix hook collisions with multiple game captures
Before this change, after a game capture source would send a signal to
init or restart a graphics hook, it would respond to any and all hook
ready signals.

With multiple game capture sources in the same scene, a source could
receive the signal intended for another source, and show the wrong
texture.

This change adds the window handle to the name for shared data with the
hook, resulting in hooks for other sources being ignored.
2020-02-18 21:20:48 +13:00
tujinshu
76e6a99124 win-capture: Fix rare crash when GL program exits
**Commit message modified and clarified by Jim**

When hooking a program that has both DirectX and OpenGL contexts in use,
it is possible to cause a crash on shutdown due to capture_active()
returning true when an OpenGL context is deleted.  Normally, when
capturing an OpenGL program, this would not happen because the 'active'
variable would not be set due to OpenGL capture not being initialized,
but if DirectX is captured while an OpenGL context is available, and
OpenGL could not load these required functions, then GL can crash due to
trying to use unavailable functions.

This case is extremely rare and doesn't happen under normal
circumstances; only if a program is using both DirectX and OpenGL within
the same program simultaneously, and *only* if OpenGL could not load the
required functions.  This likely almost never happens under normal
programs, games, and hardware.  This was apparently produced by hooking
a GL Qt program that used QWebEngine, which used multiple contexts at
once.
2019-11-05 13:44:15 -08:00
jpark37
3ddef4bf7d win-capture: Remove support for feature level 9.3
Be consistent with libobs-d3d11, which now uses 10_0 as the baseline.
2019-09-08 22:43:41 -07:00
jpark37
ef39715c31 win-capture: Remove unused variable from D3D12 capture 2019-09-08 22:36:25 -07:00
craftwar
e004ce8064 graphics-hook: Fix format specifier 2019-09-02 21:12:27 +08:00
jp9000
f53df7da64 clang-format: Apply formatting
Code submissions have continually suffered from formatting
inconsistencies that constantly have to be addressed.  Using
clang-format simplifies this by making code formatting more consistent,
and allows automation of the code formatting so that maintainers can
focus more on the code itself instead of code formatting.
2019-06-23 23:49:10 -07:00
geemion
f6581952bc win-capture/graphics-hook: Check if mutex abandoned
It's possible that the mutexes used with shared memory capture to return
WAIT_ABANDONED if OBS is shut down abnormally while the mutex is locked.
2019-04-14 03:44:48 -07:00
jp9000
b3ef46d986 cmake: Make static VC runtime libraries consistent
Makes Visual C runtime libraries consistent across
Debug/MinSizeRel/Release/RelWithDebInfo, rather than just changing those
flags for RelWithDebInfo.  Also adds /Zl for statically linked
libraries.

Closes obsproject/obs-studio#1421
2018-09-12 07:49:13 -07:00
jp9000
8a16fa8341 win-capture/graphics-hook: Fix memory offset calculation
Fixes an issue where align_pos could be smaller than
sizeof(struct shmem_data), potentially overwriting memory of the header.

References jp9000/obs-studio#1202
2018-02-24 21:48:50 -08:00
Bazhenoff
b9d044267d graphics-hook: blacklist OpenGL capture for "cm_client.exe"
Closes jp9000/obs-studio#1076
2017-11-16 17:57:11 -08:00
jp9000
e5d660e944 graphics-hook: Blacklist specific game from GL capture
"Life is Feudal: Your Own" will use Direct3D to render the game, then
OpenGL to render its in-game menus, which causes a conflict with itself.
This specifically blacklists the game from capturing OpenGL to prevent
that from happening.
2017-10-02 15:29:04 -07:00
jp9000
1ca9f8872e win-capture: Add IDXGISwapChain1::Present1 hook support
Allows capturing games/programs that may be using Present1 instead of
the regular Present call for rendering.
2017-06-19 15:45:00 -07:00
jp9000
c4036b18bb win-capture/graphics-hook: Fix D3D10/D3D11 detection
When hooking DXGI-based graphics programs in the hook, the first hook
point is IDXGISwapChain::Present.  To be able to initiate a capture, a
pointer to the device context that created the swap chain is required,
which can be retrieved via IDXGISwapChain::GetDevice.  Determining
whether the device context was D3D10 or D3D11 has always been somewhat
of an issue due to D3D10 and D3D11 being nearly identical, as well as
their interoperability/interchangeability.  The GetDevice function would
first be called with the UUID of ID3D10Device, then if that failed,
the UUID of ID3D11Device.

However, with certain specific D3D11 games, GetDevice would for some
unknown reason succeed with the UUID of ID3D10Device, which would cause
capture to fail.  (Conversely, attempting to call GetDevice with the
UUID of ID3D11Device on a device that's actually ID3D10Device would
always succeed, so reversing the order of the test was not an option).
There were originally three known D3D11 games that would erroneously
succeed when querying a D3D10 device interface:  Call of Duty: Ghosts,
Just Cause 3, and theHunter: Call of the Wild.  All other known D3D11
games would work correctly.  Because it was only these three games, a
hack was originally implemented in the form of an executable exception
list that would force them to capture D3D11.

Unfortunately, Oculus games are now failing under the same circumstance
as well, so a simple hack will no longer work.  To fix this, a more
reliable method of detecting which context it is had to be discovered:
simply check the feature level using ID3D11Device::GetFeatureLevel.  If
the feature level is D3D_FEATURE_LEVEL_11_0 or D3D_FEATURE_LEVEL_11_1,
the device is definitely a D3D11 device.  Otherwise, continue the tests
as they were before.  Successfully tested with many D3D10 games, many
D3D11 games, and especially those three D3D11 games that previously were
detected as D3D10 erroneously.
2017-06-15 03:48:43 -07:00
Richard Stanway
9e95b2eb6f
Various: Don't use boolean bitfields
Using bitfields causes less optimized code generation and the memory
savings are minimal as none of the objects are instantiated enough
times to be worth it.

See https://blogs.msdn.microsoft.com/oldnewthing/20081126-00/?p=20073
2017-05-10 23:28:46 +02:00
Ryan Foster
be98cee2a0 Fix various typos across multiple modules 2017-04-25 22:39:42 -04:00
Richard Stanway
d2e9e47bb4 graphics-hook: Account for sizeof(wchar_t) in len 2017-02-25 17:18:55 +01:00