134 Commits

Author SHA1 Message Date
jpark37
2472529520 win-capture: Restore GL capture deduplication
Previous fix attempted to place swap_recurse count in gl_data struct,
where it could be unexpectedly memset to 0. Separate member from struct,
and make thread-local for additional safety.
2021-03-31 02:35:22 -07:00
jpark37
17cb0acdd7 graphics-hook: Fix build without COMPILE_D3D12_HOOK
Just need to forward declare D3D12 type.
2021-02-14 22:20:14 -08:00
jpark37
23e9d4153e win-capture: D3D12 swap chain queue usage
Attempt to schedule shared texture copies against the command queue that
the game's swap chain uses to try to reduce artifacts. The heuristics
for obtaining the queue are not perfect, so provide a toggle to use the
previous behavior.
2021-02-14 02:06:02 -08:00
jpark37
535f6b0adc win-capture: Typeless game capture textures
Modify game capture shared textures to be typeless if they could
potentially need SRGB and non-SRGB views in the future.

These capture APIs have been updated: D3D 10/11/12, Vulkan.

D3D8 capture does not use shared textures.

D3D9 and GL interop do not support typeless textures.

The new game capture DLL should be compatible with old versions of OBS.

Also removed a lot of dead code around pointless SRV/RTV support.
2021-01-11 14:18:12 -08:00
jpark37
013dd5a7a3 win-capture: Fix our own Vulkan spec violation
Pass valid instance when looking up VkCreateDevice.
2020-11-14 12:03:36 -08:00
jpark37
dad861b036 win-capture: Fix D3D leaks on swap chain release
For game capture, hook DXGI release function to release D3D objects if
the related swap chain is also being destroyed.

An added bonus is that the game capture hook will handle swap chain
recreation for applications that don't use ResizeBuffers.
2020-10-13 18:47:35 -07:00
jpark37
9ba768837b win-capture: Violate Vulkan spec for compatibility
Rivatuner Statistics Server calls vkGetInstanceProcAddr for
vkCreateDevice with a NULL instance. According to spec, this should
return NULL, but RTSS will dereference this and crash.

This change returns function pointers for every intercepted function
regardless of instance, which is enough to keep RTSS alive.
2020-10-03 22:41:11 -07:00
jpark37
7cf148e013 win-capture: Improve game capture messages
Tag debug output in the game process with "[OBS]" and mention that
graphics-hook.dll is ours.
2020-09-08 09:55:46 -07:00
jpark37
94475b8795 win-capture: Fix Vulkan crash on minimize restore
Set uninitialized data members to avoid crash coming back from minimize.
2020-09-01 20:20:59 -07:00
jpark37
7a66466efc win-capture: Add Vulkan instance creation fallback
Fall back to original arguments if our 1.1 API request doesn't work.
2020-09-01 18:59:13 -07:00
jpark37
e905b4cb6e win-capture: Vulkan variable naming consistency 2020-09-01 18:56:14 -07:00
Jim
6efd70fc01
Merge pull request #3260 from jpark37/vulkan-cts-hack-remove
win-capture: Remove Vulkan CTS workaround
2020-08-27 10:18:08 -07:00
jpark37
a713a5b433 win-capture: Robust Vulkan swap chain handling
Not all surfaces support VK_IMAGE_USAGE_TRANSFER_SRC_BIT. Only
VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT is required by spec. Add fall back.
2020-08-23 11:23:12 -07:00
jpark37
9d177d0969 win-capture: Remove Vulkan CTS workaround
CTS has been fixed, so the workaround is no longer necessary.
2020-08-07 22:58:49 -07:00
jpark37
04b9b9fea3 win-capture: Make Vulkan frame data local to queue
Prevents multiple queues in the same family from racing on frame data.

Removal of vk_family_data also fixes broken capacity calculation. Two
queue families can share the same index if one is "protected-capable"
and the other isn't.
2020-07-26 13:33:01 -07:00
jpark37
e6896d79e3 win-capture: Hide Vulkan linked list internals 2020-07-26 13:32:53 -07:00
jpark37
44fb61b1f2 win-capture: Improve Vulkan hook stability
Make fixes mostly to stabilize CTS. Tested on Windows with RTX 2080 Ti.

Use separate linked-list maps for devices and queues.

Process queues upfront in OBS_CreateDevice, and not lazily in
OBS_GetDeviceQueue.

Handle failed node allocation for OBS_CreateInstance, OBS_CreateDevice,
and OBS_CreateSwapchainKHR.

Handle VK_NULL_HANDLE for OBS_DestroySurfaceKHR and
OBS_DestroySwapchainKHR.

Add temporary code path CTS_WORKAROUND to deal with CTS sharing surfaces
between instances.

Fix return value for OBS_DestroyInstance.

Add missing _freea calls, required for _malloca.

Make sure the queue passed to OBS_QueuePresentKHR supports transfer to
safely call vkCmdCopyImage.

Update GetInstanceProcAddr/GetDeviceProcAddr to conform closer to spec.

Remove extra macros because I dislike indirection more than verbosity.
2020-07-24 09:30:05 -07:00
jpark37
afbad3b337 win-capture: Remove fixed-size Vulkan arrays
Use linked lists in place of fixed-size arrays in an attempt to fix
behavior when programs generate lots of non-resource objects.
2020-07-10 20:41:00 -07:00
Jim
5728df6cef
Merge pull request #3149 from jpark37/vulkan-32bit
win-capture Fix 32-bit Vulkan capture
2020-07-10 20:39:13 -07:00
Jim
6c37f6faea
Merge pull request #3113 from jpark37/vulkan-swap-decouple
win-capture: Decouple swap and frame indices
2020-07-10 20:15:38 -07:00
jpark37
7772a56a97 win-capture: Fix 32-bit Vulkan capture
Vulkan loader needs pure name, which needs manual fix-up on 32-bit.
2020-07-09 22:10:34 -07:00
jpark37
3ac17d11c4 win-capture: Decouple swap and frame indices
The swap image index may not be a rotating value, so manage a separate
index to avoid reusing frame data early and stalling.
2020-06-30 08:22:34 -07:00
jpark37
6393399758 win-capture: Remove game capture scaling
The performance save is not worth accidental usage support.

Also remove unnecessary copies from the shmem paths.
2020-06-22 15:07:07 -07:00
jpark37
cdd912b162 win-capture: Reset command pool rather than buffer
Reorganize data to avoid best practices layer warning. Sort of a false
positive in our case because we only have one buffer per pool now, but
implicit layers should be clean citizens.
2020-06-21 06:47:22 -07:00
Jim
9c107738f6
Merge pull request #3015 from jpark37/vulkan-external-fixes
win-capture: Improve Vulkan synchronization
2020-06-21 05:02:20 -07:00
jpark37
a7813c9f87 win-capture: Remove dead VkResult values 2020-06-07 15:22:13 -07:00
jpark37
83dd9c8540 win-capture: Improve Vulkan synchronization
Use general layout and 0 access masks for external synchronization as
specified in the spec.

Also set pipeline stages for maximum synchronization just in case
because it doesn't seem like the pipeline stages are specified.

"Vulkan-incompatible APIs will require the image to be in the GENERAL
layout whenever they are accessing them."

"Whilst it is not invalid to provide destination or source access masks
for memory barriers used for release or acquire operations,
respectively, they have no practical effect. Access after a release
operation has undefined results, and so visibility for those accesses
has no practical effect. Similarly, write access before an acquire
operation will produce undefined results for future access, so
availability of those writes has no practical use. In an earlier version
of the specification, these were required to match on both sides - but
this was subsequently relaxed. These masks should be set to 0."
2020-05-31 22:31:39 -07:00
Jim
ad919b47ea
Merge pull request #2921 from jpark37/vulkan-format-fail
win-capture: Fail on unsupported Vulkan formats
2020-05-21 17:29:57 -07:00
Jim
cb98f456eb
Merge pull request #2595 from jpark37/vulkan-cleanup
Vulkan hook COM clean up, and extension check
2020-05-20 18:20:05 -07:00
Jim
23ee7aebc6
Merge pull request #2589 from jpark37/vulkan-alloc
win-capture: Use VkAllocationCallbacks
2020-05-20 18:17:20 -07:00
jp9000
7993179466 cmake: Add cmake folders 2020-05-13 06:52:37 -07:00
jpark37
30f6870b23 win-capture: Fail on unsupported Vulkan formats
Don't allow unsupported Vulkan formats to fall back to B8G8R8A8.
Probably better to fail completely than do an illegal copy.

Also remove bad conversion for VK_FORMAT_A2R10G10B10_UNORM_PACK32.
Red and blue channels were reversed, and there's no DXGI equivalent.

Addresses #2796. We can do more later if justified.
2020-05-12 12:06:03 -07:00
jpark37
8987fc6c81 win-capture: Verify VK_KHR_external_memory_win32 support
Good practice to verify necessary extensions exist. We'll probably need
keyed mutex extension in the future to support Intel.
2020-03-29 10:08:03 -07:00
jpark37
21dc1f573b win-capture: Cleaner COM usage
Remove hard-coded GUIDs, call correct functions, and clean up
unnecessary casts.
2020-03-29 10:00:36 -07:00
jpark37
348fcd5b00 win-capture: Fix crash if GPU can't Vulkan capture 2020-03-28 09:01:26 -07:00
jpark37
e1739707d0 win-capture: Use VkAllocationCallbacks 2020-03-27 23:59:37 -07:00
jp9000
59351b27ae win-capture/graphics-hook: Separate some debugging stuff
Reduces debugging spam when debugging Vulkan capture is needed in the
hook.
2020-03-13 06:57:16 -07:00
jpark37
9f15514c1a win-capture: Vulkan surface refactor
Make sure HWND tracking is cleaned up when Vulkan surfaces are
destroyed. Also use unbounded linked list to fix games that leak
surfaces on Alt+Tab like Doom.

Also replace CRITICAL_SECTION with SRWLOCK, both for claimed speed
benefit, and to remove initialization code.
2020-03-13 04:36:19 -07:00
jp9000
98c1ac2f80 win-capture/graphics-hook: Add a bit of logging
Just makes sure to be explicit when a device isn't valid.
2020-03-11 02:00:42 -07:00
jpark37
421f97e68d win-capture: Handle vkCreateSwapchainKHR errors
Fixes behavior for an application reported by AMD.
2020-03-10 21:31:40 -07:00
jp9000
428a19edd6 win-capture/graphics-hook: Make vulkan layer all capital 2020-03-09 10:44:04 -07:00
jp9000
229f8d37cb win-capture/graphics-hook: Fix reacquire bug
Sometimes the width/height would initially be at 0x0 when the swap chain
was alt-tabbing, causing the capture to fail full-stop when trying to
reacquire.
2020-03-09 02:50:34 -07:00
jp9000
30e3ef3b49 win-capture/graphics-hook: Log vulkan capture dimensions 2020-03-09 02:34:33 -07:00
jpark37
4f4fc3bfe6 win-capture: Restore Vulkan 1.1 version hack 2020-03-07 20:07:56 -08:00
jpark37
3439be57fe win-capture: Support VK_FORMAT_A8B8G8R8_UNORM_PACK32 2020-03-07 09:58:07 -08:00
jp9000
87f5bd6e9f win-capture/graphics-hook: Fix wrong HWND
Game capture assumes the top-level ancestor HWND, not a child HWND.
This fixes bugs such as the VLC window not capturing anymore due to the
fact that the capture HWND of VLC is a child window of the main VLC
window and not the main VLC window itself.
2020-03-07 07:35:34 -08:00
jpark37
9da6a4d6f6 win-capture: Vulkan capture clean-up
Remove unnecessary DXGI swap chain creation, and use VK_NULL_HANDLE.
2020-03-06 17:04:31 -08:00
jpark37
fc4f3c0934 win-capture: Clean up various VC++ warnings 2020-03-06 17:02:38 -08:00
craftwar
3821a0344e win-capture: Fix format string warning 2020-03-06 19:52:02 +08:00
jpark37
c706a7bed5 win-capture: Handle NULL VkApplicationInfo 2020-03-05 21:20:00 -08:00