I've come to realize that it's probably not wise to deviate from the
original version's functionality due to the fact that the original
version works without issues. I'm wondering if some of the capture
problems have been due to the fact that the direct hook method (via
CreateRemoteThread) was removed, so I put it back in, made it default,
and added an option to use anti-cheat compatibility just like in the
original version.
When hooking 64bit functions, sometimes the offset between the function
being hooked and the hook itself can be large enough to where it
requires a 64bit offset to be used. However, because a 64bit jump
requires overwriting so many code instructions in the function, it can
sometimes overwrite code in to an adjacent function, thereby causing a
crash.
The 64bit hook bounce (created by R1CH) is designed to prevent using
very long jumps in the target by creating executable memory within a
32bit offset of that target, and then writing it with the 64bit long
jump instruction instead. Then in the target function, it will jump to
that memory instead, thus forcing the actual hooked function to use a
32bit hook instead of a 64bit hook, and using at most 5 bytes for the
actual hook, preventing any likelihood of it overwriting an adjacent
function.
If capture starts too quickly, the file mapping will return 2, which
means file not found, and it would then reset the capture and try again.
Sometimes this would result in long intervals where it wouldn't capture.
This fixes the issue by simply making game capture retry if file mapping
returns error number 2.
This fixes an issue primarily with filter rendering: when capturing
windows and displays, their alpha channel is almost always 0, causing
the image to be completely invisible unintentionally. The original fix
for this for many sources was just to turn off the blending, which would
be fine if you're not rendering any filters, but filters will render to
render targets first, and that lack of alpha will end up carrying over
in to the final image.
This doesn't apply to any mac captures because mac actually seems to set
the alpha channel to 1.
I had this issue where IDXGISwapChain::ResizeBuffers would fail in the
hooks, causing games to crash when they resized their backbuffers
because ResizeBuffers would return an 'invalid call' HRESULT value. In
the ResizeBuffers documentation it says that it will only happen if a
backbuffer currently has any outstanding references, but there's no way
this would happen unless ResizeBuffers internally calls Present or vise
versa.
After ResizeBuffers has been called, the very first call to Present will
somehow seemingly invalidate and/or destroy the current backbuffer.
It's very strange, but that seems to be what's going on, at least for
the game I was testing. So if you are performing a post-overlay
capture, then you must ignore the capture on the very first call to
Present.
It's Microsoft's code so you can't really know what's going on, you just
have to work around these strange issues seemingly in the dark.
Apparently someone dumb (aka me) neglected to properly handle the inline
graphics hook API functions. You're not supposed to 'extern' inline
functions, they need to be defined for each file when ever they're used.
Apparently neglected to use the reference operator. I think this may
partially be one of the reasons why many developers still choose to use
pointers instead of references, but fortunately an actual GOOD compiler
warns about this (aka anything but vc)
Clears up a warning (to prevent && and || confusion), and clarifies what
specifically the if statement is trying to accomplish (check to see if
the capture is valid)
The HWND type is a void pointer, but HWND values are global and always
32bit despite, so casting to 32bit can cause cast warnings on actual
good compilers like gcc via mingw. This change correctly handles the
casting to 32bits without producing unwanted warnings or errors on
mingw.
win-capture should not postfix .lib to psapi.
The graphics hook also requires psapi when linking.
Also change some link libs as mingw-w64 libraries are not postfixed
.lib.
If the PSAPI_VERSION macro is not set to 1 when using
GetProcessImageFileName, it will attempt to import it as
K32GetProcessImageFileName from kernel32.dll instead of psapi.dll, which
breaks compatibility with vista and xp.
Having macros that state what these numbers mean is much more ideal than
just having a random number thrown in there, wondering why it was used
and what its purpose is (magic numbers).
The activate button is just silly for configuration in retrospect. It's
confusing to users, and was even confusing to some other developers.
Instead of using an 'Activate' button for game capture every time you
want to capture a window, just make the 'window' list have a default 'no
window' value (empty), and then have it always active when an actual
window is selected. The way syphon handles this on mac is actually
where I took the idea from (as suggested by Palana).
With the new code that checks to see if the source is visible, I didn't
realize that I actually didn't set the source variable, so it would end
up never actually drawing.
If shared memory file mapping fails, I've found that it's somewhat
normal due to something in windows -- usually the capture will always
eventually start up after a few tries. Only seems to apply to some
games though, for example seems to happen with counterstrike a lot for
some strange reason. Capture always eventually starts back up though.
I remember seeing this with OBS1 as well in many cases but always
thought it was some sort of fluke