When using a chain hook method (forward or reverse), it was unwisely
assumed that the previous hook in the chain would not overwrite new
hooks when it's called. When the game capture hook calls the previous
hook in the chain, certain other programs that hook (in this case,
rivatuner on-screen display) would overwrite the hook with older data
and erase the game capture hook, causing it to only capture the first
frame and then never capture again.
This patch ensures that the hook is always saved before calling the next
hook in the chain and then restored after the call returns. It also
preserves any new hooks that may be added on top of it at any point.
The new 'offset' value was not being passed back to the caller, which
caused the caller to continue to use the old value and thus would cause
an invalid hook and crash.
When hooking 64bit functions, sometimes the offset between the function
being hooked and the hook itself can be large enough to where it
requires a 64bit offset to be used. However, because a 64bit jump
requires overwriting so many code instructions in the function, it can
sometimes overwrite code in to an adjacent function, thereby causing a
crash.
The 64bit hook bounce (created by R1CH) is designed to prevent using
very long jumps in the target by creating executable memory within a
32bit offset of that target, and then writing it with the 64bit long
jump instruction instead. Then in the target function, it will jump to
that memory instead, thus forcing the actual hooked function to use a
32bit hook instead of a 64bit hook, and using at most 5 bytes for the
actual hook, preventing any likelihood of it overwriting an adjacent
function.
These functions allow the safe hooking of windows functions,
specifically windows API functions that may or may not have built-in
machine code to help aid in reverse chain hooks.
If a new hook is applied to an existing forward hook, that hook will be
preserved to prevent that new hook's data from being removed
unintentionally.
Hopefully with all these precautions this will reduce the likelihood of
crashes and abnormal hook behavior, while allowing existing hooks to be
preserved, and allowing new hooks to be applied.
jp9000