The xcomposite window capture crashes were due to a few factors:
-------------------------------
1.) The source's X error handler was possibly being overwritten by
another part of the program despite us locking the display, presumably
something in Qt which isn't locking the display when pushing/popping its
own error handler (though this is not yet certain). The source's calls
to X functions happen in the graphics thread, which is separate from the
UI thread, and it was noticed that somehow the error handler would be
overwritten almost seemingly at random, indicating that something else
in the program outside of OBS code was not locking the display while
pushing/popping the error handler.
To replicate this, make it so that the source cannot find the target
window and so it continually searches for it each video_tick call, then
resize the main OBS window continually (which causes Qt to push/pop its
own error handlers). A crash will almost always occur due to BadWindow
despite our error handling.
2.) Calling X functions with a window ID that no longer exists,
particularly XGetWindowAttributes, in conjunction the unknown error
handler set in case #1 would cause the program to outright crash because
that error handler is programmed to crash on BadWindow for whatever
reason. The source would call X functions without even checking if
'win' was 0.
3.) The source stored window IDs (in JSON, even if they've long since
become invalid/pointless, such as system restarts). This is a bad
practice and will result in more cases of BadWindow.
Fixing the problem (reducing the possibility of getting BadWindow):
-------------------------------
Step 1.) Deprecate and ignore window IDs in stored settings. Instead of
using window IDs to find the window, we now must always search the
windows and find the target window via the window name exclusively.
This helps ensure that we actually consistently have a working window
ID.
Step 2.) Do not call any X functions if the window ID is 0.
Step 3.) Reset the window ID to 0 any time the window has updated, and
make the source find the window again to ensure it still exists before
attempting to use any X functions on the window ID again.
Add checks to make sure the pulseaudio operation objects are actually
valid and the request did not fail right away. Other pulseaudio
functions dealing with the objects expect them not to be NULL,
otherwise they will kill the application with a failing assert.
For some reason in the FFmpeg output, this AVCodecContext variable is
being set to 1 by FFmpeg itself somewhere, and it's causing a massive
slowdown when encoding with FFmpeg directly. This should be set to 0 to
specify to use as many threads as necessary.
This reverts commit 8d520b970d.
This can actually cause a hard lock due to the windows API when
destroying window capture. When the graphics thread locks the source
list for doing tick or render, and then the UI thread tries to destroy a
source, the UI thread will wait for the graphics thread to complete
rendering/ticking of sources. The video_tick of window capture would
then check windows in the same process and try to query the window's
name via GetWindowText. However, GetWindowText is synchronous, and will
not return until the window event has been processed by the UI thread,
so it will perpetually lock because the two threads are waiting for each
other to finish.
On windows, if you were saving a file name or directory with characters
that are not of the current windows character set, it could cause the
file saving process to fail. This fixes it so that on windows it uses
wmain and converts the unicode command line to a UTF-8 command line,
which works with FFmpeg.
Prevents game capture from acting as a global source. This fixes an
issue where a game capture in another scene could capture a window and
prevent a separate game capture in the current scene from being able to
capture that same window.
Completely shut down monitor capture when it's not being shown in the
program (for example in a different scene). This fixes an issue where
it would cause lag when a game enters fullscreen mode.
Instead of having a "cbr" setting that turns CBR on and off, adds a
"rate_control" parameter that sets the rate control method, which can be
one of the following: CBR, ABR, VBR, CRF.
If the "cbr" setting is used, it will throw a deprecation warning to the
log.
Instead of using an option that turns CBR on/off, adds rate control
methods: VBR, CBR, CQP, Lossless.
This moves lossless from being a preset to being a rate control method.
When using per-encoder rescaling, QSV would overwrite the current
encoder scale value in the get_video_info callback with the base video
width/height instead of using the current encoder width/height.
(Also modifies obs-ffmpeg to handle empty frames on EOF)
Previously the demuxer could hit EOF before the decoder threads are
finished, resulting in truncated output. In the worse case scenario the
demuxer could read small files before ff_decoder_refresh even has a chance
to start the clocks, resulting in no output at all.
There was no error checking when sending headers/metadata, so what would
happen is that if a header/metadata send failed (meaning the socket was
disconnected), it would continue to act as if it was still connected,
and it would block and lock up on the next send/recv call.
In aa4e18740a I mistakenly thought that I could add the variables
back in and that it would automatically cull variables that aren't used,
but that wasn't the case -- the shader parser always checks to see
whether parameters were set, and if they're not, it'll fail. This fixes
an issue where the shader would try to access parameters that are no
longer needed and fail due to the shader parameter check.
YUV-based shader support has been removed (due to the fact that no
sources ever use YUV shading) so there's no reason to keep around the
YUV processing code.
Currently, multiple QSV encoders cannot be active at the same time
(otherwise it will crash). This is a temporary solution to prevent
crashes from occurring when more than one QSV encoder tries to start up
at the same time.
Additionally, in the future there should be a way for encoders to be
able to communicate with the front-end when an error such as this
occurs.
