The xcomposite window capture crashes were due to a few factors:
-------------------------------
1.) The source's X error handler was possibly being overwritten by
another part of the program despite us locking the display, presumably
something in Qt which isn't locking the display when pushing/popping its
own error handler (though this is not yet certain). The source's calls
to X functions happen in the graphics thread, which is separate from the
UI thread, and it was noticed that somehow the error handler would be
overwritten almost seemingly at random, indicating that something else
in the program outside of OBS code was not locking the display while
pushing/popping the error handler.
To replicate this, make it so that the source cannot find the target
window and so it continually searches for it each video_tick call, then
resize the main OBS window continually (which causes Qt to push/pop its
own error handlers). A crash will almost always occur due to BadWindow
despite our error handling.
2.) Calling X functions with a window ID that no longer exists,
particularly XGetWindowAttributes, in conjunction the unknown error
handler set in case #1 would cause the program to outright crash because
that error handler is programmed to crash on BadWindow for whatever
reason. The source would call X functions without even checking if
'win' was 0.
3.) The source stored window IDs (in JSON, even if they've long since
become invalid/pointless, such as system restarts). This is a bad
practice and will result in more cases of BadWindow.
Fixing the problem (reducing the possibility of getting BadWindow):
-------------------------------
Step 1.) Deprecate and ignore window IDs in stored settings. Instead of
using window IDs to find the window, we now must always search the
windows and find the target window via the window name exclusively.
This helps ensure that we actually consistently have a working window
ID.
Step 2.) Do not call any X functions if the window ID is 0.
Step 3.) Reset the window ID to 0 any time the window has updated, and
make the source find the window again to ensure it still exists before
attempting to use any X functions on the window ID again.
This is probably not necessary but might fix an issue where errors pass
through to other parts of the program, possibly causing the crash on
exit related to the xcomposite capture.
On i3wm, windows aren't unmapped when switching away from a window's
workspace, but it does cause OBS to lose the capture. Because
switching back will not trigger a MapNotify, the capture fails to
restart unless you resize or move the window (ConfigureNotify). An
Expose event is fired by the wm, however, so catching this correctly
restarts the capture.
jp9000
Lucian Poston
raincomplex