I can't believe I wasn't doing this. This is why file output was
getting corrupted. Audio and video send in data from separate threads.
I should be embarassed for not having considered that.
Key lesson: Increase threading paranoia levels. Apparently my
threading paranoid levels are lackluster.
I'm not entirely sure what's going on with my FFmpeg code, but it's
definitely not generating the proper footers for MP4 files, despite the
fact that the footer function succeeds. Going to use AVIs for the time
being still.
- Add CoreAudio device input capture for mac audio capturing. The code
should cover just about everything for capturing mac input device
audio. Because of the way mac audio is designed, users may have no
choice but to obtain the open source soundflower software to capture
their mac's desktop audio. It may be necessary for us to distribute
it with the program as well.
- Hide event backend
- Use win32 events for windows
- Allow timed waits for events
- Fix a few warnings
the signals for scenes could have potentially conflicted with default
source signals. "remove" should be used for source removal, for
example. Changed the scene signals to "item-add" and "item-remove" for
its items.
Implement a few audio options in to the user interface as well as a few
inline audio functions in audio-io.h.
Make it so ffmpeg plugin automatically converts to the desired format.
Use regular interleaved float internally for audio instead of planar
float.
This allows the changing of bideo settings without having to completely
reset all graphics data. Will recreate internal output/conversion
buffers and such and reset the main preview.
Now sources will be properly activated and deactivated when they are in
use or not in use.
Had to figure out a way to handle child sources, and children of
children, just ended up implementing simple functions that parents use
to signal adding/removal to help with hierarchial activation and
deactivation of child sources.
To prevent the source activate/deactivate callbacks from being called
more than once, added an activation reference counter. The first
increment will call the activate callback, and the last decrement will
call the deactivate callback.
Added "source-activate" and "source-deactivate" signals to the main obs
signal handler, and "activate" and "deactivate" to individual source
signal handlers.
Also, fixed the main window so it properly selects a source when the
current active scene has been changed.
- Changed glMapBuffer to glMapBufferRange to allow invalidation. Using
just glMapBuffer alone was causing some unacceptable stalls.
- Changed dynamic buffers from GL_DYNAMIC_WRITE to GL_STREAM_WRITE
because I had misunderstood the OpenGL specification
- Added _OPENGL and _D3D11 builtin preprocessor macros to effects to
allow special processing if needed
- Added fmod support to shaders (NOTE: D3D and GL do not function
identically with negative numbers when using this. Positive numbers
however function identically)
- Created a planar conversion shader that converts from packed YUV to
planar 420 right on the GPU without any CPU processing. Reduces
required GPU download size to approximately 37.5% of its normal rate
as well. GPU usage down by 10 entire percentage points despite the
extra required pass.
There were a *lot* of warnings, managed to remove most of them.
Also, put warning flags before C_FLAGS and CXX_FLAGS, rather than after,
as -Wall -Wextra was overwriting flags that came before it.
Originally, the rendering system was designed to only display sources
and such, but I realized there would be a flaw; if you wanted to render
the main viewport in a custom way, or maybe even the entire application
as a graphics-based front end, you wouldn't have been able to do that.
Displays have now been separated in to viewports and displays. A
viewport is used to store and draw sources, a display is used to handle
draw callbacks. You can even use displays without using viewports to
draw custom render displays containing graphics calls if you wish, but
usually they would be used in combination with source viewports at
least.
This requires a tiny bit more work to create simple source displays, but
in the end its worth it for the added flexibility and options it brings.
The API used to be designed in such a way to where it would expect
exports for each individual source/output/encoder/etc. You would export
functions for each and it would automatically load those functions based
on a specific naming scheme from the module.
The idea behind this was that I wanted to limit the usage of structures
in the API so only functions could be used. It was an interesting idea
in theory, but this idea turned out to be flawed in a number of ways:
1.) Requiring exports to create sources/outputs/encoders/etc meant that
you could not create them by any other means, which meant that
things like faruton's .net plugin would become difficult.
2.) Export function declarations could not be checked, therefore if you
created a function with the wrong parameters and parameter types,
the compiler wouldn't know how to check for that.
3.) Required overly complex load functions in libobs just to handle it.
It makes much more sense to just have a load function that you call
manually. Complexity is the bane of all good programs.
4.) It required that you have functions of specific names, which looked
and felt somewhat unsightly.
So, to fix these issues, I replaced it with a more commonly used API
scheme, seen commonly in places like kernels and typical C libraries
with abstraction. You simply create a structure that contains the
callback definitions, and you pass it to a function to register that
definition (such as obs_register_source), which you call in the
obs_module_load of the module.
It will also automatically check the structure size and ensure that it
only loads the required values if the structure happened to add new
values in an API change.
The "main" source file for each module must include obs-module.h, and
must use OBS_DECLARE_MODULE() within that source file.
Also, started writing some doxygen documentation in to the main library
headers. Will add more detailed documentation as I go.
- Fill in the rest of the FFmpeg test output code for testing so it
actually properly outputs data.
- Improve the main video subsystem to be a bit more optimal and
automatically output I420 or NV12 if needed.
- Fix audio subsystem insertation and byte calculation. Now it will
seamlessly insert new audio data in to the audio stream based upon
its timestamp value. (Be extremely cautious when using floating
point calculations for important things like this, and always round
your values and check your values)
- Use 32 byte alignment in case of future optimizations and export a
function to get the current alignment.
- Make os_sleepto_ns return true if slept, false if the time has
already been passed before the call.
- Fix sinewave output so that it actually properly calculates a middle
C sinewave.
- Change the use of row_bytes to linesize (also makes it a bit more
consistent with FFmpeg's naming as well)
- Add planar audio support. FFmpeg and libav use planar audio for many
encoders, so it was somewhat necessary to add support in libobs
itself.
- Improve/adjust FFmpeg test output plugin. The exports were somewhat
messed up (making me rethink how exports should be done). Not yet
functional; it handles video properly, but it still does not handle
audio properly.
- Improve planar video code. The planar video code was not properly
accounting for row sizes for each plane. Specifying row sizes for
each plane has now been added. This will also make it more compatible
with FFmpeg/libav.
- Fixed a bug where callbacks wouldn't create properly in audio-io and
video-io code.
- Implement 'blogva' function to allow for va_list usage with libobs
logging.
- Implemented better C++ classes for handling scenes/sources/items in
obs.hpp, allowing them to automatically increment and decrement the
references of each, as well as assign them to QVariants.
- Because QVariants are now using the C++ classes, remove the pointer
QVariant wrapper.
- Use the new C++ classes with the QVariant user data of list box items,
both for the sake of thread safety and to ensure that the data
referenced is not freed until removed. NOTE: still might need some
testing.
- Implemented a source-remove signal from libobs, and start using that
signal instead of the source-destroy signal for signalling item
removal.
Scene items previously were removed by calling obs_sceneitem_destroy,
but this proved to be a potential race condition where two different
threads could try to destroy the same scene item at the same time.
Instead of doing that, reference counting is now used on scene items,
and an explicit obs_sceneitem_remove function is used instead for item
removal, which sets a 'removed' variable to ensure it can only be called
exactly one time.
The previous commit used the scene as a parameter to check to see if
the scene item was still present within the scene before destroying, but
this was actually unnecessary because the fault was because the destroy
signal was being triggered *before* the scene's mutex locked, thus
causing a race condition. I changed the code so that it signals after
the lock instead of before, so the scene parameter should no longer be
necessary.
Fixes a deadlock when trying to remove a source from the GUI. The scene
item signal handlers would mark the source as removed which results in
the video thread also trying to run obs_sceneitem_destroy thereby
deadlocking the video thread (and the GUI thread)
- Fix the size issue with list boxes on mac. Was displaying the list
boxes with an improper size. Turns out it was just the wrong size
policies on the frame below.
- Ensure the main windows are fully displayed *before* initializing
subsystems. This ensures that the graphics system will properly start
up on macos, and allows the glitch fix.
- Made a workaround for weird QT glitch that would happen to the parent
of a pure native widget that also has internal painting fully
disabled. (Should definitely write an example and report this bug on
the QT forums)
- Had the wrong names set for the up/down widgets for sources/scenes
- Updated the settings dialog and gave most of the widgets actual object
names
- Added code for the settings window. Settings window should now at
least display.
- I seem to have fixed ths issues with the main preview widget. It
seems you just need to set the right window attributes to stop it from
breaking. Though when opengl is enabled, there appears to be a weird
background glitch in the Qt stuff -- I'm not entirely sure what's
going on. Bug in Qt?
Also fixed the layout issues, and the widget now properly resizes and
centers in to its parent widget.
- Prevent the render loop from accessing data if the data isn't valid.
Because obs->data is freed before the graphics stuff, it can cause
the graphics to keep trying to query the obs->data.displays_mutex
after it had already been destroyed.
--------------------------------------------------
Notes and details
--------------------------------------------------
Why was this done? Because wxWidgets was just lacking in many areas. I
know wxWidgets is designed to be used with native controls, and that's
great, but wxWidgets just is not a feature-complete toolkit for
multiplatform applications. It lacks in dialog editors, its code is
archaic and outdated, and I just feel frustrated every time I try to do
things with it.
Qt on the other hand.. I had to actually try Qt to realize how much
better it was as a toolkit. They've got everything from dialog editors,
to an IDE, a debugger, build tools, just everything, and it's all
top-notch and highly maintained. The focus of the toolkit is
application development, and they spend their time trying to help
people do exactly that: make programs. Great support, great tools,
and because of that, great toolkit. I just didn't want to alienate any
developers by being stubborn about native widgets.
There *are* some things that are rather lackluster about it and design
choices I disagree with though. For example, I realize that to have an
easy to use toolkit you have to have some level of code generation.
However, in my personal and humble opinion, moc just feels like a
terrible way to approach the problem. Even now I feel like there are a
variety of ways you could handle code generation and automatic
management of things like that. I don't like the idea of circumventing
the language itself like that. It feels like one giant massive hack.
--------------------------------------------------
Things that aren't working properly:
--------------------------------------------------
- Settings dialog is not implemented. The dialog is complete but the
code to handle the dialog hasn't been constructed yet.
- There is a problem with using Qt widgets as a device target on
windows, with at least OpenGL: if I have the preview widget
automatically resize itself, it seems to cause some sort of video
card failure that I don't understand.
- Because of the above, resizing the preview widget has been disabled
until I can figure out what's going on, so it's currently only a
32x32 area.
- Direct3D doesn't seem to render correctly either, seems that the
viewport is messed up or something. I'm sort of confused about
what's going on with it.
- The new main window seems to be triggering more race conditions than
the wxWidgets main window dialog did. I'm not entirely sure what's
going on here, but this may just be existing race conditions within
libobs itself that I just never spotted before (even though I tend to
be very thorough with race conditions any time I use variables
cross-thread)
Completely revamped the entire media i/o data and handlers. The
original idea was to have a system that would have connecting media
inputs and outputs, but at a certain point I realized that this was an
unnecessary complexity for what we wanted to do. (Also, it reminded me
of directshow filters, and I HATE directshow with a passion, and
wouldn't wish it upon my greatest enemy)
Now, audio/video outputs are connected to directly, with better callback
handlers, and will eventually have the ability to automatically handle
conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses
them. Doing this will allow the video/audio i/o handlers to also
prevent duplicate conversion, as well as make it easier/simple to use.
My true goal for this is to make output and encoder plugins as simple to
create as possible. I want to be able to be able to create an output
plugin with almost no real hassle of having to worry about image
conversions, media inputs/outputs, etc. A plugin developer shouldn't
have to handle that sort of stuff when he/she doesn't really need to.
Plugins will be able to simply create a callback via obs_video() and/or
obs_audio(), and they will automatically receive the audio/video data in
the formats requested via a simple callback, without needing to do
almost anything else at all.
- Mixing still isn't implemented, but the audio system should be able
to start up, and mix at least once audio line for the time being.
Will have to write some test audio sources to verify things are
working properly, and build the rest of the output functionality.
- Made it so that when a source is added or removed from a scene it
will add a reference to sourceSceneRefs (std::unordered_map). Each
source adds a reference to that every time they are added to a scene,
and releases a reference from it when they are removed from a scene.
When the value reaches 0, the source is no longer in any scenes, and
is then marked for removal and destroyed.
Before, I was using the source internal reference counter, which is a
really bad thing to do because I don't know what might actually be
referencing it. So using a separate discrete reference counter for
the number of scenes it's in is better in this case.
- Added an extra comment regarding source removal
- Made is so that when a scene item is removed from the sources list
box, it removes the item based upon the GetClientData rather than by
its string name. This is done because there may be duplicate sources
in a particular scene, and it's important to remove the right one.
- When the remove source tool icon is clicked, it will now remove the
source from the scene.
- Fixed a bug where the scene item removal callback would add the scene
item to the list instead of removing it.
- Changed AddSourcePopup to AddSourcePopupMenu. Name actually confused
me once despite being the writer, so it was clearly a bad name.
Sources can now be added to scenes via user interface. It's a little
convoluted because everything has to work through OBS signals to ensure
that plugins/etc can modify the scenes/sources exernally.
Also, when switching scenes, it will properly list sources for the
scene you changed to.
jp9000
Palana
BtbN
Zachary Lund