Allows the ability to use scale filters such as point, bicubic, lanczos
on specific scene items, disabled by default. When using one of the
latter two options, if the item's scale is under half of the source's
original size, it uses the bilinear low resolution downscale shader
instead.
(Note: This commit also modifies obs-ffmpeg and obs-outputs)
API Changed:
obs_output_info::void (*stop)(void *data);
To:
obs_output_info::void (*stop)(void *data, uint64_t ts);
This fixes the long-time design flaw where obs_output_stop and the
output 'stop' callback would just shut down the output without
considering the timing of when obs_output_stop was used, discarding any
possible buffering and causing the output to get cut off at an
unexpected timing.
The 'stop' callback of obs_output_info now takes a timestamp with the
expectation that the output will use that timestamp to stop output data
in accordance to that timing. obs_output_stop now records the timestamp
at the time that the function is called and calls the 'stop' callback
with that timestamp. If needed, obs_output_force_stop will still stop
the output immediately without buffering.
(Note: Also modified the obs-ffmpeg plugin module)
Allows the ability for frame data to pass 8-bit grayscale images (Y800
color format).
Closesjp9000/obs-studio#515
Adds deinterlacing API functions. Both standard and 2x variants are
supported. Deinterlacing is set via obs_source_set_deinterlace_mode and
obs_source_set_deinterlace_field_order.
This was implemented in to the core itself because deinterlacing should
happen before effect filters are processed, but after async filters are
processed. If this were added as a filter, there is the possibility
that a different filter is processed before deinterlacing, which could
mess with the result. It was also a bit easier to implement this way
due to the fact that that deinterlacing may need to have access to the
previous async frame.
Effects were split in to separate files to reduce load time (especially
for yadif shaders which take a significant amount of time to compile).
Instead of just updating the async texture variables directly in the
source, allow the ability to pass the async texture variables via
function parameters to allow the ability to parse more than one frame to
more than one texture.
This code is primarily intended to be used to upload/convert the
"previous" async frame for the deinterlacer (if necessary).
Just creates an effect to the target variable only if its current value
is null. This will be used for deinterlacing effects to prevent having
to compile the shaders unless they're actually being used.
If the circular audio buffer of the source has data remaining that's
less than the audio frame tick count (1024 frames), it would just leave
that audio data on the source without discarding it. However, this
could cause audio buffering to increase unnecessarily under certain
circumstances (when the next audio timestamp is within the timestamp
jump window), so it would append data to that circular buffer despite
the audio stopping that long ago, causing audio buffering to have to
increase to compensate.
Instead, just discard pending audio if it hasn't been written to. In
other words, if the audio has stopped and there's insufficient audio
left to continue processing.
With the new audio subsystem, audio buffering is minimal at all times.
However, when the audio buffering is too small or non-existent, it would
cause the audio encoders to start with a timestamp that was actually
higher than the first video frame timestamp. Video would have some
inherent buffering/delay, but then audio could return and encode almost
immediately. This created a possible window of empty time between the
first encoded video packet and the first encoded audio packet, where as
audio buffering would cause the first audio packet's timestamp to always
be way before the first video packet's timestamp. It would then
incorrectly assume the two starting points were in sync.
So instead of assuming the audio data is always first, this patch makes
video wait for audio data comes in, and conversely buffers audio data
until video comes in, and tries to find a starting point within that
video data instead, ensuring a synced starting point whether audio
buffering is active or not.
The seamless audio looping code would erroneously trigger for things
that weren't loops, causing the audio data to continually push back and
ignore timestamps, thus going out of sync.
There does need to be loop handling code, but due to the fact that other
things may need to trigger this code, it's best just to clear the audio
data and start from a fresh sync point. Unfortunately for the case of
loops, this means the window in which audio data loops and video frames
loop need to be muted.
This fixes an age-old issue where audio samples could be lost or audio
could temporarily go out of sync in the case of looping videos. When
audio/video data is looping, there's a window between when the audio
data resets its timestamp value and when the video data resets its
timestamp value. This method simply pushes back the audio data while in
that window and does not modify sync, and when it detects that its out
of the loop window it simply forces a resync of the audio data in the
circular buffer.
This ensures that minimal audio data is lost in the loop process, and
minimizes the likelihood of any sort of sync issues associated with
looping.
Instead of applying the resampler offset right away (to each audio
packet), apply the resampler offset when the timestamps are converted to
system timestamps. This fixes an issue where if audio timestamps reset
to 0 (for whatever reason), the offset would cause the timestamp to go
in to the negative.
(Note: This commit also modifies UI)
Instead of using signals, use designated callback lists for audio
capture and audio control helpers. Signals aren't suitable here due to
the fact that signals aren't meant for things that happen every frame or
things that happen every time audio/video is received. Also prevents
audio from being allocated every time these functions are called due to
the calldata structure.
Transition sources are implemented by registering a source type as
OBS_SOURCE_TYPE_TRANSITION. They're automatically marked as video
composite sources, and video_render/audio_render callbacks must be set
when registering the source. get_width and get_height callbacks are
unused for these types of sources, as transitions automatically handle
width/height behind the scenes with the transition settings.
In the video_render callback, the helper function
obs_transition_video_render is used to assist in automatically
processing and rendering the audio. A render callback is passed to the
function, which in turn passes to/from textures that are automatically
rendered in the back-end.
Similarly, in the audio_render callback, the helper function
obs_transition_audio_render is used to assist in automatically
processing and rendering the audio. Two mix callbacks are used to
handle how the source/destination sources are mixed together. To ensure
the best possible quality, audio processing is per-sample.
Transitions can be set to automatically resize, or they can be set to
have a fixed size. Sources within transitions can be made to scale to
the transition size (with or without aspect ratio), or to not scale
unless they're bigger than the transition. They can have a specific
alignment within the transition, or they just default to top-left.
These features are implemented for the purpose of extending transitions
to also act as "switch" sources later, where you can switch to/from two
different sources using the transition animation.
Planned (but not yet implemented and lower priority) features:
- "Switch" transitions which allow the ability to switch back and forth
between two sources with a transitioning animation without discarding
the references
- Easing options to allow the option to transition with a bezier or
custom curve
- Manual transitioning to allow the front-end/user to manually control
the transition offset
(Note: test and UI are also modified by this commit)
API Changed (removed "enum obs_source_type type" parameter):
-------------------------
obs_source_get_display_name
obs_source_create
obs_get_source_output_flags
obs_get_source_defaults
obs_get_source_properties
Removes the "type" parameter from these functions. The "type" parameter
really doesn't serve much of a purpose being a parameter in any of these
cases, the type is just to indicate what it's used for.
The new audio subsystem fixes two issues:
- First Primary issue it fixes is the ability for parent sources to
intercept the audio of child sources, and do custom processing on
them. The main reason for this was the ability to do custom
cross-fading in transitions, but it's also useful for things such as
side-chain effects, applying audio effects to entire scenes, applying
scene-specific audio filters on sub-sources, and other such
possibilities.
- The secondary issue that needed fixing was audio buffering.
Previously, audio buffering was always a fixed buffer size, so it
would always have exactly a certain number of milliseconds of audio
buffering (and thus output delay). Instead, it now dynamically
increases audio buffering only as necessary, minimizing output delay,
and removing the need for users to have to worry about an audio
buffering setting.
The new design makes it so that audio from the leaves of the scene graph
flow to the root nodes, and can be intercepted by parent sources. Each
audio source handles its own buffering, and each audio tick a specific
number of audio frames are popped from the front of the circular buffer
on each audio source. Composite sources (such as scenes) can access the
audio for child sources and do custom processing or mixing on that
audio. Composite sources use the audio_render callback of sources to do
synchronous or deferred audio processing per audio tick. Things like
scenes now mix audio from their sub-sources.
(Note: This commit breaks libobs compilation. Skip if bisecting)
Adds a "composite" source type which is used for sources that composite
one or more sub-sources. The audio_render callback is called for
composite sources to allow those types of sources to do custom
processing of the audio of its sub-sources.
(Note: This commit breaks libobs compilation. Skip if bisecting)
Removes audio lines and stores the circular buffer for the audio on the
source itself.
(Note: This commit breaks libobs compilation. Skip if bisecting)
The mixers that a source was assigned to were originally stored in the
audio line. This will store it in the sources themselves instead.
Ensures that the packet dts_usec vals which are generated for
syncing/interleaving use the proper offset relative to where they're
supposed to be starting from. The negative DTS of a first video packet
could potentially have been applied twice due to this.
This was originally used for calculating audio volume if transitions
were active, but transitions won't work that way so tracking the active
transitions is no longer needed.
(Note: This commit breaks UI compilation. Skip if bisecting)
API Removed:
------------------------
obs_add_source
API Changed:
------------------------
obs_source_remove: Now just marks/signals a source for removal
The concept of "user sources" is flawed: it was something that the
front-end was forced to deal with if it wanted to automate source
saving/loading, and often it had to code around it. That's not how
saving/loading should work, a front-end should be allowed to manage
lists of sources in the way it explicitly chooses, and it should be able
to choose which sources it wants to save/load.
This prevents encoders (hardware encoders in particular) from being
continually active when all outputs disconnect from an encoder. This is
mostly just a temporary measure; the encoding interface may need a bit
of a redesign. It will also definitely needs to be able to flush at
some point. Currently when an output is stopped, the pending data is
discarded, which needs to be fixed.
Allows objects to be created regardless of whether the actual id exists
or not. This is a precaution that preserves objects/settings if for
some reason the id was removed for whatever reason (plugin removed, or
hardware encoder that disappeared). This was already added for sources,
but really needs to be added for other libobs objects as well: outputs,
encoders, services.
This feature allows a user to delay an output (as long as the output
itself supports it). Needless to say this intended for live streams,
where users may want to delay their streams to prevent stream sniping,
cheating, and other such things.
The design this time was a bit more elaborate, but still simple in
design: the user can now schedule stops/starts without having to wait
for the stream itself to stop before being able to take any action.
Optionally, they can also forcibly stop stream (and delay) in case
something happens which they might not want to be streamed.
Additionally, a new option was added to preserve stream cutoff point on
disconnections/reconnections, so that if you get disconnected while
streaming, when it reconnects, it will reconnect right at the point
where it left off. This will probably be quite useful for a number of
applications in addition to regular delay, such as setting the delay to
1 second and then using this feature to minimize, for example, a
critical stream such as a tournament stream from getting any of its
stream data cut off. However, using this feature will of course cause
the stream data to buffer and increase delay (and memory usage) while
it's in the process of reconnecting.
API Changed:
---------------------------
From:
- bool obs_startup(const char *locale, profiler_name_store_t *store);
To:
- bool obs_startup(const char *locale, const char *module_config_path,
profiler_name_store_t *store);
Summary:
---------------------------
This allows plugin modules to store plugin-specific configuration data
(rather than only allowing objects to store configuration data). This
will be useful for things like caching data, for example looking up and
storing ingests from remote (rather than storing locally), or caching
font data (so it doesn't have to build a font cache each time), among
other things.
Also adds a module-specific directory for the UI
Due to all the threads in libobs it wouldn't be safe to make that
parameter reconfigurable after libobs is initialized without adding
even more synchronization. On the other hand, adding a function to set
the name store before calling obs_startup would solve the problem of
passing a name store into libobs, but it can lead to more complicated
semantics for obs_get_profiler_name_store (e.g., should it always return
the current name store even if libobs isn't initialized until someone
calls set_name_store(NULL)? should obs_shutdown call
set_name_store(NULL)? Passing it as obs_startup parameter avoids
these (and hopefully other) potential misunderstandings
(Non-compiling commit: windowless-context branch)
API Changed:
---------------------
Removed functions:
- obs_add_draw_callback
- obs_remove_draw_callback
- obs_resize
- obs_preview_set_enabled
- obs_preview_enabled
Removed member variables from struct obs_video_info:
- window_width
- window_height
- window
Summary:
---------------------
Changes the core libobs API to not be dependent upon a main window/view.
If you wish to draw to a window/view, use an obs_display object to
handle it.
This allows the use of libobs without requiring a window to be present
on the system. This is also prunes code that had to be needlessly
duplicated to handle the "main" window.
The "clamped" video time is the system time per video frame that is
closest to the current system time, but always divisible by the frame
interval. For example, if the last frame system timestamp was 1600 and
the new frame is 2500, but the frame interval is 800, then the
"clamped" video time is 2400.
This clamped value is useful to get the relative system time without any
jitter.
Implements exponential backoff for consecutive reconnects, which is
useful to prevent too many connections from trying to reconnect back to
a service at once over a short period of time in the case of potential
service downtime. Exponential backoff causes each subsequent reconnect
attempt to double its timeout duration.
Allows the ability to hint at encoders what format should be used.
This is particularly useful if libobs is currently operating in planar
4:4:4, but you want to force an encoder used for streaming to convert to
NV12 to prevent streaming issues.
The obs_source::async_reset_texture variable can cause a data race
between threads to occur because it could be set to true in one thread
then changed back to false in another thread. This could cause the
async texture to not update its size when it's supposed to, which can
cause a crash or corruption when copying data from a frame of a
differing size.
The solution to this is to:
- Delete the async_reset_texture variable, and make the
set_async_texture_size function change the texture size if the
async_width, async_height, or async_format variables differ from the
frame's width/height/format. Those variables are then only ever set
in the libobs graphics thread.
- Make the cache_video function use separate variables from other
functions to detect a change in size (due to the fact that the texture
size should only be resized in the libobs graphics thread). These
variables are async_cache_width, async_cache_height, and
async_cache_format, which are only be set in the thread that calls
obs_source_output_video.
How to replicate the data race:
- On OSX, use window capture on a textedit window, then continually
resize the textedit window.
This fixes an issue where cache frames would not free at all after
having been allocated with no upper limit on the cached frame size. If
cached frames go unused for a specific period of time, they are
deallocated and removed from the cache.
This is preferable to having an upper cache limit due to the potential
for async delay filtering.
Async frames are only swapping when rendering, or when not visible.
This is a flawed design due to the fact that there are certain
circumstances where the source is neither visible nor currently
rendering.
This is what caused a memory leak when scene items were marked as
invisible, because if a source has an async child source and decides not
to render that source for whatever reason, the child source would not
process the async frames at all, and the cache would just grow.
To fix this, simply moving the async frame cycle to tick fixes the issue
due to the fact that tick is always called regardless of circumstance.
obs_source_process_filter tried to do everything in a single function,
but the problem is that effect parameters would not properly be
accounted for due to the way it internally draws, therefore it was
necessary to split the functions in to two, you first call
obs_source_process_filter_begin, then you set your effect parameters,
then you finally call obs_source_process_filter_end. This ensures that
when the filter is drawn, that the effect parameters are set.
For the show/hide and activate/deactivate callbacks, schedule these
callbacks to only be called from within the video thread rather than in
a separate thread. This ensures that any potential graphics activity
that occurs within them is kept in the same thread.
API changed:
--------------------------
void obs_output_set_audio_encoder(
obs_output_t *output,
obs_encoder_t *encoder);
obs_encoder_t *obs_output_get_audio_encoder(
const obs_output_t *output);
obs_encoder_t *obs_audio_encoder_create(
const char *id,
const char *name,
obs_data_t *settings);
Changed to:
--------------------------
/* 'idx' specifies the track index of the output */
void obs_output_set_audio_encoder(
obs_output_t *output,
obs_encoder_t *encoder,
size_t idx);
/* 'idx' specifies the track index of the output */
obs_encoder_t *obs_output_get_audio_encoder(
const obs_output_t *output,
size_t idx);
/* 'mixer_idx' specifies the mixer index to capture audio from */
obs_encoder_t *obs_audio_encoder_create(
const char *id,
const char *name,
obs_data_t *settings,
size_t mixer_idx);
Overview
--------------------------
This feature allows multiple audio mixers to be used at a time. This
capability was able to be added with surprisingly very little extra
overhead. Audio will not be mixed unless it's assigned to a specific
mixer, and mixers will not mix unless they have an active mix
connection.
Mostly this will be useful for being able to separate out specific audio
for recording versus streaming, but will also be useful for certain
streaming services that support multiple audio streams via RTMP.
I didn't want to use a variable amount of mixers due to the desire to
reduce heap allocations, so currently I set the limit to 4 simultaneous
mixers; this number can be increased later if needed, but honestly I
feel like it's just the right number to use.
Sources:
Sources can now specify which audio mixers their audio is mixed to; this
can be a single mixer or multiple mixers at a time. The
obs_source_set_audio_mixers function sets the audio mixer which an audio
source applies to. For example, 0xF would mean that the source applies
to all four mixers.
Audio Encoders:
Audio encoders now must specify which specific audio mixer they use when
they encode audio data.
Outputs:
Outputs that use encoders can now support multiple audio tracks at once
if they have the OBS_OUTPUT_MULTI_TRACK capability flag set. This is
mostly only useful for certain types of RTMP transmissions, though may
be useful for file formats that support multiple audio tracks as well
later on.
The temporary unoptimized code we were using before just completely
allocated a new copy of each frame every single time a new async frame
was output by the source plugin. This just creates a cache of frames as
needed for the current format/width/height to minimize the allocation
and deallocation. If new frames come in that are of a different
format/width/height, it'll just clear the cache. This is a fairly
important optimization.
all the async video related stuff usually started with async_*, and
there were two that didn't. So I just renamed them so they have the
same naming convention
If an async video source stops video for whatever reason, it would get
stuck on the last frame that was played. This was particularly awkward
when I wanted to give the user the ability to deactivate a source such
as a webcam because it would get stuck on the last frame.
Previously, the design for the interaction between the encoder thread
and the graphics thread was that the encoder thread would signal to the
graphics thread when to start drawing each frame. The original idea
behind this was to prevent mutually cascading stalls of encoding or
graphics rendering (i.e., if rendering took too long, then encoding
would have to catch up, then rendering would have to catch up again, and
so on, cascading upon each other). The ultimate goal was to prevent
encoding from impacting graphics and vise versa.
However, eventually it was realized that there were some fundamental
flaws with this design.
1. Stray frame duplication. You could not guarantee that a frame would
render on time, so sometimes frames would unintentionally be lost if
there was any sort of minor hiccup or if the thread took too long to
be scheduled I'm guessing.
2. Frame timing in the rendering thread was less accurate. The only
place where frame timing was accurate was in the encoder thread, and
the graphics thread was at the whim of thread scheduling. On higher
end computers it was typically fine, but it was just generally not
guaranteed that a frame would be rendered when it was supposed to be
rendered.
So the solution (originally proposed by r1ch and paibox) is to instead
keep the encoding and graphics threads separate as usual, but instead of
the encoder thread controlling the graphics thread, the graphics thread
now controls the encoder thread. The encoder thread keeps a limited
cache of frames, then the graphics thread copies frames in to the cache
and increments a semaphore to schedule the encoder thread to encode that
data.
In the cache, each frame has an encode counter. If the frame cache is
full (e.g., the encoder taking too long to return frames), it will not
cache a new frame, but instead will just increment the counter on the
last frame in the cache to schedule that frame to encode again, ensuring
that frames are on time and reducing CPU usage by lowering video
complexity. If the graphics thread takes too long to render a frame,
then it will add that frame with the count value set to the total amount
of frames that were missed (actual legitimately duplicated frames).
Because the cache gives many frames of breathing room for the encoder to
encode frames, this design helps improve results especially when using
encoding presets that have higher complexity and CPU usage, minimizing
the risk of needlessly skipped or duplicated frames.
I also managed to sneak in what should be a bit of an optimization to
reduce copying of frame data, though how much of an optimization it
ultimately ends up being is debatable.
So to sum it up, this commit increases accuracy of frame timing,
completely removes stray frame duplication, gives better results for
higher complexity encoding presets, and potentially optimizes the frame
pipeline a tiny bit.
In certain circumstances where the output was stopping, and where data
took a long enough time to send (such as when using an encoding preset
that causes high CPU usage), the output would sometimes still send data
even after it was stopped, typically causing the output to crash.
This changes the way source volume handles transitioning between being
active and inactive states.
The previous way that transitioning handled volume was that it set the
presentation volume of the source and all of its sub-sources to 0.0 if
the source was inactive, and 1.0 if active. Transition sources would
then also set the presentation volume for sub-sources to whatever their
transitioning volume was. However, the problem with this is that the
design didn't take in to account if the source or its sub-sources were
active anywhere else, so because of that it would break if that ever
happened, and I didn't realize that when I was designing it.
So instead, this completely overhauls the design of handling
transitioning volume. Each frame, it'll go through all sources and
check whether they're active or inactive and set the base volume
accordingly. If transitions are currently active, it will actually walk
the active source tree and check whether the source is in a
transitioning state somewhere.
- If the source is a sub-source of a transition, and it's not active
outside of the transition, then the transition will control the
volume of the source.
- If the source is a sub-source of a transition, but it's also active
outside of the transition, it'll defer to whichever is louder.
This also adds a new callback to the obs_source_info structure for
transition sources, get_transition_volume, which is called to get the
transitioning volume of a sub-source.
The reason to keep a reference counter for transitions is due to an
optimization I'm planning on when calculating transition volumes. I'm
planning on walking the source tree to be able to calculate the current
base volume of a source, but *only* if there are transitions active,
because the only time that the volume can be anything other than 1.0
or 0.0 is when there are active transitions, which may change the base
volume of a source.
Changed the design from using obs_source::enum_refs to just simply
preventing infinite source recursion in general, rather than allowing it
through the enum_refs variable. obs_source_add_child has been changed
so that it now returns a boolean, and if the function fails, it means
that the child cannot be added due to that potential recursion.
This adds bicubic and lanczos scaling capability to libobs to improve
scaling quality and sharpness when the output resolution has to be
scaled relative to the base resolution. Bilinear is also available,
although bilinear has rather poor quality and causes scaling to appear
blurry.
If the output resolution is close to the base resolution, then bilinear
is used instead as an optimization, as there's no need to use these
shaders if scaling is not in use.
The Bicubic and Lanczos effects are also exposed via exported function
to allow the ability to use those shaders in plugin modules if desired.
The API change adds a variable 'scale_type' to the obs_video_info
structure that allows the user interface to choose what type of scaling
filter should be used.
This was an important change because we were originally using an
hard-coded 709/partial range color matrix for the output, which was
causing problems for people wanting to use different formats or color
spaces. This will now automatically generate the color matrix depending
on the format, color space, and range, or use an identity matrix if the
video format is RGB instead of YUV.
This moves the 'flags' variable from the obs_source_frame structure to
the obs_source structure, and allows user flags to be set for a specific
source. Having it set on the obs_source_frame structure didn't make
much sense.
OBS_SOURCE_UNBUFFERED makes it so that the source does not buffer its
async video output in order to try to play it on time. In other words,
frames are played as soon as possible after being received.
Useful when you want a source to play back as quickly as possible
(webcams, certain types of capture devices)
This bug would happen if audio packets started being received before
video packets. It would erroneously cause audio packets to be
completely thrown away, and in certain cases would cause audio and video
to start way out of sync.
My original intention was "don't accept audio until video has started",
but instead mistakenly had the effect of "don't start audio until a
video packet has been received". This was originally was intended as a
way to handle outputs hooking in to active encoders and compensating
their existing timestamp information.
However, this made me realize that there was a major flaw in the design
for handling this, so I basically rewrote the entire thing.
Now, it does the following steps when inserting packets:
- Insert packets in to the interleaved packet array
- When both audio/video packets are received, prune packets up until the
point in which both audio/video start at the same time
- Resort the interleaved packet array
I have tested this code extensively and it appears to be working well,
regardless of whether or not the encoders were already active with
another output.
At the start of each render loop, it would get the timestamp, and then
it would then assign that timestamp to whatever frame was downloaded.
However, the frame that was downloaded was usually occurred a number of
frames ago, so it would assign the wrong timestamp value to that frame.
This fixes that issue by storing the timestamps in a circular buffer.
This Fixes a minor flaw with the API where data had to always be mutable
to be usable by the API.
Functions that do not modify the fundamental underlying data of a
structure should be marked as constant, both for safety and to signify
that the parameter is input only and will not be modified by the
function using it.
Typedef pointers are unsafe. If you do:
typedef struct bla *bla_t;
then you cannot use it as a constant, such as: const bla_t, because
that constant will be to the pointer itself rather than to the
underlying data. I admit this was a fundamental mistake that must
be corrected.
All typedefs that were pointer types will now have their pointers
removed from the type itself, and the pointers will be used when they
are actually used as variables/parameters/returns instead.
This does not break ABI though, which is pretty nice.
This prevents multiple needless calls to obs_source_get_frame and other
functions. If the texture has already been processed, then just render
it as-is in any subsequent calls to obs_source_video_render.
This is actually unnecessary now that there's a hard limit on the
maximum offset in which audio can be inserted.
This also assumes too much about the audio; it assumes audio is always
on, where as with some devices (such as the elgato) audio is not on
until the stream starts, and when the video has already incremented the
counter.
This makes it easier to do two things:
1.) Get the skipped frames count relative to each specific output
2.) Make it so that getting the 'current' log will always contain
information about skipped frames. Before, you'd have to force the
user to restart the program and get the last log, which was really
annoying when you just wanted to see how the encoders were
performing.
API functions added:
-----------------------------------------------
obs_output_set_preferred_size
obs_output_get_width
obs_output_get_height
obs_encoder_set_scaled_size
obs_encoder_get_width
obs_encoder_get_height
These functions allow for easier means of setting a custom resolution on
an output or encoder.
If an output uses an encoder and you set the preferred width/height
using the output, then the output will attempt to set the scaled
width/height for the encoder it's currently using.
Outputs and encoders now should use these functions to determine the
width/height of the raw frame data instead of using the video-io
functions.
With the recent change to module handling by BtbN, I felt that having
this information might be useful in case someone is actually using make
install to set up their libraries.
Changed API:
- char *obs_find_plugin_file(const char *sub_path);
Changed to: char *obs_module_file(const char *file);
Cahnge it so you no longer need to specify a sub-path such as:
obs_find_plugin_file("module_name/file.ext")
Instead, now automatically handle the module data path so all you need
to do is:
obs_module_file("file.ext")
- int obs_load_module(const char *name);
Changed to: int obs_open_module(obs_module_t *module,
const char *path,
const char *data_path);
bool obs_init_module(obs_module_t module);
Change the module loading API so that if the front-end chooses, it can
load modules directly from a specified path, and associate a data
directory with it on the spot.
The module will not be initialized immediately; obs_init_module must
be called on the module pointer in order to fully initialize the
module. This is done so a module can be disabled by the front-end if
the it so chooses.
New API:
- void obs_add_module_path(const char *bin, const char *data);
These functions allow you to specify new module search paths to add,
and allow you to search through them, or optionally just load all
modules from them. If the string %module% is included, it will
replace it with the module's name when that string is used as a
lookup. Data paths are now directly added to the module's internal
storage structure, and when obs_find_module_file is used, it will look
up the pointer to the obs_module structure and get its data directory
that way.
Example:
obs_add_module_path("/opt/obs/my-modules/%module%/bin",
"/opt/obs/my-modules/%module%/data");
This would cause it to additionally look for the binary of a
hypthetical module named "foo" at /opt/obs/my-modules/foo/bin/foo.so
(or libfoo.so), and then look for the data in
/opt/obs/my-modules/foo/data.
This gives the front-end more flexibility for handling third-party
plugin modules, or handling all plugin modules in a custom way.
- void obs_find_modules(obs_find_module_callback_t callback, void
*param);
This searches the existing paths for modules and calls the callback
function when any are found. Useful for plugin management and custom
handling of the paths by the front-end if desired.
- void obs_load_all_modules(void);
Search through the paths and both loads and initializes all modules
automatically without custom handling.
- void obs_enum_modules(obs_enum_module_callback_t callback,
void *param);
Enumerates currently opened modules.
Before it would assign the encoder/media callbacks directly to the
output's callbacks, so instead of doing that, it now goes through
intermediary functions for the sake of counting the frames.
This just ensures that if an obs object is renamed that the pointer to
older names will still be valid. Prevents renames from causing any
invalid memory access.
When the obs object is destroyed, so are the cached names.
The core itself now provides reconnection options (enabled by default, 2
second timeout between reconnects, 20 retries max until actual
disconnection occurs). This will make things easier for both module
developers and UI developers.
Reconnecting treats the stream as though it were still active, and
signals are sent when reconnecting and upon successful reconnection.
Need to implement user interface information for reconnections.
The module callback obs_module_set_locale will be called after loading
the module, and any time the locale is manually changed via core API.
When this function is called, the module is expected to load new text
lookup values for all the text it uses based upon the current locale.
This API is used to set the current locale for libobs, which it will set
for all modules when a module is loaded or specifically when the locale
is manually changed.
The 'initialize' callback is used before the encoders/output start up so
it can adjust encoder settings to required values if needed.
Also added the function 'obs_encoder_active' that returns true or false
depending on whether that encoder is active or not.
This replaces the older code which simply queried the max volume level
value for any given audio.
I'm still not 100% sure on if this is how I want to approach the
problem, particularly, whether this should be done in obs_source or in
audio_line, but it can always be moved later if needed.
This uses the calculations by the awesome Bill Hamilton that OBS1 used
for its volume levels. It calculates the current max (level),
magnitude, and current peak. This data then can be used to create
awesome volume meter controls later on.
NOTE: Will probably need optimization, does one float at a time right
now.
Also, change some of the naming conventions. I actually need to change
a lot of the naming conventions in general so that all words are
separated by underscores. Kind of a bad practice there on my part.
This uses the reverse planar YUV 4:2:0 conversion shader to output a YUV
texture without having to convert it via CPU. Again, this will reduce
video upload bandwidth usage to 37.5% of the original rate. I suspect
this will be particularly useful for when an FFmpeg or libav input
plugin for playing videos is made.
NOTE: There's an issue with certain texture sizes right now I haven't
been able to identify, if the full size of texture data divided by the
base texture width is an uneven number, the V chroma plane seems like it
can potentially shift, though I only had this happen with 160x90
resolution C920. Almost all resolutions tend to be even. Needs further
testing with more devices that support planar YUV 4:2:0 output.
If a source with async video wasn't currently active, it would endlessly
buffer the video data, which would cause memory to grow endlessly until
available memory was extinguished.
This really needs to be replaced with a proper caching mechanism at some
point.
The 'wait' constant was a terrible means of trying to ensure that the
packets were interleaved. Instead, calculate the current highest
timestamps of each encoder that's present in the interleaved buffer, and
use that as a means of detecting whether the current packet should be
sent off. This will guarantee sorting without relying on some arbirary
constant that 'assumes' that it'll be interleaved. It also reduces
buffering any more than what is needed to interleave.
- Updated the services API so that it links up with an output and
the output gets data from that service rather than via settings.
This allows the service context to have control over how an output is
used, and makes it so that the URL/key/etc isn't necessarily some
static setting.
Also, if the service is attached to an output, it will stick around
until the output is destroyed.
- The settings interface has been updated so that it can allow the
usage of service plugins. What this means is that now you can create
a service plugin that can control aspects of the stream, and it
allows each service to create their own user interface if they create
a service plugin module.
- Testing out saving of current service information. Saves/loads from
JSON in to obs_data_t, seems to be working quite nicely, and the
service object information is saved/preserved on exit, and loaded
again on startup.
- I agonized over the settings user interface for days, and eventually
I just decided that the only way that users weren't going to be
fumbling over options was to split up the settings in to simple/basic
output, pre-configured, and then advanced for advanced use (such as
multiple outputs or services, which I'll implement later).
This was particularly painful to really design right, I wanted more
features and wanted to include everything in one interface but
ultimately just realized from experience that users are just not
technically knowledgable about it and will end up fumbling with the
settings rather than getting things done.
Basically, what this means is that casual users only have to enter in
about 3 things to configure their stream: Stream key, audio bitrate,
and video bitrate. I am really happy with this interface for those
types of users, but it definitely won't be sufficient for advanced
usage or for custom outputs, so that stuff will have to be separated.
- Improved the JSON usage for the 'common streaming services' context,
I realized that JSON arrays are there to ensure sorting, while
forgetting that general items are optimized for hashing. So
basically I'm just using arrays now to sort items in it.
Add API for streaming services. The services API simplifies the
creation of custom service features and user interface.
Custom streaming services later on will be able to do things such as:
- Be able to use service-specific APIs via modules, allowing a more
direct means of communicating with the service and requesting or
setting service-specific information
- Get URL/stream key via other means of authentication such as OAuth,
or be able to build custom URLs for services that require that sort
of thing.
- Query information (such as viewer count, chat, follower
notifications, and other information)
- Set channel information (such as current game, current channel title,
activating commercials)
Also, I reduce some repeated code that was used for all libobs objects.
This includes the name of the object, the private data, settings, as
well as the signal and procedure handlers.
I also switched to using linked lists for the global object lists,
rather than using an array of pointers (you could say it was..
pointless.) ..Anyway, the linked list info is also stored in the shared
context data structure.
Before, async video sources would flicker because they were only being
drawn when they were updated. So when updated, they'd draw that frame,
then it would stop drawing it until it updated again. This fixes that
issue and they should now draw properly.
Also, fix a few other minor bugs and issues relating to async video,
and make it so that non-async video filters can be properly applied to
them.
For the purposes of testing, change the 'test-random' source to an async
video source that updates every quarter of a second with a new random
face.
Also fix a bug where non-async video sources wouldn't have filter
effects applied properly.
- Fix an issue that could occur when using more than one video encoder.
Audio/video would not sync up correctly because they were expected to
be paired with a particular encoder. This simply adds a little
helper variable to encoder packets that specifies the system time in
microseconds. We then use that system time to sync
- Fix an issue with x264 with fractional FPS rates (29.97 and 59.94
particularly) where it would create ridiculously large stream
outputs. The problem was that you shouldn't set the timebase_*
variables in the x264 params manually, let x264 handle the default
values for it and leave them at 0.
- Make x264 use CFR output, because there's no reason to ever use VFR
in this case.
- Implement the RTMP output module. This time around, we just use a
simple FLV muxer, then just write to the stream with RTMP_Write.
Easy and effective.
- Fix the FLV muxer, the muxer now outputs proper FLV packets.
- Output API:
* When using encoders, automatically interleave encoded packets
before sending it to the output.
* Pair encoders and have them automatically wait for the other to
start to ensure sync.
* Change 'obs_output_signal_start_fail' to 'obs_output_signal_stop'
because it was a bit confusing, and doing this makes a lot more
sense for outputs that need to stop suddenly (disconnections/etc).
- Encoder API:
* Remove some unnecessary encoder functions from the actual API and
make them internal. Most of the encoder functions are handled
automatically by outputs anyway, so there's no real need to expose
them and end up inadvertently confusing plugin writers.
* Have audio encoders wait for the video encoder to get a frame, then
start at the exact data point that the first video frame starts to
ensure the most accrate sync of video/audio possible.
* Add a required 'frame_size' callback for audio encoders that
returns the expected number of frames desired to encode with. This
way, the libobs encoder API can handle the circular buffering
internally automatically for the encoder modules, so encoder
writers don't have to do it themselves.
- Fix a few bugs in the serializer interface. It was passing the wrong
variable for the data in a few cases.
- If a source has video, make obs_source_update defer the actual update
callback until the tick function is called to prevent threading
issues.
- Add interleaving of video/audio packets for outputs that are encoded
and expect both video and audio data, sorting the packets and sending
them to the output when both video and audio is received.
- Combine create and initialize callbacks for the encoder API callback
interface.
Improve the properties API so that it can actually respond somewhat to
user input. Maybe later this might be further improved or replaced with
something script-based.
When creating a property, you can now add a callback to that property
that notifies when the property has been changed in the user interface.
Return true if you want the properties to be refreshed, or false if not.
Though now that I think about it I doubt there would ever be a case
where you would have this callback and *not* refresh the properties.
Regardless, this allows functions to change the values of properties or
settings, or enable/disable/hide other property controls from view
dynamically.
- Make it so that encoders can be assigned to outputs. If an encoder
is destroyed, it will automatically remove itself from that output.
I specifically didn't want to do reference counting because it leaves
too much potential for unchecked references and it just felt like it
would be more trouble than it's worth.
- Add a 'flags' value to the output definition structure. This lets
the output specify if it uses video/audio, and whether the output is
meant to be used with OBS encoders or not.
- Remove boilerplate code for outputs. This makes it easier to program
outputs. The boilerplate code involved before was mostly just
involving connecting to the audio/video data streams directly in each
output plugin.
Instead of doing that, simply add plugin callback functions for
receiving video/audio (either encoded or non-encoded, whichever it's
set to use), and then call obs_output_begin_data_capture and
obs_output_end_data_capture to automatically handle setting up
connections to raw or encoded video/audio streams for the plugin.
- Remove 'active' function from output callbacks, as it's no longer
really needed now that the libobs output context automatically knows
when the output is active or not.
- Make it so that an encoder cannot be destroyed until all data
connections to the encoder have been removed.
- Change the 'start' and 'stop' functions in the encoder interface to
just an 'initialize' callback, which initializes the encoder.
- Make it so that the encoder must be initialized first before the data
stream can be started. The reason why initialization was separated
from starting the encoder stream was because we need to be able to
check that the settings used with the encoder *can* be used first.
This problem was especially annoying if you had both video/audio
encoding. Before, you'd have to check the return value from
obs_encoder_start, and if that second encoder fails, then you
basically had to stop the first encoder again, making for
unnecessary boilerplate code whenever starting up two encoders.
Also, rename atomic functions to be consistent with the rest of the
platform/threading functions, and move atomic functions to threading*
files rather than platform* files
- Implement OBS encoder interface. It was previously incomplete, but
now is reaching some level of completion, though probably should
still be considered preliminary.
I had originally implemented it so that encoders only have a 'reset'
function to reset their parameters, but I felt that having both a
'start' and 'stop' function would be useful.
Encoders are now assigned to a specific video/audio media output each
rather than implicitely assigned to the main obs video/audio
contexts. This allows separate encoder contexts that aren't
necessarily assigned to the main video/audio context (which is useful
for things such as recording specific sources). Will probably have
to do this for regular obs outputs as well.
When creating an encoder, you must now explicitely state whether that
encoder is an audio or video encoder.
Audio and video can optionally be automatically converted depending
on what the encoder specifies.
When something 'attaches' to an encoder, the first attachment starts
the encoder, and the encoder automatically attaches to the media
output context associated with it. Subsequent attachments won't have
the same effect, they will just start receiving the same encoder data
when the next keyframe plays (along with SEI if any). When detaching
from the encoder, the last detachment will fully stop the encoder and
detach the encoder from the media output context associated with the
encoder.
SEI must actually be exported separately; because new encoder
attachments may not always be at the beginning of the stream, the
first keyframe they get must have that SEI data in it. If the
encoder has SEI data, it needs only add one small function to simply
query that SEI data, and then that data will be handled automatically
by libobs for all subsequent encoder attachments.
- Implement x264 encoder plugin, move x264 files to separate plugin to
separate necessary dependencies.
- Change video/audio frame output structures to not use const
qualifiers to prevent issues with non-const function usage elsewhere.
This was an issue when writing the x264 encoder, as the x264 encoder
expects non-const frame data.
Change stagesurf_map to return a non-const data type to prevent this
as well.
- Change full range parameter of video scaler to be an enum rather than
boolean
Ensure that a source has a valid name. Duplicates aren't a big deal
internally, but sources without a name are probably something that
should be avoided. Made is so that if a source is programmatically
created without a name, it's assigned an index based name.
In the main basic-mode window, made it check to make sure the name was
valid as well.
- Add some temporary streaming code using FFmpeg. FFmpeg itself is not
very ideal for streaming; lack of direct control of the sockets and
no framedrop handling means that FFmpeg is definitely not something
you want to use without wrapper code. I'd prefer writing my own
network framework in this particular case just because you give away
so much control of the network interface. Wasted an entire day
trying to go through FFmpeg issues.
There's just no way FFmpeg should be used for real streaming (at
least without being patched or submitting some sort of patch, but I'm
sort of feeling "meh" on that idea)
I had to end up writing multiple threads just to handle both
connecting and writing, because av_interleaved_write_frame blocks
every call, stalling the main encoder thread, and thus also stalling
draw signals.
- Add some temporary user interface for streaming settings. This is
just temporary for the time being. It's in the outputs section of
the basic-mode settings
- Make it so that dynamic arrays do not free all their data when the
size just happens to be reduced to 0. This prevents constant
reallocation when an array keeps going from 1 item to 0 items. Also,
it was bad to become dependent upon that functionality. You must now
always explicitly call "free" on it to ensure the data is free, and
that's how it should be. Implicit functionality can lead to
confusion and maintainability issues.
Split off activate to activate and show callbacks, and split off
deactivate to deactivate and hide callbacks. Sources didn't previously
have a means to know whether it was actually being displayed in the main
view or just happened to be visible somewhere. Now, for things like
transition sources, they have a means of knowing when they have actually
been "activated" so they can initiate their sequence.
A source is now only considered "active" when it's being displayed by
the main view. When a source is shown in the main view, the activate
callback/signal is triggered. When it's no longer being displayed by
the main view, deactivate callback/signal is triggered.
When a source is just generally visible to see by any view, the show
callback/signal is triggered. If it's no longer visible by any views,
then the hide callback/signal is triggered.
Presentation volume will now only be active when a source is active in
the main view rather than also in auxilary views.
Also fix a potential bug where parents wouldn't properly increment or
decrement all the activation references of a child source when a child
was added or removed.
- Remove obs_source::type because it became redundant now that the
type is always stored in the obs_source::info variable.
- Apply presentation volumes of 1.0 and 0.0 to sources when they
activate/deactivate, respectively. It also applies that presentation
volume to all sub-sources, with exception of transition sources.
Transition sources must apply presentation volume manually to their
sub-sources with the new transition functions below.
- Add a "transition_volume" variable to obs_source structure, and add
three functions for handling volume for transitions:
* obs_transition_begin_frame
* obs_source_set_transition_vol
* obs_transition_end_frame
Because the to/from targets of a transition source might both contain
some of the same sources, handling the transitioning of volumes for
that specific situation becomes an issue.
So for transitions, instead of modifying the presentation volumes
directly for both sets of sources, we do this:
- First, call obs_transition_begin_frame at the beginning of each
transition frame, which will reset transition volumes for all
sub-sources to 0. Presentation volumes remain unchanged.
- Call obs_source_set_transition_vol on each sub-source, which will
then add the volume to the transition volume for each source in
that source's tree. Presentation volumes still remain unchanged.
- Then you call obs_trandition_end_frame when complete, which will
then finally set the presentation volumes to the transition
volumes.
For example, let's say that there's one source that's within both the
"transitioning from" sources and "transition to" sources. It would
add both the fade in and fade out volumes to that source, and then
when the frame is complete, it would set the presentation volume to
the sum of those two values, rather than set the presentation volume
for that same source twice which would cause weird volume jittering
and also set the wrong values.
Now sources will be properly activated and deactivated when they are in
use or not in use.
Had to figure out a way to handle child sources, and children of
children, just ended up implementing simple functions that parents use
to signal adding/removal to help with hierarchial activation and
deactivation of child sources.
To prevent the source activate/deactivate callbacks from being called
more than once, added an activation reference counter. The first
increment will call the activate callback, and the last decrement will
call the deactivate callback.
Added "source-activate" and "source-deactivate" signals to the main obs
signal handler, and "activate" and "deactivate" to individual source
signal handlers.
Also, fixed the main window so it properly selects a source when the
current active scene has been changed.
Added a "master" volume for the entire audio subsystem.
Also, added a "presentation" volume for both the master volume and for
each invidiaul source. The presentation volume is used to control
things like transitioning volumes, preventing sources from outputting
any audio when they're inactive, as well as some other uses in the
future.
- Changed glMapBuffer to glMapBufferRange to allow invalidation. Using
just glMapBuffer alone was causing some unacceptable stalls.
- Changed dynamic buffers from GL_DYNAMIC_WRITE to GL_STREAM_WRITE
because I had misunderstood the OpenGL specification
- Added _OPENGL and _D3D11 builtin preprocessor macros to effects to
allow special processing if needed
- Added fmod support to shaders (NOTE: D3D and GL do not function
identically with negative numbers when using this. Positive numbers
however function identically)
- Created a planar conversion shader that converts from packed YUV to
planar 420 right on the GPU without any CPU processing. Reduces
required GPU download size to approximately 37.5% of its normal rate
as well. GPU usage down by 10 entire percentage points despite the
extra required pass.
Originally, the rendering system was designed to only display sources
and such, but I realized there would be a flaw; if you wanted to render
the main viewport in a custom way, or maybe even the entire application
as a graphics-based front end, you wouldn't have been able to do that.
Displays have now been separated in to viewports and displays. A
viewport is used to store and draw sources, a display is used to handle
draw callbacks. You can even use displays without using viewports to
draw custom render displays containing graphics calls if you wish, but
usually they would be used in combination with source viewports at
least.
This requires a tiny bit more work to create simple source displays, but
in the end its worth it for the added flexibility and options it brings.
The API used to be designed in such a way to where it would expect
exports for each individual source/output/encoder/etc. You would export
functions for each and it would automatically load those functions based
on a specific naming scheme from the module.
The idea behind this was that I wanted to limit the usage of structures
in the API so only functions could be used. It was an interesting idea
in theory, but this idea turned out to be flawed in a number of ways:
1.) Requiring exports to create sources/outputs/encoders/etc meant that
you could not create them by any other means, which meant that
things like faruton's .net plugin would become difficult.
2.) Export function declarations could not be checked, therefore if you
created a function with the wrong parameters and parameter types,
the compiler wouldn't know how to check for that.
3.) Required overly complex load functions in libobs just to handle it.
It makes much more sense to just have a load function that you call
manually. Complexity is the bane of all good programs.
4.) It required that you have functions of specific names, which looked
and felt somewhat unsightly.
So, to fix these issues, I replaced it with a more commonly used API
scheme, seen commonly in places like kernels and typical C libraries
with abstraction. You simply create a structure that contains the
callback definitions, and you pass it to a function to register that
definition (such as obs_register_source), which you call in the
obs_module_load of the module.
It will also automatically check the structure size and ensure that it
only loads the required values if the structure happened to add new
values in an API change.
The "main" source file for each module must include obs-module.h, and
must use OBS_DECLARE_MODULE() within that source file.
Also, started writing some doxygen documentation in to the main library
headers. Will add more detailed documentation as I go.
- Fill in the rest of the FFmpeg test output code for testing so it
actually properly outputs data.
- Improve the main video subsystem to be a bit more optimal and
automatically output I420 or NV12 if needed.
- Fix audio subsystem insertation and byte calculation. Now it will
seamlessly insert new audio data in to the audio stream based upon
its timestamp value. (Be extremely cautious when using floating
point calculations for important things like this, and always round
your values and check your values)
- Use 32 byte alignment in case of future optimizations and export a
function to get the current alignment.
- Make os_sleepto_ns return true if slept, false if the time has
already been passed before the call.
- Fix sinewave output so that it actually properly calculates a middle
C sinewave.
- Change the use of row_bytes to linesize (also makes it a bit more
consistent with FFmpeg's naming as well)
- Implement texture scaling/conversion/downloading for the main view so
we can finally start getting data to output.
Also, redesign how it works a bit, it will now properly wait one full
frame for each step in the process: rendering the main texture,
scaling the main texture to an output texture, staging/downloading the
ouput texture, and then outputting that staged data. This way, the
GPU will have more than enough time to fully complete each step.
- Fix a bug with OpenGL plugin's texture staging function. Was using
glBindBuffer instead of what should have been used: glBindTexture.
- Change the naming scheme of the variables in default.effect. It's now
named with the idea of just "color matrix" in mind instead of "yuv
matrix", and instead of DrawRGBToYUV, it's now just DrawMatrix.
Reduce and simplify the UI export interface. Having to export functions
with designated names was a bit silly for this case, it makes more sense
for inputs/outputs/etc because they have more functions associated with
them, but in this case the callback can be retrieved simply through the
enumeration exports. Makes it a bit easier and a little less awkward
for this situation.
Also, changed the exports and names to be a bit more consistent,
labelling them both as either "modal" or "modeless", and changed the UI
function calls to obs_exec_ui and obs_create_ui to imply modal/modeless
functionality a bit more.
I realized that I had intended modeless UI to be usable by plugins, but
it had been pointed out to me that modeless really needs to return a
pointer/handle to the user interface object that was created.
Add the ability to be able to call and use toolkit-specific or
program-specific user interface in modules.
User interface code can be either bundled with the module, or 'split'
out in to separate libraries (recommended).
There are three reasons why splitting is recommended:
1.) It allows plugins to be able to create custom user interface for
each toolkit if desired.
2.) Often, UI will be programmed in one language (the language of the
toolkit), and core logic may be programmed in another. This
allows plugins to keep the languages separated if necessary.
3.) It prevents direct linkage of UI toolkits libraries with core
module logic.
Splitting is not required, though is recommended if you want your plugin
to be more flexible with other user interface toolkits or programs.
Will implement a generic properties lookup next, which will be used for
automatic UI handling so that plugin UI isn't necessarily required.
I'm doing this because I might create another data structure called
obs_data for a different purpose. That and obs_program_data feels a bit
less vague for what it does.
jp9000
jpk
Palana