Allows the ability for manual transitioning to smoothly flow
(interpolate) to the intended transition point over a short period of
time rather than simply setting a hard transition point number. Doing
this allows manual transitioning to occur more smoothly, and in a more
visually pleasant way.
Allows the ability to manually specify the transitioning point so the
user can transition at a custom rate, usually done by a device that can
be used as a T-bar
Co-authored-by: Jim <obs.jim@gmail.com>
Adds functions to allow sources to inform the UI whether the audio is
currently active or not. Allows the ability to turn on/off the items in
the mixer.
Adds the "audio_line" internal source type as a bare source type for the
sole purpose of outputting audio, and the obs_source_info::audio_mix
callback which allows mixing of those audio lines, which is then treated
as normal audio for the source. Audio line objects should be added as
sub-sources when multiple audio lines from a single source are needed,
then mixed together with the audio_mix callback.
The difference between the new obs_source_info::audio_mix callback and
obs_source_info::audio_render is that obs_source_info::audio_mix (along
with the audio_line source) are only one track, and it outputs audio to
the source automatically via obs_source_output_audio() when the call
completes. This allows the mixed audio to be treated like a normal
source's audio, in that you can filter it, change its volume, or monitor
it.
This change was necessary because the CEF (used with the browser source)
outputs multiple audio streams at once to a single browser source, so
it's the program's responsibility to mix those streams together itself.
The shaders to unpack YUV information from the same texture were rather
complicated. Breaking them up into separate textures makes the shaders
much simpler, and we can remove the PRECISION_OFFSET hack.
Performance also gets a nice boost on Intel for planar textures.
Intel GPA, SetStablePowerState, Intel HD Graphics 530, 1920x1080
UYVY: 473 us -> 457 us
YUY2: 492 us -> 422 us
YVYU: 491 us -> 441 us
I420: 1637 us -> 505 us
I422: 1644 us -> 482 us
I444: 1653 us -> 504 us
NV12: 1656 us -> 369 us
Y800 (limited): 270 us -> 277 us
Y800 (full): 263 us -> 289 us
RGB (limited): 341 us -> 411 us
BGR3 (limited): 512 us -> 509 us
BGR3 (full): 527 us -> 534 us
The shaders to pack YUV information into the same texture were rather
complicated and suffering precision issues. Breaking them up into
separate textures makes the shaders much simpler and avoids having to
compute large integer offsets. Unfortunately, the code to handle
multiple textures is not as pleasant, but at least the NV12 rendering
path is no longer separate.
In addition, write chroma samples to "standard" offsets. For I444,
there's no difference, but I420/NV12 formats now have chroma shifted to
the left as 4:2:0 is shown in the H.264 specification.
Intel GPA, SetStablePowerState, Intel HD Graphics 530
Expect speed incrase:
I420: 844 us -> 493 us (254 us + 190 us + 274 us)
I444: 837 us -> 747 us (258 us + 276 us + 272 us)
NV12: 450 us -> 368 us (319 us + 168 us)
Expect no change:
NV12 (HW): 580 (481 us + 166 us) us -> 588 us (468 us + 247 us)
RGB: 359 us -> 387 us
Fixes https://obsproject.com/mantis/view.php?id=624
Fixes https://obsproject.com/mantis/view.php?id=1512
This implements pausing of outputs. To accomplish this, raw audio/video
data is halted to the encoders or raw output. Pausing is as precisely
timed as possible according to the timing of the obs_output_pause call,
and audio data will be spliced down to the exact audio sample in
accordance to that timing at the start/end marks.
Outputs that support this (outputs used for recording) can set the
OBS_OUTPUT_CAN_PAUSE capability flag.
If the audio subsystem was buffered to any extent, the audio of a raw
output would start off at a negative offset, requiring each raw output
to implement a "prepare_audio" function (as seen in the FFmpeg output)
in order to ensure proper synchronization with video. This did not
apply to encoded outputs because it was already being performed by the
obs-encoder code.
Code submissions have continually suffered from formatting
inconsistencies that constantly have to be addressed. Using
clang-format simplifies this by making code formatting more consistent,
and allows automation of the code formatting so that maintainers can
focus more on the code itself instead of code formatting.
Remove three instances of unnecessary double-buffering. They are not
needed to avoid stalls, and cause increased memory traffic when
measured on Intel HD 530, presumably because texture data will remain
in cache if sampled immediately after write.
(Note: GPU timings from Intel GPA are volatile.)
NV12, 3 Draws:
RGBA -> UYVX: 628 us -> 543 us
UYVX -> Y: 522 us -> 507 us
UYVX -> UV: 315 us -> 187 us
Total, Duration: 1594 us -> 1153 us
Total, GTI Read Throughput: 25.2 MB -> 15.9 MB
(This commit also modifies the UI, obs-ffmpeg, and obs-output modules)
Fixes a long-time regression where the program would lock up if an
encode call fails. Shuts down all outputs associated with the failing
encoder and displays an error message to the user.
Ideally, it would be best if a more detailed error could be displayed to
the user about the nature of the error, though the primary problem is
the encoder errors are typically not something the user would be able to
understand. The current message is a bit of a generic error message;
improvement is welcome.
Another suggestion is to try to have the encoder restart seamlessly,
though it would take a significant amount of work to be able to make it
do something like that properly, and it sort of assumes that encoder
failures are sporadic, which may not necessarily be the case with some
hardware encoders on some systems. It may be better just to use another
encoder in that case. For now, seamless restart is ruled out.
libobs: Add support for limited to full color range conversions when
using RGB or Y800 formats, and move RGB converison for Y800 formats to
the GPU.
decklink: Stop hiding color space/range properties for RGB formats, and
remove "YUV" from "YUV Color Space" and "YUV Color Range".
win-dshow: Remove "YUV" from "YUV Color Space" and "YUV Color Range".
UI: Remove "YUV" from "YUV Color Space" and "YUV Color Range".
Currently several shaders need "DrawMatrix" techniques to support the
possibility that the input texture is a "YUV" format. Also, "DrawMatrix"
is overloaded for translation in both directions when it is written for
RGB to "YUV" only.
A cleaner solution is to handle "YUV" to RGB up-front as part of format
conversion, and ensure only RGB inputs reach the other shaders. This is
necessary to someday perform correct scale filtering without the cost of
redundant "YUV" conversions per texture tap.
A necessary prerequisite for this is to add conversion support for
VIDEO_FORMAT_I444, and that is now in place. There was already a hack in
place to cover VIDEO_FORMAT_Y800. All other "YUV" formats already have
conversion functions.
"DrawMatrix" has been removed from shaders that only supported "YUV" to
RGB conversions. It still exists in shaders that perform RGB to "YUV"
conversions, and the implementations have been sanitized accordingly.
This new scale filter computes pixels by weighing the coverage area of
source pixels over the target pixel. This algorithm works well for both
upsampling and downsampling, but was mainly designed to upscale
high-quality low-resolution sources like RGB/HDMI retro consoles. I've
heard of people using odd workarounds like scaling up to very high
resolutions before scaling back down to preserve pixel shartpness. This
algorithm directly addresses this use-case in a much more direct
fashion.
The Area scale filter does a better job of preserving the thickness of
thin features than the Point filter.
The Area scale filter does not look at source pixels that lie outside
of the target pixel, leading to a much sharper image than Bilinear,
Bicubic, and Lanczos filters.
This filter should interpolate pixels in linear space, but OBS is not
equipped to do that at the moment.
libobs: Add GPU effect, and wire up scene serialization.
obs-filters: Add Area as an option for scale_filter.
UI: Add Area as an option for both scene items, and canvas downscaling.
Adds display_duration declaring the minimum duration a caption text
is not going to be overwritten by a new one. To keep the functions
backwards-compatible obs_output_output_caption_text2 was added while
obs_output_output_caption_text1 continues having a 2 second default.
Allows the ability to encode by passing NV12 textures. This uses a
separate thread for texture-based encoders with a small queue of
textures. An output texture with a keyed mutex shared texture is locked
between OBS and each encoder. A new encoder callback and capability
flag is used to encode with textures.
This splits the "do_encode" function in to "do_encode" and
"send_off_encoder_packet", the latter of which allows the ability for
texture-based encoders to manage their own encoding and just simply send
off a packet to the outputs.
Allows the ability for one encoder to defer to another in case of
failure or unsupported feature. Okay, fine, it's mostly a hack so the
new NVENC encoder can fall back to the FFmpeg encoder if NV12 textures
aren't in use, that way it does not have to implement raw fallback
support itself. The settings and properties are pretty much the same,
so there's no reason not to utilize it in order to save time that could
otherwise be spent more productively.
Reduces GPU usage when encoding is not active. Does not perform color
conversion, frame staging, or frame downloading unless encoding is
explicitly active.
Because it would be troublesome to add the ability to remove source
types (in case for example a script fails to reload), instead make it so
source types can be temporarily disabled while the program is running.
This is to prevent confusion with video_thread in
libobs/media-io/video-io.c, which is used exclusively for video
encoding/output. Also prevents confusion in the profiler log data.
Decoupling the audio from the video causes the audio to be played right
when it's received rather than attempt to sync up to the video frames.
This is useful with certain async sources/devices when the audio/video
timestamps are not reliable.
Naturally because it plays audio right when it's received, this should
only be used when the async source is operating in unbuffered mode,
otherwise the video frame timing will be out of sync by the amount of
buffering the video currently has.
This allows the ability for certain types of modules (particularly
scripting-related modules) to initialize extra data when all other
modules have loaded. Because front-ends may wish to have custom
handling for loading modules, the front-end must manually call
obs_post_load_modules after it has completed loading all plug-in
modules.
Closesjp9000/obs-studio#965
(This commit also modifies the decklink, linux-v4l2, mac-avcapture,
obs-ffmpeg, and win-dshow modules)
Originally, async buffering for sources was supposed to be a
user-controllable flag. However, that turned out to be less than ideal
because sources (such as the win-dshow plugin) were programmed with
automatic control over their buffering (such as automatically detecting
USB 2.0 capture devices and then enabling in those cases).
The fact that it was a flag caused a design flaw to where buffering
values would be overwritten when a source is loaded from save data.
Because of that, this flag is being deprecated and replaced with a
specific function to enable unbuffered mode instead.
Originally, obs_get_video_info would recreate the obs_video_info
structure that was originally passed to it from obs_reset_video. This
changes that to just store a copy of the obs_video_info when calling
obs_reset_video, and then copying that to the parameter of
obs_get_video_info when called.
When frames are skipped the skipped frame count would increment, but the
total frame count would not increment, causing the percentage
calculation to fail.
Additionally, the skipped frames log reporting has been moved to
media-io/video-io.c instead of each output.
Adds functions to turn on audio monitoring to allow the user to hear
playback of an audio source over the user's speaker. It can be set to
turn off monitoring and only output to stream, or it can be set to
output only to monitoring, or it can be set to both.
On windows, audio monitoring uses WASAPI. Windows also is capable of
syncing the audio to the video according to when the video frame itself
was played.
On mac, it uses AudioQueue.
On linux, it's not currently implemented and won't do anything (to be
implemented).
Because D3D11 specifically does not support an L8 texture format (you
have to use a shader swizzle), manually convert Y800 signals to RGBX
instead. This also fixes a bug where Y800 signals will render red.
Closesjp9000/obs-studio#718
For displays, instead of using the draw_callbacks_mutex and risk a
reverse mutual lock scenario, use a separate mutex to lock display size
data.
This bug was exposed when trying to reorder filters in the UI module.
The UI thread would try to reorder the filters, locking the filter mutex
of the source, and then the reorder would signal the UI to resize the
display, so the display would lock its draw_callbacks_mutex. Then, in
the graphics thread, it would lock the display's draw_callbacks_mutex,
try to draw the source, and then the source would try to lock that same
filter mutex.
A mutex trace:
UI thread -> lock source filter mutex -> waiting on display mutex
graphics thread -> lock display mutex -> waiting on source filter mutex
Closesjp9000/obs-studio#714
jp9000
Clayton Groeneveld
jpark37
James Park
derrod
VodBox
Zachary Lund
SammyJames
Richard Stanway