jp9000
/
obs-studio
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
obs-studio/libobs/media-io/video-io.c

589 lines
14 KiB
C
Raw Normal View History

first commit
2013-09-30 19:37:13 -07:00
/******************************************************************************
Copyright (C) 2013 by Hugh Bailey <obs.jim@gmail.com>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
With the permission of my fellow contributors, I'm switching obs-studio back to GPL v2+ to prevent issues between this project and the original OBS project, and for personal reasons to avoid legal ambiguity (not political reasons, I admittedly would prefer GPL v3+)
2013-12-02 21:24:38 -08:00
the Free Software Foundation, either version 2 of the License, or
first commit
2013-09-30 19:37:13 -07:00
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
******************************************************************************/
#include <assert.h>
libobs: Fix skipped frames reporting When frames are skipped the skipped frame count would increment, but the total frame count would not increment, causing the percentage calculation to fail. Additionally, the skipped frames log reporting has been moved to media-io/video-io.c instead of each output.
2017-04-24 03:15:46 -07:00
#include <inttypes.h>
first commit
2013-09-30 19:37:13 -07:00
#include "../util/bmem.h"
#include "../util/platform.h"
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
#include "../util/profiler.h"
first commit
2013-09-30 19:37:13 -07:00
#include "../util/threading.h"
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
#include "../util/darray.h"
libobs: Implement and use better scaling function for 64-bit integers As os_gettime_ns() gets large the current scaling methods, mostly by casting to uint64_t, may lead to numerical overflows. Sweep the code and use util_mul_div64() where applicable. Signed-off-by: Hans Petter Selasky <hps@selasky.org>
2020-03-21 02:55:12 -07:00
#include "../util/util_uint64.h"
first commit
2013-09-30 19:37:13 -07:00
Add planar audio support, improve test output - Add planar audio support. FFmpeg and libav use planar audio for many encoders, so it was somewhat necessary to add support in libobs itself. - Improve/adjust FFmpeg test output plugin. The exports were somewhat messed up (making me rethink how exports should be done). Not yet functional; it handles video properly, but it still does not handle audio properly. - Improve planar video code. The planar video code was not properly accounting for row sizes for each plane. Specifying row sizes for each plane has now been added. This will also make it more compatible with FFmpeg/libav. - Fixed a bug where callbacks wouldn't create properly in audio-io and video-io code. - Implement 'blogva' function to allow for va_list usage with libobs logging.
2014-02-07 02:03:54 -08:00
#include "format-conversion.h"
first commit
2013-09-30 19:37:13 -07:00
#include "video-io.h"
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
#include "video-frame.h"
#include "video-scaler.h"
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
extern profiler_name_store_t *obs_get_profiler_name_store(void);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
#define MAX_CONVERT_BUFFERS 3
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
#define MAX_CACHE_SIZE 16
struct cached_frame_info {
struct video_data frame;
libobs: Don't allow lagged frames to be counted as skipped Prevents lagged frames (frames that took too long to render) to be counted as skipped frames (frames that are skipped due to encoding taking too long to process)
2017-04-28 00:43:05 -07:00
int skipped;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
int count;
};
first commit
2013-09-30 19:37:13 -07:00
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
struct video_input {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
struct video_scale_info conversion;
video_scaler_t *scaler;
struct video_frame frame[MAX_CONVERT_BUFFERS];
int cur_frame;
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
Implement encoder interface (still preliminary) - Implement OBS encoder interface. It was previously incomplete, but now is reaching some level of completion, though probably should still be considered preliminary. I had originally implemented it so that encoders only have a 'reset' function to reset their parameters, but I felt that having both a 'start' and 'stop' function would be useful. Encoders are now assigned to a specific video/audio media output each rather than implicitely assigned to the main obs video/audio contexts. This allows separate encoder contexts that aren't necessarily assigned to the main video/audio context (which is useful for things such as recording specific sources). Will probably have to do this for regular obs outputs as well. When creating an encoder, you must now explicitely state whether that encoder is an audio or video encoder. Audio and video can optionally be automatically converted depending on what the encoder specifies. When something 'attaches' to an encoder, the first attachment starts the encoder, and the encoder automatically attaches to the media output context associated with it. Subsequent attachments won't have the same effect, they will just start receiving the same encoder data when the next keyframe plays (along with SEI if any). When detaching from the encoder, the last detachment will fully stop the encoder and detach the encoder from the media output context associated with the encoder. SEI must actually be exported separately; because new encoder attachments may not always be at the beginning of the stream, the first keyframe they get must have that SEI data in it. If the encoder has SEI data, it needs only add one small function to simply query that SEI data, and then that data will be handled automatically by libobs for all subsequent encoder attachments. - Implement x264 encoder plugin, move x264 files to separate plugin to separate necessary dependencies. - Change video/audio frame output structures to not use const qualifiers to prevent issues with non-const function usage elsewhere. This was an issue when writing the x264 encoder, as the x264 encoder expects non-const frame data. Change stagesurf_map to return a non-const data type to prevent this as well. - Change full range parameter of video scaler to be an enum rather than boolean
2014-03-16 16:21:34 -07:00
void (*callback)(void *param, struct video_data *frame);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
void *param;
};
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
static inline void video_input_free(struct video_input *input)
{
for (size_t i = 0; i < MAX_CONVERT_BUFFERS; i++)
video_frame_free(&input->frame[i]);
video_scaler_destroy(input->scaler);
}
first commit
2013-09-30 19:37:13 -07:00
struct video_output {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
struct video_output_info info;
first commit
2013-09-30 19:37:13 -07:00
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
pthread_t thread;
pthread_mutex_t data_mutex;
bool stop;
Add planar audio support, improve test output - Add planar audio support. FFmpeg and libav use planar audio for many encoders, so it was somewhat necessary to add support in libobs itself. - Improve/adjust FFmpeg test output plugin. The exports were somewhat messed up (making me rethink how exports should be done). Not yet functional; it handles video properly, but it still does not handle audio properly. - Improve planar video code. The planar video code was not properly accounting for row sizes for each plane. Specifying row sizes for each plane has now been added. This will also make it more compatible with FFmpeg/libav. - Fixed a bug where callbacks wouldn't create properly in audio-io and video-io code. - Implement 'blogva' function to allow for va_list usage with libobs logging.
2014-02-07 02:03:54 -08:00
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
os_sem_t *update_semaphore;
uint64_t frame_time;
volatile long skipped_frames;
volatile long total_frames;
first commit
2013-09-30 19:37:13 -07:00
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
bool initialized;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
pthread_mutex_t input_mutex;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
DARRAY(struct video_input) inputs;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
size_t available_frames;
size_t first_added;
size_t last_added;
struct cached_frame_info cache[MAX_CACHE_SIZE];
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
volatile bool raw_active;
volatile long gpu_refs;
first commit
2013-09-30 19:37:13 -07:00
};
/* ------------------------------------------------------------------------- */
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
static inline bool scale_video_output(struct video_input *input,
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
struct video_data *data)
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
{
bool success = true;
if (input->scaler) {
struct video_frame *frame;
if (++input->cur_frame == MAX_CONVERT_BUFFERS)
input->cur_frame = 0;
frame = &input->frame[input->cur_frame];
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
success = video_scaler_scale(input->scaler, frame->data,
frame->linesize,
(const uint8_t *const *)data->data,
data->linesize);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
if (success) {
for (size_t i = 0; i < MAX_AV_PLANES; i++) {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
data->data[i] = frame->data[i];
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
data->linesize[i] = frame->linesize[i];
}
Fix bug with automatic video scaling This is sort of hard to explain: the scale_video_output function was overwriting the current frame. If scaling was disabled, it would do nothing, and return success, and all would be well. If it was enabled, it would then call the scaler, and then replace the contents of the 'data' function parameter with the scaled frame data. The problem with this is that I was passing video_output::cur_frame directly, which overwrites its previous value with the scaled frame data. Then if cur_frame was not updated on time, it would end up trying to scale the previously scaled image, if that makes sense. it would call the video scaler with the same from for both the source and destination. So the simple fix was to simply use a local variable and pass that in as a parameter to prevent this bug from occurring.
2014-08-10 12:49:44 -07:00
} else {
blog(LOG_WARNING, "video-io: Could not scale frame!");
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
}
}
return success;
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
static inline bool video_output_cur_frame(struct video_output *video)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
{
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct cached_frame_info *frame_info;
bool complete;
libobs: Don't allow lagged frames to be counted as skipped Prevents lagged frames (frames that took too long to render) to be counted as skipped frames (frames that are skipped due to encoding taking too long to process)
2017-04-28 00:43:05 -07:00
bool skipped;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
/* -------------------------------- */
pthread_mutex_lock(&video->data_mutex);
frame_info = &video->cache[video->first_added];
pthread_mutex_unlock(&video->data_mutex);
/* -------------------------------- */
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_lock(&video->input_mutex);
for (size_t i = 0; i < video->inputs.num; i++) {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
struct video_input *input = video->inputs.array + i;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct video_data frame = frame_info->frame;
media-io: Fix repeating video timestamps In video-io.c, video frames could skip, but what would happen is the frame's timestamp would repeat for the next frame, giving the next frame a non-monotonic timestamp, and then jump. This could mess up syncing slightly when the frame is finally given to an outputs.
2014-10-22 15:11:08 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (scale_video_output(input, &frame))
Fix bug with automatic video scaling This is sort of hard to explain: the scale_video_output function was overwriting the current frame. If scaling was disabled, it would do nothing, and return success, and all would be well. If it was enabled, it would then call the scaler, and then replace the contents of the 'data' function parameter with the scaled frame data. The problem with this is that I was passing video_output::cur_frame directly, which overwrites its previous value with the scaled frame data. Then if cur_frame was not updated on time, it would end up trying to scale the previously scaled image, if that makes sense. it would call the video scaler with the same from for both the source and destination. So the simple fix was to simply use a local variable and pass that in as a parameter to prevent this bug from occurring.
2014-08-10 12:49:44 -07:00
input->callback(input->param, &frame);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
pthread_mutex_unlock(&video->input_mutex);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
/* -------------------------------- */
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
pthread_mutex_lock(&video->data_mutex);
frame_info->frame.timestamp += video->frame_time;
complete = --frame_info->count == 0;
libobs: Don't allow lagged frames to be counted as skipped Prevents lagged frames (frames that took too long to render) to be counted as skipped frames (frames that are skipped due to encoding taking too long to process)
2017-04-28 00:43:05 -07:00
skipped = frame_info->skipped > 0;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (complete) {
if (++video->first_added == video->info.cache_size)
video->first_added = 0;
if (++video->available_frames == video->info.cache_size)
video->last_added = video->first_added;
libobs: Don't allow lagged frames to be counted as skipped Prevents lagged frames (frames that took too long to render) to be counted as skipped frames (frames that are skipped due to encoding taking too long to process)
2017-04-28 00:43:05 -07:00
} else if (skipped) {
--frame_info->skipped;
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
os_atomic_inc_long(&video->skipped_frames);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
}
pthread_mutex_unlock(&video->data_mutex);
/* -------------------------------- */
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
return complete;
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
}
first commit
2013-09-30 19:37:13 -07:00
static void *video_thread(void *param)
{
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct video_output *video = param;
first commit
2013-09-30 19:37:13 -07:00
Set various thread names Helps identify which threads are which when debugging
2015-01-02 05:36:09 -08:00
os_set_thread_name("video-io: video thread");
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
const char *video_thread_name =
profile_store_name(obs_get_profiler_name_store(),
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
"video_thread(%s)", video->info.name);
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
while (os_sem_wait(video->update_semaphore) == 0) {
if (video->stop)
break;
Reimplement monitor capture - Implement windows monitor capture (code is so much cleaner than in OBS1). Will implement duplication capture later - Add GDI texture support to d3d11 graphics library - Fix precision issue with sleep timing, you have to call timeBeginPeriod otherwise windows sleep will be totally erratic.
2014-03-05 09:43:14 -08:00
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
profile_start(video_thread_name);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
while (!video->stop && !video_output_cur_frame(video)) {
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
os_atomic_inc_long(&video->total_frames);
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
}
first commit
2013-09-30 19:37:13 -07:00
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
os_atomic_inc_long(&video->total_frames);
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
profile_end(video_thread_name);
profile_reenable_thread();
first commit
2013-09-30 19:37:13 -07:00
}
return NULL;
}
/* ------------------------------------------------------------------------- */
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
static inline bool valid_video_params(const struct video_output_info *info)
first commit
2013-09-30 19:37:13 -07:00
{
return info->height != 0 && info->width != 0 && info->fps_den != 0 &&
finish up most of the source audio stuff and rename some variables/structs/enum to be a bit more consistent
2013-10-31 10:28:47 -07:00
info->fps_num != 0;
first commit
2013-09-30 19:37:13 -07:00
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
static inline void init_cache(struct video_output *video)
{
if (video->info.cache_size > MAX_CACHE_SIZE)
video->info.cache_size = MAX_CACHE_SIZE;
for (size_t i = 0; i < video->info.cache_size; i++) {
struct video_frame *frame;
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
frame = (struct video_frame *)&video->cache[i];
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
video_frame_init(frame, video->info.format, video->info.width,
video->info.height);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
}
video->available_frames = video->info.cache_size;
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
int video_output_open(video_t **video, struct video_output_info *info)
first commit
2013-09-30 19:37:13 -07:00
{
struct video_output *out;
if (!valid_video_params(info))
return VIDEO_OUTPUT_INVALIDPARAM;
Use bzalloc instead of bmalloc then memset Reduces needless code repetition and still allows for proper memory alignment. Cleans up the code a bit.
2014-02-09 11:34:07 -08:00
out = bzalloc(sizeof(struct video_output));
Add NULL checks and assertions to fix clang static analysis problems Also remove an unused variable from obs-encoder.c (via clang static analysis)
2014-04-14 13:55:14 -07:00
if (!out)
libobs/media-io: Fix leaks and error handling
2021-08-23 22:02:17 -07:00
goto fail0;
first commit
2013-09-30 19:37:13 -07:00
Add preliminary ffmpeg plugin (still testing) - Added some code for FFmpeg output that I'm still playing around with. Right now I'm just trying to get it to output to file and try to understand the FFmpeg/libav APIs. Hopefully in the future this plugin can be used for any sort of output to FFmpeg. - Fixed a cast warning in audio-io.c with size_t -> uint32_t - Renamed the 'video_info' and 'audio_info' structures to 'video_conver_info' and 'audio_convert_info' to better represent their actual purpose, and to avoid confusion with 'audio_output_info' and 'video_output_info' structures. - Removed a few macros from obs-def.h that were at one point going to be used but no longer going to be used (at least for now)
2014-01-19 02:16:41 -08:00
memcpy(&out->info, info, sizeof(struct video_output_info));
libobs: Implement and use better scaling function for 64-bit integers As os_gettime_ns() gets large the current scaling methods, mostly by casting to uint64_t, may lead to numerical overflows. Sweep the code and use util_mul_div64() where applicable. Signed-off-by: Hans Petter Selasky <hps@selasky.org>
2020-03-21 02:55:12 -07:00
out->frame_time =
util_mul_div64(1000000000ULL, info->fps_den, info->fps_num);
first commit
2013-09-30 19:37:13 -07:00
out->initialized = false;
libobs/media-io: Fix leaks and error handling
2021-08-23 22:02:17 -07:00
if (pthread_mutex_init_recursive(&out->data_mutex) != 0)
goto fail0;
if (pthread_mutex_init_recursive(&out->input_mutex) != 0)
goto fail1;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (os_sem_init(&out->update_semaphore, 0) != 0)
libobs/media-io: Fix leaks and error handling
2021-08-23 22:02:17 -07:00
goto fail2;
first commit
2013-09-30 19:37:13 -07:00
if (pthread_create(&out->thread, NULL, video_thread, out) != 0)
libobs/media-io: Fix leaks and error handling
2021-08-23 22:02:17 -07:00
goto fail3;
first commit
2013-09-30 19:37:13 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
init_cache(out);
first commit
2013-09-30 19:37:13 -07:00
out->initialized = true;
*video = out;
return VIDEO_OUTPUT_SUCCESS;
libobs/media-io: Fix leaks and error handling
2021-08-23 22:02:17 -07:00
fail3:
os_sem_destroy(out->update_semaphore);
fail2:
pthread_mutex_destroy(&out->input_mutex);
fail1:
pthread_mutex_destroy(&out->data_mutex);
fail0:
first commit
2013-09-30 19:37:13 -07:00
video_output_close(out);
return VIDEO_OUTPUT_FAIL;
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
void video_output_close(video_t *video)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
{
if (!video)
return;
video_output_stop(video);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
for (size_t i = 0; i < video->inputs.num; i++)
video_input_free(&video->inputs.array[i]);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
da_free(video->inputs);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
for (size_t i = 0; i < video->info.cache_size; i++)
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
video_frame_free((struct video_frame *)&video->cache[i]);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
bfree(video);
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
static size_t video_get_input_idx(const video_t *video,
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
void (*callback)(void *param,
struct video_data *frame),
void *param)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
{
for (size_t i = 0; i < video->inputs.num; i++) {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
struct video_input *input = video->inputs.array + i;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
if (input->callback == callback && input->param == param)
return i;
}
return DARRAY_INVALID;
}
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
static inline bool video_input_init(struct video_input *input,
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
struct video_output *video)
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
{
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
if (input->conversion.width != video->info.width ||
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
input->conversion.height != video->info.height ||
input->conversion.format != video->info.format) {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
struct video_scale_info from = {.format = video->info.format,
.width = video->info.width,
.height = video->info.height,
.range = video->info.range,
.colorspace =
video->info.colorspace};
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
int ret = video_scaler_create(&input->scaler,
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
&input->conversion, &from,
VIDEO_SCALE_FAST_BILINEAR);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
if (ret != VIDEO_SCALER_SUCCESS) {
if (ret == VIDEO_SCALER_BAD_CONVERSION)
Be more consistent about log levels LOG_ERROR should be used in places where though recoverable (or at least something that can be handled safely), was unexpected, and may affect the user/application. LOG_WARNING should be used in places where it's not entirely unexpected, is recoverable, and doesn't really affect the user/application.
2014-02-28 19:02:29 -08:00
blog(LOG_ERROR, "video_input_init: Bad "
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
"scale conversion type");
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
else
Be more consistent about log levels LOG_ERROR should be used in places where though recoverable (or at least something that can be handled safely), was unexpected, and may affect the user/application. LOG_WARNING should be used in places where it's not entirely unexpected, is recoverable, and doesn't really affect the user/application.
2014-02-28 19:02:29 -08:00
blog(LOG_ERROR, "video_input_init: Failed to "
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
"create scaler");
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
return false;
}
for (size_t i = 0; i < MAX_CONVERT_BUFFERS; i++)
video_frame_init(&input->frame[i],
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
input->conversion.format,
input->conversion.width,
input->conversion.height);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
}
return true;
}
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
static inline void reset_frames(video_t *video)
{
os_atomic_set_long(&video->skipped_frames, 0);
os_atomic_set_long(&video->total_frames, 0);
}
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
bool video_output_connect(
video_t *video, const struct video_scale_info *conversion,
void (*callback)(void *param, struct video_data *frame), void *param)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
{
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
bool success = false;
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
if (!video || !callback)
return false;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_lock(&video->input_mutex);
Add planar audio support, improve test output - Add planar audio support. FFmpeg and libav use planar audio for many encoders, so it was somewhat necessary to add support in libobs itself. - Improve/adjust FFmpeg test output plugin. The exports were somewhat messed up (making me rethink how exports should be done). Not yet functional; it handles video properly, but it still does not handle audio properly. - Improve planar video code. The planar video code was not properly accounting for row sizes for each plane. Specifying row sizes for each plane has now been added. This will also make it more compatible with FFmpeg/libav. - Fixed a bug where callbacks wouldn't create properly in audio-io and video-io code. - Implement 'blogva' function to allow for va_list usage with libobs logging.
2014-02-07 02:03:54 -08:00
if (video_get_input_idx(video, callback, param) == DARRAY_INVALID) {
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
struct video_input input;
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
memset(&input, 0, sizeof(input));
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
input.callback = callback;
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
input.param = param;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
Add preliminary ffmpeg plugin (still testing) - Added some code for FFmpeg output that I'm still playing around with. Right now I'm just trying to get it to output to file and try to understand the FFmpeg/libav APIs. Hopefully in the future this plugin can be used for any sort of output to FFmpeg. - Fixed a cast warning in audio-io.c with size_t -> uint32_t - Renamed the 'video_info' and 'audio_info' structures to 'video_conver_info' and 'audio_convert_info' to better represent their actual purpose, and to avoid confusion with 'audio_output_info' and 'video_output_info' structures. - Removed a few macros from obs-def.h that were at one point going to be used but no longer going to be used (at least for now)
2014-01-19 02:16:41 -08:00
if (conversion) {
input.conversion = *conversion;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
} else {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
input.conversion.format = video->info.format;
input.conversion.width = video->info.width;
input.conversion.height = video->info.height;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
if (input.conversion.width == 0)
input.conversion.width = video->info.width;
if (input.conversion.height == 0)
input.conversion.height = video->info.height;
success = video_input_init(&input, video);
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
if (success) {
if (video->inputs.num == 0) {
if (!os_atomic_load_long(&video->gpu_refs)) {
reset_frames(video);
}
os_atomic_set_bool(&video->raw_active, true);
}
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
da_push_back(video->inputs, &input);
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
}
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
pthread_mutex_unlock(&video->input_mutex);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
return success;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
static void log_skipped(video_t *video)
{
long skipped = os_atomic_load_long(&video->skipped_frames);
double percentage_skipped =
(double)skipped /
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
(double)os_atomic_load_long(&video->total_frames) * 100.0;
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
if (skipped)
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
blog(LOG_INFO,
"Video stopped, number of "
"skipped frames due "
"to encoding lag: "
"%ld/%ld (%0.1f%%)",
video->skipped_frames, video->total_frames,
percentage_skipped);
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
void video_output_disconnect(video_t *video,
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
void (*callback)(void *param,
struct video_data *frame),
void *param)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
{
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
if (!video || !callback)
return;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_lock(&video->input_mutex);
size_t idx = video_get_input_idx(video, callback, param);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
if (idx != DARRAY_INVALID) {
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
video_input_free(video->inputs.array + idx);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
da_erase(video->inputs, idx);
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
if (video->inputs.num == 0) {
os_atomic_set_bool(&video->raw_active, false);
if (!os_atomic_load_long(&video->gpu_refs)) {
log_skipped(video);
}
}
libobs: Fix skipped frames reporting When frames are skipped the skipped frame count would increment, but the total frame count would not increment, causing the percentage calculation to fail. Additionally, the skipped frames log reporting has been moved to media-io/video-io.c instead of each output.
2017-04-24 03:15:46 -07:00
}
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_unlock(&video->input_mutex);
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
bool video_output_active(const video_t *video)
Fix video reset and apply new video settings This allows the changing of bideo settings without having to completely reset all graphics data. Will recreate internal output/conversion buffers and such and reset the main preview.
2014-02-22 19:14:19 -08:00
{
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
if (!video)
return false;
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
return os_atomic_load_bool(&video->raw_active);
Fix video reset and apply new video settings This allows the changing of bideo settings without having to completely reset all graphics data. Will recreate internal output/conversion buffers and such and reset the main preview.
2014-02-22 19:14:19 -08:00
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
const struct video_output_info *video_output_get_info(const video_t *video)
added functions to get current audio/video output information
2013-12-06 05:38:19 -08:00
{
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
return video ? &video->info : NULL;
added functions to get current audio/video output information
2013-12-06 05:38:19 -08:00
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
bool video_output_lock_frame(video_t *video, struct video_frame *frame,
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
int count, uint64_t timestamp)
first commit
2013-09-30 19:37:13 -07:00
{
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct cached_frame_info *cfi;
bool locked;
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
if (!video)
return false;
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
first commit
2013-09-30 19:37:13 -07:00
pthread_mutex_lock(&video->data_mutex);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (video->available_frames == 0) {
video->cache[video->last_added].count += count;
libobs: Don't allow lagged frames to be counted as skipped Prevents lagged frames (frames that took too long to render) to be counted as skipped frames (frames that are skipped due to encoding taking too long to process)
2017-04-28 00:43:05 -07:00
video->cache[video->last_added].skipped += count;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
locked = false;
} else {
if (video->available_frames != video->info.cache_size) {
if (++video->last_added == video->info.cache_size)
video->last_added = 0;
}
cfi = &video->cache[video->last_added];
cfi->frame.timestamp = timestamp;
cfi->count = count;
libobs: Don't allow lagged frames to be counted as skipped Prevents lagged frames (frames that took too long to render) to be counted as skipped frames (frames that are skipped due to encoding taking too long to process)
2017-04-28 00:43:05 -07:00
cfi->skipped = 0;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
memcpy(frame, &cfi->frame, sizeof(*frame));
locked = true;
}
first commit
2013-09-30 19:37:13 -07:00
pthread_mutex_unlock(&video->data_mutex);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
return locked;
first commit
2013-09-30 19:37:13 -07:00
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
void video_output_unlock_frame(video_t *video)
first commit
2013-09-30 19:37:13 -07:00
{
clang-format: Apply formatting Code submissions have continually suffered from formatting inconsistencies that constantly have to be addressed. Using clang-format simplifies this by making code formatting more consistent, and allows automation of the code formatting so that maintainers can focus more on the code itself instead of code formatting.
2019-06-22 22:13:45 -07:00
if (!video)
return;
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
pthread_mutex_lock(&video->data_mutex);
video->available_frames--;
os_sem_post(video->update_semaphore);
pthread_mutex_unlock(&video->data_mutex);
first commit
2013-09-30 19:37:13 -07:00
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint64_t video_output_get_frame_time(const video_t *video)
first commit
2013-09-30 19:37:13 -07:00
{
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
return video ? video->frame_time : 0;
first commit
2013-09-30 19:37:13 -07:00
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
void video_output_stop(video_t *video)
first commit
2013-09-30 19:37:13 -07:00
{
void *thread_ret;
if (!video)
return;
if (video->initialized) {
Fix destructor video outputs. This fixes a crash on Linux that supposedly occurs when pthread_join is called multiple times.
2014-04-25 16:39:43 -07:00
video->initialized = false;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
video->stop = true;
os_sem_post(video->update_semaphore);
first commit
2013-09-30 19:37:13 -07:00
pthread_join(video->thread, &thread_ret);
libobs/media-io: Fix leaks and error handling
2021-08-23 22:02:17 -07:00
os_sem_destroy(video->update_semaphore);
pthread_mutex_destroy(&video->data_mutex);
pthread_mutex_destroy(&video->input_mutex);
first commit
2013-09-30 19:37:13 -07:00
}
}
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
bool video_output_stopped(video_t *video)
{
if (!video)
return true;
return video->stop;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
enum video_format video_output_get_format(const video_t *video)
video-io: Add function to just get video format
2014-08-10 16:39:36 -07:00
{
return video ? video->info.format : VIDEO_FORMAT_NONE;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_width(const video_t *video)
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
{
return video ? video->info.width : 0;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_height(const video_t *video)
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
{
return video ? video->info.height : 0;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
double video_output_get_frame_rate(const video_t *video)
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
{
if (!video)
return 0.0;
return (double)video->info.fps_num / (double)video->info.fps_den;
}
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_skipped_frames(const video_t *video)
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
{
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
return (uint32_t)os_atomic_load_long(&video->skipped_frames);
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
}
media-io: Add function for total video frames
2014-07-27 09:25:19 -07:00
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_total_frames(const video_t *video)
media-io: Add function for total video frames
2014-07-27 09:25:19 -07:00
{
libobs/media-io: Add frame funcs for separate GPU thread Normally, the total and skipped frame count for the encoder is performed in the video-io thread. However, because a new thread is being introduced for texture-based encoding, the frontend has no way of being able to query that. So, instead of making the frontend query that data separately, just make the texture encoder thread increment the values of video-io. That way, the frontend doesn't need to change any code, and can continue using the same functions for determining the total/skipped frame count. This can cause the frame count to be doubled if both a texture-based encoder and a raw data encoder is active at the same time, but it's an acceptable alternative.
2019-02-05 17:16:40 -08:00
return (uint32_t)os_atomic_load_long(&video->total_frames);
}
/* Note: These four functions below are a very slight bit of a hack. If the
* texture encoder thread is active while the raw encoder thread is active, the
* total frame count will just be doubled while they're both active. Which is
* fine. What's more important is having a relatively accurate skipped frame
* count. */
void video_output_inc_texture_encoders(video_t *video)
{
if (os_atomic_inc_long(&video->gpu_refs) == 1 &&
!os_atomic_load_bool(&video->raw_active)) {
reset_frames(video);
}
}
void video_output_dec_texture_encoders(video_t *video)
{
if (os_atomic_dec_long(&video->gpu_refs) == 0 &&
!os_atomic_load_bool(&video->raw_active)) {
log_skipped(video);
}
}
void video_output_inc_texture_frames(video_t *video)
{
os_atomic_inc_long(&video->total_frames);
}
void video_output_inc_texture_skipped_frames(video_t *video)
{
os_atomic_inc_long(&video->skipped_frames);
media-io: Add function for total video frames
2014-07-27 09:25:19 -07:00
}