jp9000
/
obs-studio
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
obs-studio/libobs/media-io/video-io.c

502 lines
12 KiB
C
Raw Normal View History

first commit
2013-09-30 19:37:13 -07:00
/******************************************************************************
Copyright (C) 2013 by Hugh Bailey <obs.jim@gmail.com>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
With the permission of my fellow contributors, I'm switching obs-studio back to GPL v2+ to prevent issues between this project and the original OBS project, and for personal reasons to avoid legal ambiguity (not political reasons, I admittedly would prefer GPL v3+)
2013-12-02 21:24:38 -08:00
the Free Software Foundation, either version 2 of the License, or
first commit
2013-09-30 19:37:13 -07:00
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
******************************************************************************/
#include <assert.h>
#include "../util/bmem.h"
#include "../util/platform.h"
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
#include "../util/profiler.h"
first commit
2013-09-30 19:37:13 -07:00
#include "../util/threading.h"
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
#include "../util/darray.h"
first commit
2013-09-30 19:37:13 -07:00
Add planar audio support, improve test output - Add planar audio support. FFmpeg and libav use planar audio for many encoders, so it was somewhat necessary to add support in libobs itself. - Improve/adjust FFmpeg test output plugin. The exports were somewhat messed up (making me rethink how exports should be done). Not yet functional; it handles video properly, but it still does not handle audio properly. - Improve planar video code. The planar video code was not properly accounting for row sizes for each plane. Specifying row sizes for each plane has now been added. This will also make it more compatible with FFmpeg/libav. - Fixed a bug where callbacks wouldn't create properly in audio-io and video-io code. - Implement 'blogva' function to allow for va_list usage with libobs logging.
2014-02-07 02:03:54 -08:00
#include "format-conversion.h"
first commit
2013-09-30 19:37:13 -07:00
#include "video-io.h"
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
#include "video-frame.h"
#include "video-scaler.h"
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
extern profiler_name_store_t *obs_get_profiler_name_store(void);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
#define MAX_CONVERT_BUFFERS 3
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
#define MAX_CACHE_SIZE 16
struct cached_frame_info {
struct video_data frame;
int count;
};
first commit
2013-09-30 19:37:13 -07:00
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
struct video_input {
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
struct video_scale_info conversion;
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
video_scaler_t *scaler;
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
struct video_frame frame[MAX_CONVERT_BUFFERS];
int cur_frame;
Implement encoder interface (still preliminary) - Implement OBS encoder interface. It was previously incomplete, but now is reaching some level of completion, though probably should still be considered preliminary. I had originally implemented it so that encoders only have a 'reset' function to reset their parameters, but I felt that having both a 'start' and 'stop' function would be useful. Encoders are now assigned to a specific video/audio media output each rather than implicitely assigned to the main obs video/audio contexts. This allows separate encoder contexts that aren't necessarily assigned to the main video/audio context (which is useful for things such as recording specific sources). Will probably have to do this for regular obs outputs as well. When creating an encoder, you must now explicitely state whether that encoder is an audio or video encoder. Audio and video can optionally be automatically converted depending on what the encoder specifies. When something 'attaches' to an encoder, the first attachment starts the encoder, and the encoder automatically attaches to the media output context associated with it. Subsequent attachments won't have the same effect, they will just start receiving the same encoder data when the next keyframe plays (along with SEI if any). When detaching from the encoder, the last detachment will fully stop the encoder and detach the encoder from the media output context associated with the encoder. SEI must actually be exported separately; because new encoder attachments may not always be at the beginning of the stream, the first keyframe they get must have that SEI data in it. If the encoder has SEI data, it needs only add one small function to simply query that SEI data, and then that data will be handled automatically by libobs for all subsequent encoder attachments. - Implement x264 encoder plugin, move x264 files to separate plugin to separate necessary dependencies. - Change video/audio frame output structures to not use const qualifiers to prevent issues with non-const function usage elsewhere. This was an issue when writing the x264 encoder, as the x264 encoder expects non-const frame data. Change stagesurf_map to return a non-const data type to prevent this as well. - Change full range parameter of video scaler to be an enum rather than boolean
2014-03-16 16:21:34 -07:00
void (*callback)(void *param, struct video_data *frame);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
void *param;
};
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
static inline void video_input_free(struct video_input *input)
{
for (size_t i = 0; i < MAX_CONVERT_BUFFERS; i++)
video_frame_free(&input->frame[i]);
video_scaler_destroy(input->scaler);
}
first commit
2013-09-30 19:37:13 -07:00
struct video_output {
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
struct video_output_info info;
first commit
2013-09-30 19:37:13 -07:00
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_t thread;
pthread_mutex_t data_mutex;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
bool stop;
Add planar audio support, improve test output - Add planar audio support. FFmpeg and libav use planar audio for many encoders, so it was somewhat necessary to add support in libobs itself. - Improve/adjust FFmpeg test output plugin. The exports were somewhat messed up (making me rethink how exports should be done). Not yet functional; it handles video properly, but it still does not handle audio properly. - Improve planar video code. The planar video code was not properly accounting for row sizes for each plane. Specifying row sizes for each plane has now been added. This will also make it more compatible with FFmpeg/libav. - Fixed a bug where callbacks wouldn't create properly in audio-io and video-io code. - Implement 'blogva' function to allow for va_list usage with libobs logging.
2014-02-07 02:03:54 -08:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
os_sem_t *update_semaphore;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
uint64_t frame_time;
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
uint32_t skipped_frames;
media-io: Add function for total video frames
2014-07-27 09:25:19 -07:00
uint32_t total_frames;
first commit
2013-09-30 19:37:13 -07:00
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
bool initialized;
pthread_mutex_t input_mutex;
DARRAY(struct video_input) inputs;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
size_t available_frames;
size_t first_added;
size_t last_added;
struct cached_frame_info cache[MAX_CACHE_SIZE];
first commit
2013-09-30 19:37:13 -07:00
};
/* ------------------------------------------------------------------------- */
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
static inline bool scale_video_output(struct video_input *input,
struct video_data *data)
{
bool success = true;
if (input->scaler) {
struct video_frame *frame;
if (++input->cur_frame == MAX_CONVERT_BUFFERS)
input->cur_frame = 0;
frame = &input->frame[input->cur_frame];
success = video_scaler_scale(input->scaler,
frame->data, frame->linesize,
Implement encoder interface (still preliminary) - Implement OBS encoder interface. It was previously incomplete, but now is reaching some level of completion, though probably should still be considered preliminary. I had originally implemented it so that encoders only have a 'reset' function to reset their parameters, but I felt that having both a 'start' and 'stop' function would be useful. Encoders are now assigned to a specific video/audio media output each rather than implicitely assigned to the main obs video/audio contexts. This allows separate encoder contexts that aren't necessarily assigned to the main video/audio context (which is useful for things such as recording specific sources). Will probably have to do this for regular obs outputs as well. When creating an encoder, you must now explicitely state whether that encoder is an audio or video encoder. Audio and video can optionally be automatically converted depending on what the encoder specifies. When something 'attaches' to an encoder, the first attachment starts the encoder, and the encoder automatically attaches to the media output context associated with it. Subsequent attachments won't have the same effect, they will just start receiving the same encoder data when the next keyframe plays (along with SEI if any). When detaching from the encoder, the last detachment will fully stop the encoder and detach the encoder from the media output context associated with the encoder. SEI must actually be exported separately; because new encoder attachments may not always be at the beginning of the stream, the first keyframe they get must have that SEI data in it. If the encoder has SEI data, it needs only add one small function to simply query that SEI data, and then that data will be handled automatically by libobs for all subsequent encoder attachments. - Implement x264 encoder plugin, move x264 files to separate plugin to separate necessary dependencies. - Change video/audio frame output structures to not use const qualifiers to prevent issues with non-const function usage elsewhere. This was an issue when writing the x264 encoder, as the x264 encoder expects non-const frame data. Change stagesurf_map to return a non-const data type to prevent this as well. - Change full range parameter of video scaler to be an enum rather than boolean
2014-03-16 16:21:34 -07:00
(const uint8_t * const*)data->data,
data->linesize);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
if (success) {
for (size_t i = 0; i < MAX_AV_PLANES; i++) {
data->data[i] = frame->data[i];
data->linesize[i] = frame->linesize[i];
}
Fix bug with automatic video scaling This is sort of hard to explain: the scale_video_output function was overwriting the current frame. If scaling was disabled, it would do nothing, and return success, and all would be well. If it was enabled, it would then call the scaler, and then replace the contents of the 'data' function parameter with the scaled frame data. The problem with this is that I was passing video_output::cur_frame directly, which overwrites its previous value with the scaled frame data. Then if cur_frame was not updated on time, it would end up trying to scale the previously scaled image, if that makes sense. it would call the video scaler with the same from for both the source and destination. So the simple fix was to simply use a local variable and pass that in as a parameter to prevent this bug from occurring.
2014-08-10 12:49:44 -07:00
} else {
blog(LOG_WARNING, "video-io: Could not scale frame!");
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
}
}
return success;
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
static inline bool video_output_cur_frame(struct video_output *video)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
{
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct cached_frame_info *frame_info;
bool complete;
/* -------------------------------- */
pthread_mutex_lock(&video->data_mutex);
frame_info = &video->cache[video->first_added];
pthread_mutex_unlock(&video->data_mutex);
/* -------------------------------- */
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_lock(&video->input_mutex);
for (size_t i = 0; i < video->inputs.num; i++) {
struct video_input *input = video->inputs.array+i;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct video_data frame = frame_info->frame;
media-io: Fix repeating video timestamps In video-io.c, video frames could skip, but what would happen is the frame's timestamp would repeat for the next frame, giving the next frame a non-monotonic timestamp, and then jump. This could mess up syncing slightly when the frame is finally given to an outputs.
2014-10-22 15:11:08 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (scale_video_output(input, &frame))
Fix bug with automatic video scaling This is sort of hard to explain: the scale_video_output function was overwriting the current frame. If scaling was disabled, it would do nothing, and return success, and all would be well. If it was enabled, it would then call the scaler, and then replace the contents of the 'data' function parameter with the scaled frame data. The problem with this is that I was passing video_output::cur_frame directly, which overwrites its previous value with the scaled frame data. Then if cur_frame was not updated on time, it would end up trying to scale the previously scaled image, if that makes sense. it would call the video scaler with the same from for both the source and destination. So the simple fix was to simply use a local variable and pass that in as a parameter to prevent this bug from occurring.
2014-08-10 12:49:44 -07:00
input->callback(input->param, &frame);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
pthread_mutex_unlock(&video->input_mutex);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
/* -------------------------------- */
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
pthread_mutex_lock(&video->data_mutex);
frame_info->frame.timestamp += video->frame_time;
complete = --frame_info->count == 0;
if (complete) {
if (++video->first_added == video->info.cache_size)
video->first_added = 0;
if (++video->available_frames == video->info.cache_size)
video->last_added = video->first_added;
}
pthread_mutex_unlock(&video->data_mutex);
/* -------------------------------- */
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
return complete;
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
}
first commit
2013-09-30 19:37:13 -07:00
static void *video_thread(void *param)
{
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct video_output *video = param;
first commit
2013-09-30 19:37:13 -07:00
Set various thread names Helps identify which threads are which when debugging
2015-01-02 05:36:09 -08:00
os_set_thread_name("video-io: video thread");
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
const char *video_thread_name =
profile_store_name(obs_get_profiler_name_store(),
"video_thread(%s)", video->info.name);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
while (os_sem_wait(video->update_semaphore) == 0) {
if (video->stop)
break;
Reimplement monitor capture - Implement windows monitor capture (code is so much cleaner than in OBS1). Will implement duplication capture later - Add GDI texture support to d3d11 graphics library - Fix precision issue with sleep timing, you have to call timeBeginPeriod otherwise windows sleep will be totally erratic.
2014-03-05 09:43:14 -08:00
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
profile_start(video_thread_name);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
while (!video->stop && !video_output_cur_frame(video)) {
video->total_frames++;
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
}
first commit
2013-09-30 19:37:13 -07:00
media-io: Add function for total video frames
2014-07-27 09:25:19 -07:00
video->total_frames++;
libobs/media-io: Profile audio/video_thread
2015-07-11 00:14:34 -07:00
profile_end(video_thread_name);
profile_reenable_thread();
first commit
2013-09-30 19:37:13 -07:00
}
return NULL;
}
/* ------------------------------------------------------------------------- */
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
static inline bool valid_video_params(const struct video_output_info *info)
first commit
2013-09-30 19:37:13 -07:00
{
return info->height != 0 && info->width != 0 && info->fps_den != 0 &&
finish up most of the source audio stuff and rename some variables/structs/enum to be a bit more consistent
2013-10-31 10:28:47 -07:00
info->fps_num != 0;
first commit
2013-09-30 19:37:13 -07:00
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
static inline void init_cache(struct video_output *video)
{
if (video->info.cache_size > MAX_CACHE_SIZE)
video->info.cache_size = MAX_CACHE_SIZE;
for (size_t i = 0; i < video->info.cache_size; i++) {
struct video_frame *frame;
frame = (struct video_frame*)&video->cache[i];
video_frame_init(frame, video->info.format,
video->info.width, video->info.height);
}
video->available_frames = video->info.cache_size;
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
int video_output_open(video_t **video, struct video_output_info *info)
first commit
2013-09-30 19:37:13 -07:00
{
struct video_output *out;
libobs/media-io: Fix recursive lock in video In certain cases the video thread could end up trying to lock itself recursively. This just allows the mutexes to safely be locked recursively.
2014-12-18 11:39:50 -08:00
pthread_mutexattr_t attr;
first commit
2013-09-30 19:37:13 -07:00
if (!valid_video_params(info))
return VIDEO_OUTPUT_INVALIDPARAM;
Use bzalloc instead of bmalloc then memset Reduces needless code repetition and still allows for proper memory alignment. Cleans up the code a bit.
2014-02-09 11:34:07 -08:00
out = bzalloc(sizeof(struct video_output));
Add NULL checks and assertions to fix clang static analysis problems Also remove an unused variable from obs-encoder.c (via clang static analysis)
2014-04-14 13:55:14 -07:00
if (!out)
goto fail;
first commit
2013-09-30 19:37:13 -07:00
Add preliminary ffmpeg plugin (still testing) - Added some code for FFmpeg output that I'm still playing around with. Right now I'm just trying to get it to output to file and try to understand the FFmpeg/libav APIs. Hopefully in the future this plugin can be used for any sort of output to FFmpeg. - Fixed a cast warning in audio-io.c with size_t -> uint32_t - Renamed the 'video_info' and 'audio_info' structures to 'video_conver_info' and 'audio_convert_info' to better represent their actual purpose, and to avoid confusion with 'audio_output_info' and 'video_output_info' structures. - Removed a few macros from obs-def.h that were at one point going to be used but no longer going to be used (at least for now)
2014-01-19 02:16:41 -08:00
memcpy(&out->info, info, sizeof(struct video_output_info));
first commit
2013-09-30 19:37:13 -07:00
out->frame_time = (uint64_t)(1000000000.0 * (double)info->fps_den /
(double)info->fps_num);
out->initialized = false;
libobs/media-io: Fix recursive lock in video In certain cases the video thread could end up trying to lock itself recursively. This just allows the mutexes to safely be locked recursively.
2014-12-18 11:39:50 -08:00
if (pthread_mutexattr_init(&attr) != 0)
first commit
2013-09-30 19:37:13 -07:00
goto fail;
libobs/media-io: Fix recursive lock in video In certain cases the video thread could end up trying to lock itself recursively. This just allows the mutexes to safely be locked recursively.
2014-12-18 11:39:50 -08:00
if (pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE) != 0)
goto fail;
if (pthread_mutex_init(&out->data_mutex, &attr) != 0)
goto fail;
if (pthread_mutex_init(&out->input_mutex, &attr) != 0)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
goto fail;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (os_sem_init(&out->update_semaphore, 0) != 0)
first commit
2013-09-30 19:37:13 -07:00
goto fail;
if (pthread_create(&out->thread, NULL, video_thread, out) != 0)
goto fail;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
init_cache(out);
first commit
2013-09-30 19:37:13 -07:00
out->initialized = true;
*video = out;
return VIDEO_OUTPUT_SUCCESS;
fail:
video_output_close(out);
return VIDEO_OUTPUT_FAIL;
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
void video_output_close(video_t *video)
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
{
if (!video)
return;
video_output_stop(video);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
for (size_t i = 0; i < video->inputs.num; i++)
video_input_free(&video->inputs.array[i]);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
da_free(video->inputs);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
for (size_t i = 0; i < video->info.cache_size; i++)
video_frame_free((struct video_frame*)&video->cache[i]);
os_sem_destroy(video->update_semaphore);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_destroy(&video->data_mutex);
pthread_mutex_destroy(&video->input_mutex);
bfree(video);
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
static size_t video_get_input_idx(const video_t *video,
Implement encoder interface (still preliminary) - Implement OBS encoder interface. It was previously incomplete, but now is reaching some level of completion, though probably should still be considered preliminary. I had originally implemented it so that encoders only have a 'reset' function to reset their parameters, but I felt that having both a 'start' and 'stop' function would be useful. Encoders are now assigned to a specific video/audio media output each rather than implicitely assigned to the main obs video/audio contexts. This allows separate encoder contexts that aren't necessarily assigned to the main video/audio context (which is useful for things such as recording specific sources). Will probably have to do this for regular obs outputs as well. When creating an encoder, you must now explicitely state whether that encoder is an audio or video encoder. Audio and video can optionally be automatically converted depending on what the encoder specifies. When something 'attaches' to an encoder, the first attachment starts the encoder, and the encoder automatically attaches to the media output context associated with it. Subsequent attachments won't have the same effect, they will just start receiving the same encoder data when the next keyframe plays (along with SEI if any). When detaching from the encoder, the last detachment will fully stop the encoder and detach the encoder from the media output context associated with the encoder. SEI must actually be exported separately; because new encoder attachments may not always be at the beginning of the stream, the first keyframe they get must have that SEI data in it. If the encoder has SEI data, it needs only add one small function to simply query that SEI data, and then that data will be handled automatically by libobs for all subsequent encoder attachments. - Implement x264 encoder plugin, move x264 files to separate plugin to separate necessary dependencies. - Change video/audio frame output structures to not use const qualifiers to prevent issues with non-const function usage elsewhere. This was an issue when writing the x264 encoder, as the x264 encoder expects non-const frame data. Change stagesurf_map to return a non-const data type to prevent this as well. - Change full range parameter of video scaler to be an enum rather than boolean
2014-03-16 16:21:34 -07:00
void (*callback)(void *param, struct video_data *frame),
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
void *param)
{
for (size_t i = 0; i < video->inputs.num; i++) {
struct video_input *input = video->inputs.array+i;
if (input->callback == callback && input->param == param)
return i;
}
return DARRAY_INVALID;
}
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
static inline bool video_input_init(struct video_input *input,
struct video_output *video)
{
if (input->conversion.width != video->info.width ||
input->conversion.height != video->info.height ||
input->conversion.format != video->info.format) {
struct video_scale_info from = {
.format = video->info.format,
.width = video->info.width,
.height = video->info.height,
};
int ret = video_scaler_create(&input->scaler,
&input->conversion, &from,
VIDEO_SCALE_FAST_BILINEAR);
if (ret != VIDEO_SCALER_SUCCESS) {
if (ret == VIDEO_SCALER_BAD_CONVERSION)
Be more consistent about log levels LOG_ERROR should be used in places where though recoverable (or at least something that can be handled safely), was unexpected, and may affect the user/application. LOG_WARNING should be used in places where it's not entirely unexpected, is recoverable, and doesn't really affect the user/application.
2014-02-28 19:02:29 -08:00
blog(LOG_ERROR, "video_input_init: Bad "
"scale conversion type");
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
else
Be more consistent about log levels LOG_ERROR should be used in places where though recoverable (or at least something that can be handled safely), was unexpected, and may affect the user/application. LOG_WARNING should be used in places where it's not entirely unexpected, is recoverable, and doesn't really affect the user/application.
2014-02-28 19:02:29 -08:00
blog(LOG_ERROR, "video_input_init: Failed to "
"create scaler");
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
return false;
}
for (size_t i = 0; i < MAX_CONVERT_BUFFERS; i++)
video_frame_init(&input->frame[i],
Fix bug with automatic video scaling This is sort of hard to explain: the scale_video_output function was overwriting the current frame. If scaling was disabled, it would do nothing, and return success, and all would be well. If it was enabled, it would then call the scaler, and then replace the contents of the 'data' function parameter with the scaled frame data. The problem with this is that I was passing video_output::cur_frame directly, which overwrites its previous value with the scaled frame data. Then if cur_frame was not updated on time, it would end up trying to scale the previously scaled image, if that makes sense. it would call the video scaler with the same from for both the source and destination. So the simple fix was to simply use a local variable and pass that in as a parameter to prevent this bug from occurring.
2014-08-10 12:49:44 -07:00
input->conversion.format,
input->conversion.width,
input->conversion.height);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
}
return true;
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
bool video_output_connect(video_t *video,
Use MP4s when not on windows Also, make it use 'veryfast' preset. Still testing this, might have to revise this later.
2014-02-27 22:14:03 -08:00
const struct video_scale_info *conversion,
Implement encoder interface (still preliminary) - Implement OBS encoder interface. It was previously incomplete, but now is reaching some level of completion, though probably should still be considered preliminary. I had originally implemented it so that encoders only have a 'reset' function to reset their parameters, but I felt that having both a 'start' and 'stop' function would be useful. Encoders are now assigned to a specific video/audio media output each rather than implicitely assigned to the main obs video/audio contexts. This allows separate encoder contexts that aren't necessarily assigned to the main video/audio context (which is useful for things such as recording specific sources). Will probably have to do this for regular obs outputs as well. When creating an encoder, you must now explicitely state whether that encoder is an audio or video encoder. Audio and video can optionally be automatically converted depending on what the encoder specifies. When something 'attaches' to an encoder, the first attachment starts the encoder, and the encoder automatically attaches to the media output context associated with it. Subsequent attachments won't have the same effect, they will just start receiving the same encoder data when the next keyframe plays (along with SEI if any). When detaching from the encoder, the last detachment will fully stop the encoder and detach the encoder from the media output context associated with the encoder. SEI must actually be exported separately; because new encoder attachments may not always be at the beginning of the stream, the first keyframe they get must have that SEI data in it. If the encoder has SEI data, it needs only add one small function to simply query that SEI data, and then that data will be handled automatically by libobs for all subsequent encoder attachments. - Implement x264 encoder plugin, move x264 files to separate plugin to separate necessary dependencies. - Change video/audio frame output structures to not use const qualifiers to prevent issues with non-const function usage elsewhere. This was an issue when writing the x264 encoder, as the x264 encoder expects non-const frame data. Change stagesurf_map to return a non-const data type to prevent this as well. - Change full range parameter of video scaler to be an enum rather than boolean
2014-03-16 16:21:34 -07:00
void (*callback)(void *param, struct video_data *frame),
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
void *param)
{
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
bool success = false;
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
if (!video || !callback)
return false;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_lock(&video->input_mutex);
Add planar audio support, improve test output - Add planar audio support. FFmpeg and libav use planar audio for many encoders, so it was somewhat necessary to add support in libobs itself. - Improve/adjust FFmpeg test output plugin. The exports were somewhat messed up (making me rethink how exports should be done). Not yet functional; it handles video properly, but it still does not handle audio properly. - Improve planar video code. The planar video code was not properly accounting for row sizes for each plane. Specifying row sizes for each plane has now been added. This will also make it more compatible with FFmpeg/libav. - Fixed a bug where callbacks wouldn't create properly in audio-io and video-io code. - Implement 'blogva' function to allow for va_list usage with libobs logging.
2014-02-07 02:03:54 -08:00
if (video_get_input_idx(video, callback, param) == DARRAY_INVALID) {
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
struct video_input input;
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
memset(&input, 0, sizeof(input));
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
input.callback = callback;
input.param = param;
Add preliminary ffmpeg plugin (still testing) - Added some code for FFmpeg output that I'm still playing around with. Right now I'm just trying to get it to output to file and try to understand the FFmpeg/libav APIs. Hopefully in the future this plugin can be used for any sort of output to FFmpeg. - Fixed a cast warning in audio-io.c with size_t -> uint32_t - Renamed the 'video_info' and 'audio_info' structures to 'video_conver_info' and 'audio_convert_info' to better represent their actual purpose, and to avoid confusion with 'audio_output_info' and 'video_output_info' structures. - Removed a few macros from obs-def.h that were at one point going to be used but no longer going to be used (at least for now)
2014-01-19 02:16:41 -08:00
if (conversion) {
input.conversion = *conversion;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
} else {
Add preliminary ffmpeg plugin (still testing) - Added some code for FFmpeg output that I'm still playing around with. Right now I'm just trying to get it to output to file and try to understand the FFmpeg/libav APIs. Hopefully in the future this plugin can be used for any sort of output to FFmpeg. - Fixed a cast warning in audio-io.c with size_t -> uint32_t - Renamed the 'video_info' and 'audio_info' structures to 'video_conver_info' and 'audio_convert_info' to better represent their actual purpose, and to avoid confusion with 'audio_output_info' and 'video_output_info' structures. - Removed a few macros from obs-def.h that were at one point going to be used but no longer going to be used (at least for now)
2014-01-19 02:16:41 -08:00
input.conversion.format = video->info.format;
input.conversion.width = video->info.width;
input.conversion.height = video->info.height;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
if (input.conversion.width == 0)
input.conversion.width = video->info.width;
if (input.conversion.height == 0)
input.conversion.height = video->info.height;
success = video_input_init(&input, video);
if (success)
da_push_back(video->inputs, &input);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
pthread_mutex_unlock(&video->input_mutex);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
return success;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
void video_output_disconnect(video_t *video,
Implement encoder interface (still preliminary) - Implement OBS encoder interface. It was previously incomplete, but now is reaching some level of completion, though probably should still be considered preliminary. I had originally implemented it so that encoders only have a 'reset' function to reset their parameters, but I felt that having both a 'start' and 'stop' function would be useful. Encoders are now assigned to a specific video/audio media output each rather than implicitely assigned to the main obs video/audio contexts. This allows separate encoder contexts that aren't necessarily assigned to the main video/audio context (which is useful for things such as recording specific sources). Will probably have to do this for regular obs outputs as well. When creating an encoder, you must now explicitely state whether that encoder is an audio or video encoder. Audio and video can optionally be automatically converted depending on what the encoder specifies. When something 'attaches' to an encoder, the first attachment starts the encoder, and the encoder automatically attaches to the media output context associated with it. Subsequent attachments won't have the same effect, they will just start receiving the same encoder data when the next keyframe plays (along with SEI if any). When detaching from the encoder, the last detachment will fully stop the encoder and detach the encoder from the media output context associated with the encoder. SEI must actually be exported separately; because new encoder attachments may not always be at the beginning of the stream, the first keyframe they get must have that SEI data in it. If the encoder has SEI data, it needs only add one small function to simply query that SEI data, and then that data will be handled automatically by libobs for all subsequent encoder attachments. - Implement x264 encoder plugin, move x264 files to separate plugin to separate necessary dependencies. - Change video/audio frame output structures to not use const qualifiers to prevent issues with non-const function usage elsewhere. This was an issue when writing the x264 encoder, as the x264 encoder expects non-const frame data. Change stagesurf_map to return a non-const data type to prevent this as well. - Change full range parameter of video scaler to be an enum rather than boolean
2014-03-16 16:21:34 -07:00
void (*callback)(void *param, struct video_data *frame),
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
void *param)
{
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
if (!video || !callback)
return;
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_lock(&video->input_mutex);
size_t idx = video_get_input_idx(video, callback, param);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
if (idx != DARRAY_INVALID) {
video_input_free(video->inputs.array+idx);
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
da_erase(video->inputs, idx);
Implement automatic video scaling (if requested) Add a scaler interface (defaults to swscale), and if a separate output wants to use a different scale or format than the default output format, allow a scaler instance to be created automatically for that output, which will then receive the new scaled output.
2014-02-18 12:37:56 -08:00
}
Simplify media i/o interfaces Completely revamped the entire media i/o data and handlers. The original idea was to have a system that would have connecting media inputs and outputs, but at a certain point I realized that this was an unnecessary complexity for what we wanted to do. (Also, it reminded me of directshow filters, and I HATE directshow with a passion, and wouldn't wish it upon my greatest enemy) Now, audio/video outputs are connected to directly, with better callback handlers, and will eventually have the ability to automatically handle conversions such as 4:4:4 to 4:2:0 when connecting to an input that uses them. Doing this will allow the video/audio i/o handlers to also prevent duplicate conversion, as well as make it easier/simple to use. My true goal for this is to make output and encoder plugins as simple to create as possible. I want to be able to be able to create an output plugin with almost no real hassle of having to worry about image conversions, media inputs/outputs, etc. A plugin developer shouldn't have to handle that sort of stuff when he/she doesn't really need to. Plugins will be able to simply create a callback via obs_video() and/or obs_audio(), and they will automatically receive the audio/video data in the formats requested via a simple callback, without needing to do almost anything else at all.
2014-01-14 00:58:47 -08:00
pthread_mutex_unlock(&video->input_mutex);
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
bool video_output_active(const video_t *video)
Fix video reset and apply new video settings This allows the changing of bideo settings without having to completely reset all graphics data. Will recreate internal output/conversion buffers and such and reset the main preview.
2014-02-22 19:14:19 -08:00
{
if (!video) return false;
return video->inputs.num != 0;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
const struct video_output_info *video_output_get_info(const video_t *video)
added functions to get current audio/video output information
2013-12-06 05:38:19 -08:00
{
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
return video ? &video->info : NULL;
added functions to get current audio/video output information
2013-12-06 05:38:19 -08:00
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
bool video_output_lock_frame(video_t *video, struct video_frame *frame,
int count, uint64_t timestamp)
first commit
2013-09-30 19:37:13 -07:00
{
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
struct cached_frame_info *cfi;
bool locked;
if (!video) return false;
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
first commit
2013-09-30 19:37:13 -07:00
pthread_mutex_lock(&video->data_mutex);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (video->available_frames == 0) {
libobs/media-io: Fix bug (skipped frames including lagged frames) The skipped frame count (dropped frames due to encoding being overloaded) would erroneously include lagged frames (dropped frames due to render stalls). This will make diagnosing actual issues a user might be having a bit easier.
2016-01-25 03:56:03 -08:00
video->skipped_frames += count;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
video->cache[video->last_added].count += count;
locked = false;
} else {
if (video->available_frames != video->info.cache_size) {
if (++video->last_added == video->info.cache_size)
video->last_added = 0;
}
cfi = &video->cache[video->last_added];
cfi->frame.timestamp = timestamp;
cfi->count = count;
memcpy(frame, &cfi->frame, sizeof(*frame));
locked = true;
}
first commit
2013-09-30 19:37:13 -07:00
pthread_mutex_unlock(&video->data_mutex);
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
return locked;
first commit
2013-09-30 19:37:13 -07:00
}
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
void video_output_unlock_frame(video_t *video)
first commit
2013-09-30 19:37:13 -07:00
{
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
if (!video) return;
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
pthread_mutex_lock(&video->data_mutex);
video->available_frames--;
os_sem_post(video->update_semaphore);
pthread_mutex_unlock(&video->data_mutex);
first commit
2013-09-30 19:37:13 -07:00
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint64_t video_output_get_frame_time(const video_t *video)
first commit
2013-09-30 19:37:13 -07:00
{
Add more checks for NULL pointers
2014-02-23 21:39:33 -08:00
return video ? video->frame_time : 0;
first commit
2013-09-30 19:37:13 -07:00
}
(API Change) Remove pointers from all typedefs Typedef pointers are unsafe. If you do: typedef struct bla *bla_t; then you cannot use it as a constant, such as: const bla_t, because that constant will be to the pointer itself rather than to the underlying data. I admit this was a fundamental mistake that must be corrected. All typedefs that were pointer types will now have their pointers removed from the type itself, and the pointers will be used when they are actually used as variables/parameters/returns instead. This does not break ABI though, which is pretty nice.
2014-09-25 17:44:05 -07:00
void video_output_stop(video_t *video)
first commit
2013-09-30 19:37:13 -07:00
{
void *thread_ret;
if (!video)
return;
if (video->initialized) {
Fix destructor video outputs. This fixes a crash on Linux that supposedly occurs when pthread_join is called multiple times.
2014-04-25 16:39:43 -07:00
video->initialized = false;
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
video->stop = true;
os_sem_post(video->update_semaphore);
first commit
2013-09-30 19:37:13 -07:00
pthread_join(video->thread, &thread_ret);
}
}
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
libobs: Redesign/optimize frame encoding handling Previously, the design for the interaction between the encoder thread and the graphics thread was that the encoder thread would signal to the graphics thread when to start drawing each frame. The original idea behind this was to prevent mutually cascading stalls of encoding or graphics rendering (i.e., if rendering took too long, then encoding would have to catch up, then rendering would have to catch up again, and so on, cascading upon each other). The ultimate goal was to prevent encoding from impacting graphics and vise versa. However, eventually it was realized that there were some fundamental flaws with this design. 1. Stray frame duplication. You could not guarantee that a frame would render on time, so sometimes frames would unintentionally be lost if there was any sort of minor hiccup or if the thread took too long to be scheduled I'm guessing. 2. Frame timing in the rendering thread was less accurate. The only place where frame timing was accurate was in the encoder thread, and the graphics thread was at the whim of thread scheduling. On higher end computers it was typically fine, but it was just generally not guaranteed that a frame would be rendered when it was supposed to be rendered. So the solution (originally proposed by r1ch and paibox) is to instead keep the encoding and graphics threads separate as usual, but instead of the encoder thread controlling the graphics thread, the graphics thread now controls the encoder thread. The encoder thread keeps a limited cache of frames, then the graphics thread copies frames in to the cache and increments a semaphore to schedule the encoder thread to encode that data. In the cache, each frame has an encode counter. If the frame cache is full (e.g., the encoder taking too long to return frames), it will not cache a new frame, but instead will just increment the counter on the last frame in the cache to schedule that frame to encode again, ensuring that frames are on time and reducing CPU usage by lowering video complexity. If the graphics thread takes too long to render a frame, then it will add that frame with the count value set to the total amount of frames that were missed (actual legitimately duplicated frames). Because the cache gives many frames of breathing room for the encoder to encode frames, this design helps improve results especially when using encoding presets that have higher complexity and CPU usage, minimizing the risk of needlessly skipped or duplicated frames. I also managed to sneak in what should be a bit of an optimization to reduce copying of frame data, though how much of an optimization it ultimately ends up being is debatable. So to sum it up, this commit increases accuracy of frame timing, completely removes stray frame duplication, gives better results for higher complexity encoding presets, and potentially optimizes the frame pipeline a tiny bit.
2014-12-31 01:53:13 -08:00
bool video_output_stopped(video_t *video)
{
if (!video)
return true;
return video->stop;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
enum video_format video_output_get_format(const video_t *video)
video-io: Add function to just get video format
2014-08-10 16:39:36 -07:00
{
return video ? video->info.format : VIDEO_FORMAT_NONE;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_width(const video_t *video)
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
{
return video ? video->info.width : 0;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_height(const video_t *video)
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
{
return video ? video->info.height : 0;
}
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
double video_output_get_frame_rate(const video_t *video)
Add preliminary FLV/RTMP output (incomplete) - obs-outputs module: Add preliminary code to send out data, and add an FLV muxer. This time we don't really need to build the packets ourselves, we can just use the FLV muxer and send it directly to RTMP_Write and it should automatically parse the entire stream for us without us having to do much manual code at all. We'll see how it goes. - libobs: Add AVC NAL packet parsing code - libobs/media-io: Add quick helper functions for audio/video to get the width/height/fps/samplerate/etc rather than having to query the info structures each time. - libobs (obs-output.c): Change 'connect' signal to 'start' and 'stop' signals. 'start' now specifies an error code rather than whether it simply failed, that way the client can actually know *why* a failure occurred. Added those error codes to obs-defs.h. - libobs: Add a few functions to duplicate/free encoder packets
2014-04-01 11:55:18 -07:00
{
if (!video)
return 0.0;
return (double)video->info.fps_num / (double)video->info.fps_den;
}
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_skipped_frames(const video_t *video)
Implement high encoder CPU usage handling This implements the 'frame skipping' mechanism to forcibly cause frames to be duplicated in order to reduce encoder complexity so the encoder can catch up to the video, otherwise it will continue to be progressively behind and will cause a desync of the video. Typically, if a user gets this issue, they should turn down their settings. For the love of god do not tell them that 'frames are skipping', just tell them that CPU usage is high, and that they should consider turning down their settings.
2014-07-01 11:01:22 -07:00
{
return video->skipped_frames;
}
media-io: Add function for total video frames
2014-07-27 09:25:19 -07:00
(API Change) Use const params where applicable This Fixes a minor flaw with the API where data had to always be mutable to be usable by the API. Functions that do not modify the fundamental underlying data of a structure should be marked as constant, both for safety and to signify that the parameter is input only and will not be modified by the function using it.
2014-09-26 15:25:59 -07:00
uint32_t video_output_get_total_frames(const video_t *video)
media-io: Add function for total video frames
2014-07-27 09:25:19 -07:00
{
return video->total_frames;
}