This code is very old and seems to be non-functional in its current
state. The TLS support is also complicated to maintain across multiple
deprecated mbedtls functions.
Per mbedtls documentation, "If you share a context between threads, you
need to call these functions only from the main thread, at the beginning
and end of the context's lifetime.". OBS violated this since librtmp
uses a global context and it was allocated and freed in different
threads such as the auto config test.
This commit attaches the mbedtls context to an RTMP structure so there
is no more global state. It also fixes a rare double-free crash that
could occur if RTMP_TLS_Free was called twice (this happened in rare
situations such as the auto config running followed by a mode change
from Advanced to Simple).
Some security layer libraries code path used by the rtmp output had a
not used variable and not used param on HMAC_finish macro that was
triggering warnings during compilation.
This diff adds mbedTLS support to the obs-outputs plugin. PolarSSL and
mbedTLS have grown so different between 2015-or-so when libRTMP was
written, and now it's no longer feasible to just use the USE_POLARSSL
flag.
This commit adds a WITH_RTMPS tri-state CMake variable (auto/on/off),
set to "Auto" by default. "Auto" will use RTMPS if mbedTLS is found,
otherwise will disable RTMPS. "On" will make it require mbedTLS,
otherwise fails configuration, and "Off" disables RTMPS support
altogether.
Closesobsproject/obs-studio#1360
Note that this is a somewhat heavily modified custom version of librtmp.
I modified all the platform specific code that we were using for the
OBS1 to make it platform-independent.
I don't really like the code in this library, but it works well enough,
so I can't really fault anyone for it. It's just very.. unclean. Even
for a C library, quite unclean. Some parts are also a little less safe
than I'd prefer as well.
Kurt Kartaltepe
Richard Stanway
Shaolin
Michael Goulet
jp9000