Per mbedtls documentation, "If you share a context between threads, you
need to call these functions only from the main thread, at the beginning
and end of the context's lifetime.". OBS violated this since librtmp
uses a global context and it was allocated and freed in different
threads such as the auto config test.
This commit attaches the mbedtls context to an RTMP structure so there
is no more global state. It also fixes a rare double-free crash that
could occur if RTMP_TLS_Free was called twice (this happened in rare
situations such as the auto config running followed by a mode change
from Advanced to Simple).
mbedtls_md5_starts(), mbedtls_md5_update() and mbedtls_md5_finish()
have been marked deprecated since mbedtls version 2.7 and may go
away in the future.
These function have been superseded by versions with a return
value. They are suffixed with "_ret". We do not actually check
return values.
Also the header "mbedtls/net.h" has been superseded by
"mbedtls/net_sockets.h".
This diff adds mbedTLS support to the obs-outputs plugin. PolarSSL and
mbedTLS have grown so different between 2015-or-so when libRTMP was
written, and now it's no longer feasible to just use the USE_POLARSSL
flag.
This commit adds a WITH_RTMPS tri-state CMake variable (auto/on/off),
set to "Auto" by default. "Auto" will use RTMPS if mbedTLS is found,
otherwise will disable RTMPS. "On" will make it require mbedTLS,
otherwise fails configuration, and "Off" disables RTMPS support
altogether.
Closesobsproject/obs-studio#1360
On windows, for whatever reason sockets use the SOCKET type which is not
a signed integer. Still, even though it's not a signed integer, -1 is
used to indicate an invalid socket, but the way you use it is via
microsoft's fabulously dumb little INVALID_SOCKET define, so we have to
make librtmp use that instead.
Note that this is a somewhat heavily modified custom version of librtmp.
I modified all the platform specific code that we were using for the
OBS1 to make it platform-independent.
I don't really like the code in this library, but it works well enough,
so I can't really fault anyone for it. It's just very.. unclean. Even
for a C library, quite unclean. Some parts are also a little less safe
than I'd prefer as well.
