RTMP authentication requires the curStreamIdx and nStreams be set
between the disconnect / reconnect that occurs in PublisherAuth. Now
that there's no good place to clean them up, zero the whole rtmp->Link
before starting a new stream.
Every addStream call would increment this counter. After merging the
mbedTLS fixes, we no longer have extraneous RTMP_Init calls which were
masking the prescence of this bug. This caused every stream after the
first stream to have the wrong channel index, and eventually OBS would
crash due to an out of bounds write if the counter reached
RTMP_MAX_STREAMS.
The report in https://github.com/obsproject/obs-studio/issues/2350
identified the issue as being caused by mbedtls not following symbolic
links, but it turns out the issue was the mbedtls_x509_crt_parse_path
return value which was already fixed in 4d89123c. So these changes are
no longer necessary.
This code is very old and seems to be non-functional in its current
state. The TLS support is also complicated to maintain across multiple
deprecated mbedtls functions.
Though this should now be very rare, it's more helpful than "Failed to
connect to server". Other TLS error codes are now also stored for future
use instead of copying them on a case by case basis.
Per mbedtls documentation, "If you share a context between threads, you
need to call these functions only from the main thread, at the beginning
and end of the context's lifetime.". OBS violated this since librtmp
uses a global context and it was allocated and freed in different
threads such as the auto config test.
This commit attaches the mbedtls context to an RTMP structure so there
is no more global state. It also fixes a rare double-free crash that
could occur if RTMP_TLS_Free was called twice (this happened in rare
situations such as the auto config running followed by a mode change
from Advanced to Simple).
mbedtls_x509_crt_parse_path returns a positive number if it partially
succeeds and a negative number on complete failure. This changes the
positive result to no longer error and prevent TLS connections (OBS
verifies all endpoints so having no CA chain prevents TLS).
mbedtls_md5_starts(), mbedtls_md5_update() and mbedtls_md5_finish()
have been marked deprecated since mbedtls version 2.7 and may go
away in the future.
These function have been superseded by versions with a return
value. They are suffixed with "_ret". We do not actually check
return values.
Also the header "mbedtls/net.h" has been superseded by
"mbedtls/net_sockets.h".
Some security layer libraries code path used by the rtmp output had a
not used variable and not used param on HMAC_finish macro that was
triggering warnings during compilation.
On 64bit systems, this check will always evaluate to false due to
SIZE_MAX type and triggers a compiler warning.
This both makes it clearer that its only needed on 32bit system and
clear the compiler warning.
This reverts commit f1f49bc197f44f1f20cb268da2f9fdcda82049a0, which
breaks RTMP authentication as these parameters are intended to "leak"
into the next use of the RTMP context.
This diff adds mbedTLS support to the obs-outputs plugin. PolarSSL and
mbedTLS have grown so different between 2015-or-so when libRTMP was
written, and now it's no longer feasible to just use the USE_POLARSSL
flag.
This commit adds a WITH_RTMPS tri-state CMake variable (auto/on/off),
set to "Auto" by default. "Auto" will use RTMPS if mbedTLS is found,
otherwise will disable RTMPS. "On" will make it require mbedTLS,
otherwise fails configuration, and "Off" disables RTMPS support
altogether.
Closesobsproject/obs-studio#1360
For some reason librtmp treats a close message as a request to reconnect.
It does this syncronously, and uses the same event processing so that
another close request will continue the cycle until the stack is exhausted
and the application crashes.
Fixes https://obsproject.com/mantis/view.php?id=634
Apple wants to get people to move over to their own crypto APIs instead
of using OpenSSL, so disable the warning in the files where OpenSSL is
used for the time being.
With no stream key, no streams were actually being created.
This is a crazy configuration anyway, but it resulted in OBS getting
stuck in the "Connecting" state with no way to cancel.
We now just use the blank key and hope for the best.
Reinstate flag checks in RTMP_Close that were erroneously removed.
Clear out the Link state before we establish a new connection. There is
too much state carried around during authentication that has no good
place to clear it in librtmp, which assumes a clean structure when the
connection is initially established.
Authentication code has been updated as per the changes to support
multiple streams.
Authentication is now also enabled by default, and should be a no-op
if the server does not request authentication or username and password
details are not provided.
This was caused to do the new RTMP code that added support for multiple
streams; the stream index needs to be reset on RTMP_Close otherwise it
will keep using the wrong stream information.
On windows, for whatever reason sockets use the SOCKET type which is not
a signed integer. Still, even though it's not a signed integer, -1 is
used to indicate an invalid socket, but the way you use it is via
microsoft's fabulously dumb little INVALID_SOCKET define, so we have to
make librtmp use that instead.
Certain RTMP services will support multi audio tracks via RTMP. This
updates librtmp with custom code that enables multiple streams per
connection to be used; each subsequent stream typically containing extra
audio tracks. The audio encoder names are used to indicate the names of
tracks, and the name of the tracks are used for the stream keys for
those subsequent tracks.
