8384c8455e  Type Confusion in RTCEncodedStreamProducer.cpp Results in OOB Read

The bug at a glance

The bug lives in WebRTC Encoded Transform, a surface reachable from any page allowed to use RTCPeerConnection/RTCRtpScriptTransform with no special permission. A type confusion between audio and video encoded frames lets attacker script route a frame of the wrong media type into a sender/receiver back-end whose native code assumes the other type, yielding an out-of-bounds read of encoded-frame memory. Because the mismatch reaches native RTP packetization/decode paths in the WebContent process and can be driven deterministically from JS, high severity is justified; it is a read primitive rather than a direct write, which caps it below critical.

A web page opens an RTCRtpScriptTransform, whose transformer exposes a ReadableStream of encoded frames and a WritableStream back into the pipeline. Nothing stopped script from writing a video RTCEncodedFrame into an audio transformer’s writable (or a frame belonging to a different transformer entirely), so the native side processed audio bytes as a video frame layout and vice versa.

Root cause

RTCRtpScriptTransformer wires a per-track transform: encoded frames flow out through RTCEncodedStreamProducer’s ReadableStream as either RTCEncodedAudioFrame or RTCEncodedVideoFrame JS wrappers, and script writes frames back through writeFrame(). Before the patch, RTCEncodedStreamProducer::writeFrame() decoded whatever JS handed it via WTF::switchOn over the RTCEncodedAudioFrame/RTCEncodedVideoFrame variant, extracted the underlying rtcFrame, and passed it straight to transformBackend->processTransformedFrame(rtcFrame.get()). The back-end is fixed to one media type (the sender or receiver it belongs to), but writeFrame() never checked that the frame’s media type matched, nor that the frame actually originated from this transformer’s own ReadableStream.

That allows two distinct confusions. First, media-type confusion: a video frame can be written into an audio sender/receiver (or vice versa). The native RTP transform back-end then interprets a buffer laid out as one codec’s encoded frame using the header/field expectations of the other, and downstream packetization/decode reads past the real bounds of the smaller/differently-shaped buffer. Second, cross-transformer confusion: an RTCEncodedFrame obtained from one transformer’s readable could be injected into a different transformer’s writable, again violating the invariant the back-end relies on.

The fix records provenance and type on the producer. RTCEncodedStreamProducer::start() now takes an RTCRtpScriptTransformer* and stores m_hasTransformer plus a WeakPtr<RTCRtpScriptTransformer> m_transformer; RTCRtpScriptTransformer::start() passes this. When frames are handed out, enqueueFrame() now stamps each RTCRtpTransformableFrame with frame->setTransformer(m_transformer). writeFrame() computes the incoming frame’s media type (isVideo, set true in the video arm of the switchOn) and rejects the frame with an early return { } whenever m_isVideo != isVideo (media-type mismatch) or, for the script-transformer case, m_hasTransformer && !rtcFrame->isFromTransformer(m_transformer.get()) (the frame did not come from this transformer). RTCRtpTransformableFrame::isFromTransformer/setTransformer were re-typed to take a raw pointer / WeakPtr so a null/foreign transformer compares unequal safely.

The WeakPtr matters: the transformer can be torn down while frames are in flight, so comparing a raw address could itself be unsafe; storing a WeakPtr means a dead transformer resolves to null and the equality check simply fails closed. Net effect: the enqueue side tags the frame with its origin and the write side enforces both media type and origin before any native back-end touches the buffer.

Key code

Added media-type and provenance guard in RTCEncodedStreamProducer::writeFrame

    bool isVideo = false;
    auto frame = frameConversionResult.releaseReturnValue();
    auto rtcFrame = WTF::switchOn(frame,
        [&](Ref<RTCEncodedAudioFrame>& value) {
            return value->rtcFrame(vm);
        },
        [&](Ref<RTCEncodedVideoFrame>& value) {
            isVideo = true;
            return value->rtcFrame(vm);
        }
    );

    if (m_isVideo != isVideo || (m_hasTransformer && !rtcFrame->isFromTransformer(m_transformer.get())))
        return { };

    // If no data, skip the frame since there is nothing to packetize or decode.
    if (rtcFrame->data().data())
        transformBackend->processTransformedFrame(rtcFrame.get());

Patch walkthrough

• Source/WebCore/Modules/mediastream/RTCEncodedStreamProducer.cpp — start() gains an RTCRtpScriptTransformer* parameter and stores m_hasTransformer/m_transformer; enqueueFrame() stamps each outgoing frame with setTransformer(m_transformer); writeFrame() computes the frame’s isVideo and adds the guard if (m_isVideo != isVideo || (m_hasTransformer && !rtcFrame->isFromTransformer(m_transformer.get()))) return { }; before processTransformedFrame.
• Source/WebCore/Modules/mediastream/RTCEncodedStreamProducer.h — start() signature gains a defaulted RTCRtpScriptTransformer* argument; adds members bool m_hasTransformer { false } and WeakPtr<RTCRtpScriptTransformer> m_transformer.
• Source/WebCore/Modules/mediastream/RTCRtpScriptTransformer.cpp — start() now passes this as the transformer when calling m_streamProducer->start(…), establishing the provenance link.
• Source/WebCore/Modules/mediastream/RTCRtpTransformableFrame.h — setTransformer now takes a WeakPtr<RTCRtpScriptTransformer> and isFromTransformer takes a raw RTCRtpScriptTransformer* so mismatched/dead transformers compare unequal (fail-closed).
• LayoutTests/http/wpt/webrtc/audio-video-transform.js and audiovideo-script-transform.html — Test harness extended with tryWritingAudio/tryWritingVideo cases that attempt to write audio data on a video sender and video data on an audio sender, expecting the write to be silently dropped (PASS).

Background

RTCRtpScriptTransform / Encoded Transform — A WebRTC API that inserts a Web Worker transform into a sender or receiver’s encoded-media pipeline. The transformer receives RTCEncodedAudioFrame or RTCEncodedVideoFrame objects on a ReadableStream, may modify them, and writes them back on a WritableStream. It is available to ordinary web content that uses RTCPeerConnection.

RTCEncodedStreamProducer — The WebCore glue that bridges a native RTCRtpTransformBackend to the JS ReadableStream/WritableStream. enqueueFrame() wraps native RTCRtpTransformableFrames as JS frame objects going out; writeFrame() unwraps JS frames coming back in and feeds them to processTransformedFrame on the back-end.

RTCRtpTransformBackend media type — Each back-end instance is bound to exactly one media type (audio or video) and one side (sender or receiver). Its native frame handling assumes a specific encoded-frame memory layout. m_isVideo on the producer mirrors backend->mediaType() == Video.

Type confusion — A memory-safety flaw where a value of one type is operated on as if it were another, so code reads or writes fields at offsets valid for the wrong type. Here an audio-encoded buffer is parsed with video-frame expectations (or vice versa), driving reads beyond the actual buffer bounds.

WeakPtr fail-closed check — m_transformer is stored as a WeakPtr so that if the transformer is destroyed while frames are in flight, isFromTransformer() compares against a null pointer and returns false, dropping the frame rather than dereferencing freed memory.

Vulnerability window

1. Introduction — writeFrame() accepted any RTCEncodedFrame variant and forwarded it to the back-end without checking that the frame’s media type matched the back-end’s type or that the frame originated from this transformer.
2. Setup — Attacker page creates a peer connection with both audio and video tracks and installs RTCRtpScriptTransforms on the corresponding senders/receivers, capturing each transformer’s readable/writable.
3. Trigger — Script reads a video encoded frame from the video transformer and writes it into the audio sender’s writable (or the reverse), as the added test’s tryWritingVideo/tryWritingAudio paths do.
4. Confusion — writeFrame() unwraps the foreign frame and calls processTransformedFrame on an audio back-end with a video-layout buffer; the native side reads header/payload fields at video offsets.
5. OOB read — Because the buffer’s real size/layout differs, packetization or decode reads past the buffer, disclosing adjacent WebContent heap contents or crashing.
6. Fix — 315256@main adds the m_isVideo/isVideo match plus the isFromTransformer provenance check, dropping mismatched or foreign frames before the back-end sees them.

Proof of concept

The updated WPT harness captures a video-sender chunk in the shared videoChunk variable, then writes it into the audio sender transformer’s writable via this.writer.write(videoChunk.value). The corresponding html driver posts ’tryWritingVideo’ to audioSenderTransform and also exercises ’tryWritingAudio’ on the video sender, ’tryWritingAudio’ on the audio sender with receiver data, and ’tryWritingVideo’ on the video sender. Post-fix these writes are dropped by writeFrame and the test observes PASS instead of a crash/OOB.

// From audio-video-transform.js process(): write a captured video chunk
// into the audio sender transformer (media-type confusion)
if (audioSenderTransformer && audioSenderTransformer.tryWritingVideo) {
    if (audioSenderTransformer === this) {
        this.writer.write(chunk.value);
        if (videoChunk !== undefined) {
           this.writer.write(videoChunk.value);
           audioSenderTransformer.tryWritingVideo = false;
           this.context.options.port.postMessage("PASS");
        }
        this.process();
        return;
    }
    if(videoSenderTransformer === this) {
        videoChunk = chunk;
        while (audioSenderTransformer.tryWritingVideo)
            await new Promise(resolve => setTimeout(resolve, 50));
        videoSenderTransformer.writer.write(videoChunk);
        videoChunk = undefined;
        this.process();
        return;
    }
}

Exploitation

1. Reconnaissance — Attacker page enumerates its own senders/receivers and installs script transforms; entirely first-party, no cross-origin or user gesture needed beyond permission to use the media pipeline.
2. Primitive — Cross-type write yields an OOB read: encoded audio/video buffers of one layout are parsed with the other’s field offsets, reading beyond the real allocation.
3. Info leak — Leaked bytes are adjacent WebContent heap; with the read framed by the transform output, an attacker may recover heap data useful for ASLR/pointer disclosure. This is a read primitive, not a direct write.
4. Escalation — On its own this is disclosure/crash; chaining to code execution would require a separate write bug. Honest assessment: primarily an OOB-read/info-leak and reliable WebContent crash.

Detection & hunting

For defenders and SOC / detection engineers:

• WebContent crashes in RTP transform/packetization —
• Anomalous RTCRtpScriptTransform usage —
• Repeated peer connections to loopback/no ICE —

Audit directions

• Other writeFrame/enqueue paths —
• Frame provenance model —
• Back-end type assumptions —
• Lifetime of transformer references —