Heap buffer overflow write in libaom AV1 SVC encoding via WebRTC

Summary

A heap buffer overflow write exists in libaom’s AV1 encoder when encoding video with spatial Scalable Video Coding (SVC) through WebRTC. The chroma_check function in var_based_part.c uses the LAST_FRAME reference’s scale factors when computing buffer offsets for GOLDEN_FRAME and ALTREF_FRAME, which have different resolutions in spatial SVC configurations. This causes setup_pred_plane to calculate an out-of-bounds buffer pointer, and subsequent frame copy operations write past the end of the allocated YV12 frame buffer. A web page can trigger this by establishing a WebRTC peer connection with AV1 and the L2T1 scalability mode. The vulnerability affects all platforms where WebRTC AV1 SVC encoding is supported (Windows, Linux, macOS, ChromeOS, Android).

Bisect

Introducing Commit (GOLDEN path): c6a503f

• Date: August 18, 2022
• Author: Marco Paniconi (marpan@google.com)
• Review: https://aomedia-review.googlesource.com/c/aom/+/161261
• Subject: “rtc: Fix to color artifacts under high motion”

Extended to ALTREF path: 555b3aa

• Date: March 6, 2023
• Author: Marco Paniconi (marpan@google.com)
• Review: https://aomedia-review.googlesource.com/c/aom/+/171601
• Subject: “rtc: Color sensitivity for altref in nonrd”

Both commits are in the upstream aom.git repository. The original setup_pred_plane with LAST_FRAME’s scale factors was introduced in commit 309d0af (July 27, 2022) for the LAST_FRAME path, where using LAST’s scale factors is correct. When the GOLDEN and ALTREF paths were added in the subsequent commits, they reused the same sf variable without obtaining the appropriate scale factors for those reference frames.

Root Cause

The chroma_check function in var_based_part.c computes UV chroma SAD values for three reference frames (LAST, GOLDEN, ALTREF) to determine color sensitivity. It retrieves the scale factors once, for LAST_FRAME only:

// av1/encoder/var_based_part.c:1026-1029
// https://aomedia.googlesource.com/aom/+/39606bf4bae/av1/encoder/var_based_part.c#1026
const YV12_BUFFER_CONFIG *yv12_g = get_ref_frame_yv12_buf(cm, GOLDEN_FRAME);
const YV12_BUFFER_CONFIG *yv12_alt = get_ref_frame_yv12_buf(cm, ALTREF_FRAME);
const struct scale_factors *const sf =
    get_ref_scale_factors_const(cm, LAST_FRAME);

This sf is then passed to setup_pred_plane for all three reference frames. For LAST_FRAME, this is correct. For GOLDEN_FRAME and ALTREF_FRAME, it is wrong:

// av1/encoder/var_based_part.c:1062-1069
// https://aomedia.googlesource.com/aom/+/39606bf4bae/av1/encoder/var_based_part.c#1062
if (y_sad_g != UINT_MAX) {
    uint8_t *src = (plane == 1) ? yv12_g->u_buffer : yv12_g->v_buffer;
    setup_pred_plane(&dst, xd->mi[0]->bsize, src, yv12_g->uv_crop_width,
                     yv12_g->uv_crop_height, yv12_g->uv_stride, xd->mi_row,
                     xd->mi_col, sf, xd->plane[plane].subsampling_x,
                     xd->plane[plane].subsampling_y);
    uv_sad_g = cpi->ppi->fn_ptr[bs].sdf(p->src.buf, p->src.stride,
                                         dst.buf, dst.stride);
}

The setup_pred_plane function computes the buffer pointer as dst->buf = src + scaled_buffer_offset(x, y, stride, scale), where x and y are derived from the current macroblock’s mi_row and mi_col:

// av1/common/reconinter.h:386-404
// https://aomedia.googlesource.com/aom/+/39606bf4bae/av1/common/reconinter.h#386
static inline void setup_pred_plane(struct buf_2d *dst, BLOCK_SIZE bsize,
                                    uint8_t *src, int width, int height,
                                    int stride, int mi_row, int mi_col,
                                    const struct scale_factors *scale,
                                    int subsampling_x, int subsampling_y) {
    const int x = (MI_SIZE * mi_col) >> subsampling_x;
    const int y = (MI_SIZE * mi_row) >> subsampling_y;
    dst->buf = src + scaled_buffer_offset(x, y, stride, scale);
}

In AV1 spatial SVC, the enhancement layer (e.g., 1280x720) references a lower-resolution base layer (e.g., 640x360) as its GOLDEN frame. LAST_FRAME, being from the same spatial layer, has identity scale factors (no scaling). When these identity scale factors are applied to mi_row/mi_col coordinates from the 1280x720 layer while indexing into the 640x360 GOLDEN buffer, the resulting offset exceeds the smaller buffer’s allocation. For a 1280x720 frame with 4:2:0 chroma subsampling, the UV plane coordinates can reach up to y=358, but the GOLDEN frame’s UV buffer at 640x360 only accommodates rows up to y=179, producing an overflow of approximately 114 rows worth of data.

The ASAN report shows the overflow manifests as a write of 640 bytes immediately past the end of a 688,935-byte allocation in aom_yv12_copy_v_c, which copies the V chroma plane row by row using memcpy. The overflow occurs because the source or destination frame buffer was allocated for the smaller spatial layer dimensions, while the copy loop iterates over the larger layer’s height.

Reproduce

Configure out/asan-release/args.gn:

is_asan = true
is_debug = false
dcheck_always_on = false
target_cpu = "x64"
is_component_build = true

Then build and run:

autoninja -C out/asan-release chrome
ASAN_OPTIONS=detect_odr_violation=0 out/asan-release/chrome --no-sandbox poc.html

The PoC opens a WebRTC peer connection with AV1 SVC (L2T1 scalability mode) encoding a 1280x720 canvas stream. The heap-buffer-overflow fires within seconds of encoding.

The ASAN report:

=================================================================
==43000==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x1409738e9b27 at pc 0x7ff8de4eb48c bp 0x00e97d7fc8e0 sp 0x00e97d7fc928
WRITE of size 640 at 0x1409738e9b27 thread T5
    #0 0x7ff8de4eb48b  (clang_rt.asan_dynamic-x86_64.dll+0x18004b48b)
    #1 0x7ff8b03c7b9a in aom_yv12_copy_v_c yv12extend.c:313:5
    #2 0x7ff8b07b14f1 in encode_with_recode_loop_and_filter encoder.c:3706:9
    #3 0x7ff8b07a1f41 in av1_encode encoder.c:4611:9
    #4 0x7ff8b0eecde7 in av1_encode_strategy encode_strategy.c:1713:7
    #5 0x7ff8b07a8765 in av1_get_compressed_data encoder.c:5360:22
    #6 0x7ff8b03a627f in encoder_encode av1_cx_iface.c:3620:20
    #7 0x7ff8b03c3dc8 in aom_codec_encode aom_encoder.c:191:11
    #8 0x7ff8afda2f26 in LibaomAv1Encoder::DoEncode libaom_av1_encoder.cc:1101:7
    #9 0x7ff8afd9af17 in LibaomAv1Encoder::Encode libaom_av1_encoder.cc:993:16
    #10 0x7ff8ae3d79d3 in SimulcastEncoderAdapter::Encode simulcast_encoder_adapter.cc:672:23
    #11 0x7ff84aee87fe in StatsCollectingEncoder::Encode stats_collecting_encoder.cc:123:20
    #12 0x7ff8aef38013 in VideoStreamEncoder::EncodeVideoFrame video_stream_encoder.cc:2226:43

0x1409738e9b27 is located 0 bytes after 688935-byte region [0x140973841800,0x1409738e9b27)
allocated by thread T5 here:
    #0 0x7ff8de4ec8df  (clang_rt.asan_dynamic-x86_64.dll+0x18004c8df)
    #1 0x7ff8b07b7dee in aom_memalign aom_mem.c:59:22
    #2 0x7ff8b086bc5a in aom_realloc_frame_buffer yv12config.c:259:12
    #3 0x7ff8b086c2f8 in aom_alloc_frame_buffer yv12config.c:274:12
    #4 0x7ff8b07b143e in encode_with_recode_loop_and_filter encoder.c:3706:9

SUMMARY: AddressSanitizer: heap-buffer-overflow yv12extend.c:313:5 in aom_yv12_copy_v_c

References

• chroma_check (var_based_part.c:987) — buggy function, uses LAST’s sf for GOLDEN/ALTREF
• setup_pred_plane (reconinter.h:386) — computes dst->buf via scaled_buffer_offset
• aom_yv12_copy_v_c (yv12extend.c:293) — crash site, memcpy overflow
• LibaomAv1Encoder::DoEncode (libaom_av1_encoder.cc) — WebRTC entry point into libaom

Credit

Please use c6eed09fc8b174b0f3eebedcceb1e792 as the credit for this vulnerability. Thank you.