Project Fortify, an experimental security project, has identified the following potential security issue. If you’re a feature owner CC-ed on this bug, please do your best to review these reports without the security team.

Overview: A compromised renderer can bypass Mojo dimension checks by supplying a small base resolution and maliciously large spatial layer resolutions in the VideoEncodeAccelerator config. This causes the VA-API hardware driver to allocate a small context but process a massive frame, leading to a potential heap out-of-bounds write in the GPU process.

Affected files:

• media/gpu/vaapi/vp9_vaapi_video_encoder_delegate.cc
• media/gpu/vaapi/vaapi_video_encode_accelerator.cc
• media/mojo/mojom/video_encode_accelerator_mojom_traits.cc
• media/mojo/services/mojo_video_encode_accelerator_service.cc
• media/gpu/vaapi/vaapi_wrapper.cc

Estimated timestamp from git blame: 2025-03-26

Description

A vulnerability exists in the VP9 VA-API video encoding implementation where a compromised renderer can potentially trigger a heap out-of-bounds (OOB) write in the GPU process. The issue arises from a failure to validate the dimensions of spatial_layers provided by the renderer during encoder initialization.

Technical Details

1. Validation Bypass: In media/mojo/services/mojo_video_encode_accelerator_service.cc, MojoVideoEncodeAcceleratorService::Initialize validates config.input_visible_size against media::limits::kMaxDimension and limits::kMaxCanvas. However, it does not perform any validation on the dimensions of the elements within the config.spatial_layers array.
2. Hardware Profile Bypass: Similarly, VaapiVideoEncodeAccelerator::Initialize verifies that spatial_layers are sorted by size but fails to check their individual dimensions against the maximum supported resolution of the hardware profile.
3. Mismatched Initialization: VP9VaapiVideoEncoderDelegate::Initialize extracts these unvalidated spatial_layers dimensions. The encoder then creates a VA-API context and allocates output buffers sized according to the small, validated input_visible_size.
4. Implicit Truncation and OOB: During frame submission, VP9VaapiVideoEncoderDelegate::SubmitFrameParameters copies the spatial layer dimensions to the VAEncPictureParameterBufferVP9 fields (frame_width_src, frame_height_src, etc.). Because the attacker can provide an int32_t value like INT_MAX, and the VA-API parameter fields are uint16_t, the value is truncated (e.g., INT_MAX becomes 65535). The VA-API driver is then instructed to process a massive 65535x65535 frame within internal context buffers allocated for a much smaller resolution (e.g., 320x240), causing a potential GPU heap out-of-bounds write.

Suggested Attacker Steps

Note: These are potential steps for triggering the vulnerability, as our tooling agent currently lacks the ability to execute code or build a working proof of concept.

1. From a compromised renderer process on a platform utilizing VA-API (e.g., ChromeOS, Linux), request a mojom::VideoEncodeAccelerator interface.
2. Call the Initialize method with a maliciously crafted VideoEncodeAcceleratorConfig.
3. Set input_visible_size to a completely valid, small resolution (e.g., 320x240) to bypass Mojo’s base dimension checks.
4. Add a single element to the spatial_layers array. Because it is a single element, it easily bypasses the std::ranges::is_sorted check in the GPU process.
5. Set the width and height of this spatial layer to 0x7FFFFFFF (INT_MAX).
6. Submit a video frame for encoding. The VA-API driver will receive picture parameters for a 65535x65535 frame but will execute the encode on a 320x240 heap context.

Suggested Fix

1. Mojo Service Validation: In MojoVideoEncodeAcceleratorService::Initialize, iterate over config.spatial_layers and validate that each layer’s width and height do not exceed limits::kMaxDimension, and that the calculated area does not exceed limits::kMaxCanvas.
2. VEA Validation: In VaapiVideoEncodeAccelerator::Initialize, ensure that all provided spatial layer dimensions are less than or equal to the selected profile’s max_resolution.
3. Ensure spatial layer dimensions are strictly constrained by and validated against the base input_visible_size where appropriate.

Evaluated with Chrome root at commit: f200f57a19490707ff8bc7aa5de3cbc443a3afad

Results so far have been promising, but there can be wrong deductions. If this proves to be a false positive, please close as WAI; data from false positives will be used to improve accuracy over time. And please feel free to reach out to me directly if you have concerns or feedback on the project.