Across video streaming, editing, and conferencing workflows, handling high-quality video demands significant computational overhead and memory consumption. Working with a diverse set of video sources (cameras, capture devices, video files, etc) requires dealing with multiple pixel formats such as RGB and YCbCr, with varying chroma subsampling. Color depths may be represented in 8, 10, 12 or 16 bits, but not all devices support high-color. As a result, most video applications rely on per-frame conversions in order to unify the different sources and process/render video in real time. The state-of-the-art libraries which offer this are mostly CPU-based: libswscale, used by the industry standard solution ffmpeg; libyuv, from Chromium’s WebRTC implementation; and zimg, a project-independent library that uses vectorized operations to improve performance. Being CPU-based, these solutions do not take advantage of modern GPU computing, which would enable higher performance by relieving congestion on the primary processing unit of the system.

This paper presents the open-source project Pixelweave Chromascale, a multi-platform library that leverages GPU computing to perform video frame conversions. Pixelweave is based on the multi-platform graphics library Vulkan, allowing for a single implementation of GPU kernels that is compatible with most modern graphics hardware. The library supports multiple pixel formats such as RGBA, planar YUV with 420, 422, and 444 chroma subsampling, YV12, NV12 and other interleaved formats, using up to 16 bits per channel. The implemented compute shaders use a novel generic approach that samples 2x2 pixel windows, making the rest of the code easily extensible and independent of the input’s chroma-subsampling. Initial tests show that Pixelweave is an order of magnitude faster than the state-of-the-art libraries on integrated GPUs, such as on the Apple Ms processors. It also performs faster on high-end discrete GPUs, allowing the CPU to offload resource-demanding tasks.