Skip to main content

Extracting frames from a video in JavaScript

Extracting frames from a video file, for example to display a filmstrip in an editing interface, can be done using Mediabunny.

Here's a extractFrames() function you can use coyp and paste into your project:

extract-frames.ts
ts
import {ALL_FORMATS, Input, InputDisposedError, UrlSource, VideoSample, VideoSampleSink} from 'mediabunny';
 
type Options = {
  track: {width: number; height: number};
  container: string;
  durationInSeconds: number | null;
};
 
export type ExtractFramesTimestampsInSecondsFn = (options: Options) => Promise<number[]> | number[];
 
export type ExtractFramesProps = {
  src: string;
  timestampsInSeconds: number[] | ExtractFramesTimestampsInSecondsFn;
  onVideoSample: (sample: VideoSample) => void;
  signal?: AbortSignal;
};
 
export async function extractFrames({src, timestampsInSeconds, onVideoSample, signal}: ExtractFramesProps): Promise<void> {
  const input = new Input({
    formats: ALL_FORMATS,
    source: new UrlSource(src),
  });
 
  const dispose = () => {
    input.dispose();
  };
 
  if (signal) {
    signal.addEventListener('abort', dispose, {once: true});
  }
 
  try {
    const [durationInSeconds, format, videoTrack] = await Promise.all([input.computeDuration(), input.getFormat(), input.getPrimaryVideoTrack()]);
    if (!videoTrack) {
      throw new Error('No video track found in the input');
    }
 
    const timestamps =
      typeof timestampsInSeconds === 'function'
        ? await timestampsInSeconds({
            track: {
              width: videoTrack.displayWidth,
              height: videoTrack.displayHeight,
            },
            container: format.name,
            durationInSeconds,
          })
        : timestampsInSeconds;
 
    if (timestamps.length === 0) {
      return;
    }
 
    const sink = new VideoSampleSink(videoTrack);
 
    for await (const videoSample of sink.samplesAtTimestamps(timestamps)) {
      if (signal?.aborted) {
        break;
      }
 
      if (!videoSample) {
        continue;
      }
 
      onVideoSample(videoSample);
    }
  } catch (error) {
    if (error instanceof InputDisposedError) {
      return;
    }
 
    throw error;
  } finally {
    dispose();
    if (signal) {
      signal.removeEventListener('abort', dispose);
    }
  }
}

Usage​

Basic example: Extract frames at specific times​

ts
await extractFrames({
  src: 'https://remotion.media/video.mp4',
  timestampsInSeconds: [0, 1, 2, 3, 4],
  onVideoSample: (sample) => {
    // Convert sample to VideoFrame
    const frame = sample.toVideoFrame();
    // Draw frame to canvas
    const canvas = document.createElement('canvas');
    canvas.width = frame.displayWidth;
    canvas.height = frame.displayHeight;
    const ctx = canvas.getContext('2d');
    ctx.drawImage(frame, 0, 0);
    // Don't forget to close the frame when done
    frame.close();
  },
});

Advanced: Create a filmstrip​

Extract as many frames as fit in a canvas based on the video's aspect ratio:

ts
const canvasWidth = 500;
const canvasHeight = 80;
const fromSeconds = 0;
const toSeconds = 10;
await extractFrames({
  src: 'https://example.com/video.mp4',
  timestampsInSeconds: async ({track, durationInSeconds}) => {
    const aspectRatio = track.width / track.height;
    const amountOfFramesFit = Math.ceil(canvasWidth / (canvasHeight * aspectRatio));
    const segmentDuration = toSeconds - fromSeconds;
    const timestamps: number[] = [];
    for (let i = 0; i < amountOfFramesFit; i++) {
      timestamps.push(fromSeconds + (segmentDuration / amountOfFramesFit) * (i + 0.5));
    }
    return timestamps;
  },
  onVideoSample: (sample) => {
    // Convert to VideoFrame if needed
    const frame = sample.toVideoFrame();
    // Process the frame
    console.log(`Frame at ${sample.timestamp}s`);
    // Clean up
    frame.close();
  },
});

Important notes​

Memory management​

The onVideoSample callback receives a VideoSample object. You need to convert it to a VideoFrame using .toVideoFrame(), and always close the VideoFrame when you're done with it to prevent memory leaks:

ts
onVideoSample: (sample) => {
  const frame = sample.toVideoFrame();
  // Use the frame
  ctx.drawImage(frame, 0, 0);
  // Clean up
  frame.close();
};

Abort handling​

Use an AbortSignal to cancel frame extraction:

ts
const controller = new AbortController();
// Cancel after 5 seconds
setTimeout(() => controller.abort(), 5000);
await extractFrames({
  src: 'https://example.com/video.mp4',
  timestampsInSeconds: [0, 1, 2, 3, 4],
  onVideoSample: (sample) => {
    const frame = sample.toVideoFrame();
    // Process frame
    frame.close();
  },
  signal: controller.signal,
});

See also​