Video Generation

Learn how to generate videos with the AI SDK.

Video Generation

Video generation is an experimental feature. The API may change in future versions.

The AI SDK provides the experimental_generateVideo function to generate videos based on a given prompt using a video model.

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
});

You can access the video data using the base64 or uint8Array properties:

const base64 = video.base64; // base64 video data
const uint8Array = video.uint8Array; // Uint8Array video data

Settings

Aspect Ratio

The aspect ratio is specified as a string in the format {width}:{height}. Models only support a few aspect ratios, and the supported aspect ratios are different for each model and provider.

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
  aspectRatio: '16:9',
});

Resolution

The resolution is specified as a string in the format {width}x{height}. Models only support specific resolutions, and the supported resolutions are different for each model and provider.

import { experimental_generateVideo as generateVideo } from 'ai';
import { google } from '@ai-sdk/google';

const { video } = await generateVideo({
  model: google.video('veo-2.0-generate-001'),
  prompt: 'A serene mountain landscape at sunset',
  resolution: '1280x720',
});

Duration

Some video models support specifying the duration of the generated video in seconds.

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A timelapse of clouds moving across the sky',
  duration: 5,
});

Frames Per Second (FPS)

Some video models allow you to specify the frames per second for the generated video.

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A hummingbird in slow motion',
  fps: 24,
});

Generating Multiple Videos

experimental_generateVideo supports generating multiple videos at once:

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { videos } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
  n: 2, // number of videos to generate
});

experimental_generateVideo will automatically call the model as often as needed (in parallel) to generate the requested number of videos.

Provider Options

Video model settings can be configured using providerOptions for provider-specific parameters:

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
  providerOptions: {
    fal: {
      loop: true, // provider-specific option
    },
  },
});

Response Information

The experimental_generateVideo function returns comprehensive response information:

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const result = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
});

console.log(result.videos); // Array of generated videos
console.log(result.warnings); // Any warnings from the provider
console.log(result.responses); // Raw provider responses
console.log(result.providerMetadata); // Provider-specific metadata

Retries

The experimental_generateVideo function accepts an optional maxRetries parameter that you can use to set the maximum number of retries. It defaults to 2 retries (3 attempts in total). You can set it to 0 to disable retries.

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
  maxRetries: 0, // Disable retries
});

Abort Signals and Timeouts

The experimental_generateVideo function accepts an optional abortSignal parameter of type AbortSignal that you can use to abort the generation process or set a timeout.

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
  abortSignal: AbortSignal.timeout(60000), // Abort after 60 seconds
});

Custom Headers

The experimental_generateVideo function accepts an optional headers parameter that you can use to add custom headers to the request.

import { experimental_generateVideo as generateVideo } from 'ai';
import { fal } from '@ai-sdk/fal';

const { video } = await generateVideo({
  model: fal.video('luma-dream-machine/ray-2'),
  prompt: 'A cat walking on a treadmill',
  headers: { 'X-Custom-Header': 'custom-value' },
});

Video Providers & Models

Several providers offer video generation models:

Provider	Model
fal.ai	luma-dream-machine/ray-2
fal.ai	runway-gen3/turbo/image-to-video
Google	veo-2.0-generate-001