Transcription

Learn how to transcribe audio with the AI SDK.

Transcription

Transcription is an experimental feature.

The AI SDK provides the transcribe function to transcribe audio using a transcription model.

import { experimental_transcribe as transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
});

The audio property can be a Uint8Array, ArrayBuffer, Buffer, string (base64 encoded audio data), or a URL.

To access the generated transcript:

const text = result.text; // transcript text e.g. "Hello, world!"
const segments = result.segments; // array of segments with start and end times, if available
const language = result.language; // language of the transcript e.g. "en", if available
const durationInSeconds = result.durationInSeconds; // duration of the transcript in seconds, if available

Settings

Provider-Specific Settings

Transcription models often have provider or model-specific settings which you can set using the providerOptions parameter.

import { experimental_transcribe as transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
  providerOptions: {
    openai: {
      timestampGranularities: ['word'],
    },
  },
});

Download Size Limits

When audio is a URL, the SDK downloads the file with a default 2 GiB size limit. You can customize this using createDownload:

import { experimental_transcribe as transcribe, createDownload } from 'ai';
import { openai } from '@ai-sdk/openai';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  download: createDownload({ maxBytes: 50 * 1024 * 1024 }), // 50 MB limit
});

You can also provide a fully custom download function:

import { experimental_transcribe as transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: new URL('https://example.com/audio.mp3'),
  download: async ({ url }) => {
    const res = await myAuthenticatedFetch(url);
    return {
      data: new Uint8Array(await res.arrayBuffer()),
      mediaType: res.headers.get('content-type') ?? undefined,
    };
  },
});

If a download exceeds the size limit, a DownloadError is thrown:

import { experimental_transcribe as transcribe, DownloadError } from 'ai';
import { openai } from '@ai-sdk/openai';

try {
  await transcribe({
    model: openai.transcription('whisper-1'),
    audio: new URL('https://example.com/large-audio.mp3'),
  });
} catch (error) {
  if (DownloadError.isInstance(error)) {
    console.error('Download failed:', error.message);
  }
}

Retries

The transcribe function accepts an optional maxRetries parameter that you can use to set the maximum number of retries. It defaults to 2 retries (3 attempts in total). You can set it to 0 to disable retries.

import { experimental_transcribe as transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
  maxRetries: 0, // Disable retries
});

Abort Signals and Timeouts

The transcribe function accepts an optional abortSignal parameter of type AbortSignal that you can use to abort the transcription process or set a timeout.

import { experimental_transcribe as transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
  abortSignal: AbortSignal.timeout(30000), // Abort after 30 seconds
});

Custom Headers

The transcribe function accepts an optional headers parameter that you can use to add custom headers to the request.

import { experimental_transcribe as transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
  headers: { 'X-Custom-Header': 'custom-value' },
});

Response Information

The transcribe function returns comprehensive response information:

import { experimental_transcribe as transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';

const result = await transcribe({
  model: openai.transcription('whisper-1'),
  audio: await readFile('audio.mp3'),
});

console.log(result.text); // Transcript text
console.log(result.segments); // Transcript segments with timestamps
console.log(result.language); // Detected language
console.log(result.durationInSeconds); // Audio duration
console.log(result.warnings); // Any warnings from the provider
console.log(result.responses); // Raw provider responses

Transcription Providers & Models

Several providers offer transcription models:

Provider	Model
OpenAI	whisper-1
Groq	whisper-large-v3
Groq	distil-whisper-large-v3-en