> ## Documentation Index
> Fetch the complete documentation index at: https://vibes-sdk.a7ul.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Multimodal

> Send images, audio, video, and documents to models, and receive binary content from tools.

Vibes supports four input modalities  -  images, audio, video, and documents  -  through a set of message helper functions. Each helper wraps the raw binary or URL data into a `UserModelMessage` that the agent loop passes directly to the model. On the output side, tools can return `BinaryContent` (raw bytes with a MIME type) that flows back to the model as part of the conversation.

```typescript theme={null}
import { imageMessage, audioMessage, fileMessage } from "jsr:@vibesjs/sdk";
```

## Content type routing

```mermaid theme={null}
flowchart LR
    IMG[Image] --> IM[Image Message]
    AUD[Audio] --> AM[Audio Message]
    VID[Video] --> FM_V[File Message]
    DOC[Document] --> FM_D[File Message]
    IM --> UMM[User Message]
    AM --> UMM
    FM_V --> UMM
    FM_D --> UMM
    UMM --> MODEL[Model]
```

***

## Images

```typescript theme={null}
imageMessage(image: string | Uint8Array | URL, text?: string, mediaType?: string): UserModelMessage
```

The `image` parameter accepts a URL string, a base64-encoded string, or raw bytes. The optional `text` parameter adds a text prompt alongside the image. `mediaType` is optional — when omitted, no MIME type hint is sent and the provider infers the format from the content.

<CodeGroup>
  ```typescript URL string theme={null}
  import { imageMessage } from "jsr:@vibesjs/sdk";

  const msg = imageMessage(
    "https://example.com/photo.jpg",
    "Describe what you see in this image."
  );
  ```

  ```typescript Base64 string theme={null}
  import { imageMessage } from "jsr:@vibesjs/sdk";

  const base64 = fs.readFileSync("./screenshot.png").toString("base64");

  const msg = imageMessage(base64, "What is on this screen?", "image/png");
  ```

  ```typescript Uint8Array (raw bytes) theme={null}
  import { imageMessage } from "jsr:@vibesjs/sdk";

  const bytes = await fetch("https://example.com/chart.png").then((r) =>
    r.arrayBuffer().then((b) => new Uint8Array(b))
  );

  const msg = imageMessage(bytes, "Summarize this chart.", "image/png");
  ```
</CodeGroup>

<Info>
  When passing a base64 string, include only the encoded payload  -  not the `data:image/jpeg;base64,` prefix. Vibes sends the data part directly to the model provider.
</Info>

***

## Audio

```typescript theme={null}
audioMessage(audio: string | Uint8Array, mediaType: string, text?: string): UserModelMessage
```

Unlike `imageMessage`, the `mediaType` argument is **required** for audio  -  the runtime has no way to infer the codec from raw bytes.

```typescript theme={null}
import { audioMessage } from "jsr:@vibesjs/sdk";

const audioBytes = fs.readFileSync("./recording.mp3");
const base64Audio = audioBytes.toString("base64");

const msg = audioMessage(base64Audio, "audio/mpeg", "Transcribe this recording.");
```

<Warning>
  Audio modality support depends on the model provider. Check your provider's documentation to confirm which audio formats and codecs are accepted before sending audio messages.
</Warning>

***

## Video

Video follows the same shape as audio: use `fileMessage` with a `video/*` MIME type. Provider support varies.

```typescript theme={null}
import { fileMessage } from "jsr:@vibesjs/sdk";

const videoBytes = fs.readFileSync("./clip.mp4");
const base64Video = videoBytes.toString("base64");

const msg = fileMessage(base64Video, "video/mp4", "Describe the action in this clip.");
```

<Warning>
  Video support is experimental for most providers. Verify that your chosen model accepts `video/mp4` (or the specific video/\* MIME type) before sending video in production.
</Warning>

***

## Documents

Use `fileMessage` to send PDFs, plain text, or other document formats alongside a prompt.

```typescript theme={null}
fileMessage(data: string | Uint8Array, mediaType: string, text?: string): UserModelMessage
```

```typescript theme={null}
import { fileMessage } from "jsr:@vibesjs/sdk";

const pdfBytes = fs.readFileSync("./contract.pdf");
const base64Pdf = Buffer.from(pdfBytes).toString("base64");

const msg = fileMessage(
  base64Pdf,
  "application/pdf",
  "Summarize the key obligations in this contract."
);
```

Common document MIME types:

| Format     | `mediaType`         |
| ---------- | ------------------- |
| PDF        | `"application/pdf"` |
| Plain text | `"text/plain"`      |
| HTML       | `"text/html"`       |
| CSV        | `"text/csv"`        |

***

## UploadedFile

When a provider has already stored a file server-side (for example, via the Anthropic Files API), you reference it with an `UploadedFile` object rather than re-uploading the bytes.

```typescript theme={null}
import type { UploadedFile } from "jsr:@vibesjs/sdk";

const file: UploadedFile = {
  type: "uploaded_file",  // underscore  -  not a hyphen
  fileId: "file_abc123",
  mimeType: "application/pdf",
  filename: "contract.pdf",
};
```

<Warning>
  The `type` discriminant is `"uploaded_file"` with an **underscore**. Using `"uploaded-file"` (hyphen) will fail the type check and the provider will reject the request.
</Warning>

Use `uploadedFileSchema` when an agent tool needs to accept an `UploadedFile` as a parameter:

```typescript theme={null}
import { uploadedFileSchema, tool } from "jsr:@vibesjs/sdk";
import { z } from "zod";

const analyzeTool = tool({
  name: "analyze_file",
  description: "Analyze an already-uploaded file",
  parameters: z.object({
    file: uploadedFileSchema,
    question: z.string(),
  }),
  execute: async (_ctx, { file, question }) => {
    // file.fileId, file.mimeType, file.filename available here
    return `Analyzing ${file.filename ?? file.fileId} for: ${question}`;
  },
});
```

***

## Binary content from tools

Tools can return `BinaryContent`  -  raw bytes with a MIME type  -  which the agent loop forwards to the model as part of the next turn. This lets a tool produce an image, audio clip, or document that the model can reason about.

```typescript theme={null}
import { tool } from "jsr:@vibesjs/sdk";
import type { BinaryContent } from "jsr:@vibesjs/sdk";
import { z } from "zod";

const screenshotTool = tool({
  name: "take_screenshot",
  description: "Capture a screenshot of a URL and return it as an image",
  parameters: z.object({
    url: z.string().url(),
  }),
  execute: async (_ctx, { url }): Promise<BinaryContent> => {
    const imageBytes = await captureScreenshot(url); // your implementation
    return {
      type: "binary",
      data: imageBytes,
      mimeType: "image/png",
    };
  },
});
```

### Tool multi-modal return flow

```mermaid theme={null}
sequenceDiagram
    participant App
    participant Agent
    participant Tool
    participant Model

    App->>Agent: run with messages
    Agent->>Model: send conversation
    Model->>Agent: call screenshot tool
    Agent->>Tool: execute
    Tool-->>Agent: binary image data
    Agent->>Model: tool result with image
    Model-->>Agent: final response
    Agent-->>App: Result
```

<Info>
  To learn how tools integrate with the broader agent loop and how to compose toolsets, see [Tools](/concepts/tools).
</Info>

***

## Provider compatibility

The table below summarises which content types are accepted by major providers. Always verify your provider's current documentation before sending multimodal content in production — provider capabilities change frequently.

| Modality                   | Function                             | Anthropic (Claude)               | OpenAI (GPT-4o)            | Google (Gemini) |
| -------------------------- | ------------------------------------ | -------------------------------- | -------------------------- | --------------- |
| Image (URL)                | `imageMessage`                       | Yes                              | Yes                        | Yes             |
| Image (bytes / base64)     | `imageMessage`                       | Yes                              | Yes                        | Yes             |
| Audio                      | `audioMessage`                       | No                               | Yes (Whisper/GPT-4o-audio) | Yes             |
| Video                      | `fileMessage` with `video/*`         | No                               | No                         | Yes             |
| PDF / document             | `fileMessage` with `application/pdf` | Yes (Claude 3+ with PDF support) | No                         | Yes             |
| Binary tool result (image) | `BinaryContent` from `execute`       | Yes                              | Yes                        | Yes             |
| Uploaded file reference    | `UploadedFile` from `execute`        | Yes (Files API)                  | Yes (Files API)            | No              |

<Warning>
  Audio content is encoded as a `FilePart` (type `"file"`) because the AI SDK `UserContent` union does not include a separate audio part. Providers that support audio consume it via the file part with the appropriate `mediaType` (e.g. `"audio/wav"`, `"audio/mpeg"`).
</Warning>