> ## Documentation Index
> Fetch the complete documentation index at: https://omnirelay.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Video understanding

> Ask a Gemini model about a video's picture and sound through the Gemini-native generateContent API, billed by tokens.

Gemini models can watch a video and answer questions about what is shown and said in it. OmniRelay serves them through the Gemini-native API, so Google's Gemini SDKs and existing `generateContent` requests work by changing only the base URL and the key:

* `POST https://api.omnirelay.ai/v1beta/models/{model}:generateContent` returns the whole answer as JSON.
* `POST https://api.omnirelay.ai/v1beta/models/{model}:streamGenerateContent?alt=sse` streams it as server-sent events.

Send your OmniRelay API key in the `x-goog-api-key` header, as the Gemini SDKs do, or as a Bearer token. The same key rules apply as for every other model: key expiry, [IP allowlist](https://omnirelay.ai/docs/overview/rate-limits.md#key-ip-allowlist), [model limits](https://omnirelay.ai/docs/overview/authentication.md#limit-a-key-to-some-models), budgets and [rate and concurrency limits](https://omnirelay.ai/docs/overview/rate-limits.md). A key in a `?key=` query parameter is not read.

## Models

| Model ID | Video input | Max output tokens |
| - | - | - |
| `gemini-3.1-pro-preview` | Picture and audio | 64,512 |

Other models answer this route with `unsupported_parameter` and nothing is charged. Text-only Gemini calls also work through [Chat Completions](https://omnirelay.ai/docs/llms/chat-completions.md).

## Analyse a video

Put the video in a `fileData` part with a public `https` URL, followed by your question:

```bash cURL theme={null}
curl "https://api.omnirelay.ai/v1beta/models/gemini-3.1-pro-preview:generateContent" \
  -H "x-goog-api-key: $OMNIRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "contents": [{
      "role": "user",
      "parts": [
        {"fileData": {"mimeType": "video/mp4", "fileUri": "https://example.com/clip.mp4"}},
        {"text": "Describe the video and its audio as JSON with summary, timeline (start_time, end_time, visual_description, audio_summary) and visible_text. Mark anything you cannot make out as unconfirmed."}
      ]
    }],
    "generationConfig": {"responseMimeType": "application/json", "maxOutputTokens": 8192}
  }'
```

```python Python theme={null}
from google import genai
from google.genai import types

client = genai.Client(
    api_key=OMNIRELAY_API_KEY,
    http_options=types.HttpOptions(base_url="https://api.omnirelay.ai"),
)
response = client.models.generate_content(
    model="gemini-3.1-pro-preview",
    contents=[
        types.Part.from_uri(file_uri="https://example.com/clip.mp4", mime_type="video/mp4"),
        "Describe the video and its audio as JSON with summary, timeline and visible_text.",
    ],
    config=types.GenerateContentConfig(response_mime_type="application/json", max_output_tokens=8192),
)
print(response.text)
```

The response is Gemini's own shape: the answer is in `candidates[0].content.parts`, and `usageMetadata` reports the tokens, including `promptTokensDetails` with the video's share.

### Video input

* **By URL**: `fileData.fileUri` must be a public `https` URL the model can download.
* **Inline**: a small clip can go in an `inlineData` part as base64. The whole request body is limited to 4 MB, so use a URL for anything larger.
* Gemini Files API URIs (`generativelanguage.googleapis.com/.../files/...`) and `cachedContent` are not supported.

The other fields of a `generateContent` request are passed to the model as you send them: `systemInstruction`, `generationConfig` (including `responseSchema` and `thinkingConfig`), `safetySettings`, and `tools` with `functionDeclarations` plus `toolConfig.functionCallingConfig`. Use camelCase field names. `candidateCount` must be 1, a `thinkingBudget` must fit within `maxOutputTokens`, and the built-in search, URL context, code execution and file search tools are not available on this route. Do not put `model` or `stream` in the body; the path sets both. If you leave out `generationConfig.maxOutputTokens`, OmniRelay sets the model's default of 32,000.

## Streaming

Use `:streamGenerateContent?alt=sse`. Each event is a JSON chunk in Gemini's shape, and every chunk carries the usage so far. The call is charged from the last chunk, after the model reports a finish reason. Streaming is the better choice for long videos: a non-streamed call must answer within 90 seconds, while a stream only needs its first chunk in that time.

## Pricing

Video understanding is billed by tokens at the model's [text prices](https://omnirelay.ai/docs/llms/listing-models.md). The video's frames and audio count as **input tokens**; the model's thinking counts as **output tokens**, as in Google's own billing. For `gemini-3.1-pro-preview`:

| | Input | Output (including thinking) |
| - | - | - |
| Up to 200,000 input tokens | USD 2 / M | USD 12 / M |
| Above 200,000 input tokens (whole request) | USD 4 / M | USD 18 / M |

A short clip with a detailed JSON answer is typically about **USD 0.01–0.02**. Your discount applies as for text models.

When the call starts, OmniRelay holds its maximum cost: the text of your request, **131,072 input tokens for each video by URL**, and the full `maxOutputTokens` plus 1,024 tokens, because a model may finish a few tokens past its limit. You pay only for the tokens reported; the rest of the hold returns to your balance when the call ends. Keep `maxOutputTokens` near what you need, so the hold stays small. A call whose usage goes well beyond its hold, such as a very long video, is held for review instead of being charged automatically.

The charge for each call is in the `X-OmniRelay-Cost-Microusd` response header of a non-streamed call, and on the [usage page](https://omnirelay.ai/docs/general/usage-billing.md) for both.

## Errors

Errors come back in Gemini's shape, so the Gemini SDKs raise them as API errors:

```json theme={null}
{"error": {"code": 400, "message": "This model does not support one of the request parameters. Nothing was charged.", "status": "unsupported_parameter", "param": "cachedContent"}, "request_id": "…"}
```

`code` is the HTTP status, `status` is the OmniRelay [error code](https://omnirelay.ai/docs/general/errors.md). If the model cannot fetch or read the video, the call fails with `invalid_parameter` and nothing is charged. In a stream, a failure arrives as a final event with the same `error` object.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.