> This page is for version v1.3 (default). > For other versions, use one of these documentation indexes: > - v1.3 (default): https://beta.docs.twelvelabs.io/v1.3/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://beta.docs.twelvelabs.io/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://beta.docs.twelvelabs.io/_mcp/server. # Create an async analysis task POST https://api.twelvelabs.io/v1.3/analyze/tasks Content-Type: application/json This method analyzes a video or one or more images asynchronously. Each request must contain a video or one or more images, but not both. The method supports two analysis modes: * **General analysis**: Prompt-based text generation with a video or one or more images. * **Video segmentation**: Timestamped metadata with custom segment definitions. **Videos** * The video can be up to 2 hours long, or up to 4 hours when you analyze only a portion of it. You can analyze between 1 second and 2 hours of the video. HLS and base64 videos are limited to 2 hours. * Formats: [FFmpeg supported formats](https://ffmpeg.org/ffmpeg-formats.html) * Resolution: 360x360 to 5184x2160 pixels * Aspect ratio: Between 1:1 and 1:2.4, or between 2.4:1 and 1:1. **Images** * You can provide one to twenty images per request. * Formats: JPEG, PNG, WebP, GIF, and BMP. * Maximum size: 20 MB per image. * Maximum pixel count: 16,777,216 pixels per image (width × height). **When to use this method**: * Generate custom text from your video or one or more images using a prompt (general analysis) * Extract timestamped metadata with custom segment definitions from your video * Analyze videos longer than 1 hour, or a portion of a video up to 4 hours long * Process videos or images asynchronously without blocking your application **Do not use this method for**: * Videos or images for which you need immediate results or real-time streaming. Use the [`POST`](/v1.3/api-reference/analyze-videos/sync-analysis) method of the `/analyze` endpoint instead. Analyzing content asynchronously requires three steps: 1. Create an analysis task using this method. The platform returns a task identifier. 2. Poll the status of the task using the [`GET`](/v1.3/api-reference/analyze-videos/retrieve-analysis-task-status-results) method of the `/analyze/tasks/{task_id}` endpoint. Wait until the status is `ready`, `failed`, or `canceled`. 3. When the status is `ready`, retrieve the results using the [`GET`](/v1.3/api-reference/analyze-videos/retrieve-analysis-task-status-results) method of the `/analyze/tasks/{task_id}` endpoint. On the Free plan, you have a total of 600 minutes (10 hours) shared across indexing, analysis, and segmentation. For details, see the [Video hours and video count limits](/v1.3/docs/concepts/indexes#video-hours-and-video-count-limits) section. This endpoint is rate-limited. For details, see the [Rate limits](/v1.3/docs/get-started/rate-limits) page. Reference: https://beta.docs.twelvelabs.io/api-reference/analyze-videos/create-async-analysis-task ## Authentication - `x-api-key` header (required) — Your API key. You can find your API key on the API Keys page. ## Request ### Body (application/json) This endpoint expects a CreateAsyncAnalyzeRequest. - `model_name` (enum, optional, default: pegasus1.5) — The video understanding model to use for analysis. - `pegasus1.6`: For details about this version, see the [Pegasus 1.6](/v1.3/docs/concepts/models/pegasus/pegasus-1-6) page. - `pegasus1.5`: For details about this version, see the [Pegasus 1.5](/v1.3/docs/concepts/models/pegasus/pegasus-1-5) page. **Default:** `pegasus1.5` - Allowed values: `pegasus1.6`, `pegasus1.5` - `custom_id` (string, optional) — An optional identifier that you set when you create the task. Use this field to correlate tasks across responses, for example, to distinguish tasks by type or environment. The platform stores this value unchanged and returns it in the following responses: * The [`GET`](/v1.3/api-reference/analyze-videos/retrieve-analysis-task-status-results) method of the `/analyze/tasks/{task_id}` endpoint * The [`GET`](/v1.3/api-reference/analyze-videos/list-async-analysis-tasks) method of the `/analyze/tasks` endpoint * The `analyze.task.ready`, `analyze.task.failed`, and `analyze.task.canceled` webhook payloads **Format**: 1–64 characters. Alphanumeric, hyphens (`-`), and underscores (`_`) only. An empty string is rejected with a `400 Bad Request`. This field does not enforce uniqueness. You can submit multiple tasks with the same `custom_id`. To prevent duplicate task creation, use an `Idempotency-Key` header instead. - `video` (VideoContext, optional) — An object specifying the source of the video content. Include exactly one source. Mutually exclusive with the `image` parameter. - `image` (list of AnalyzeImageInput, optional) — A list of up to twenty objects containing the images to analyze. For each image, include exactly one source. Requires Pegasus 1.6. Using any other model returns a `parameter_invalid` error. Mutually exclusive with the `video` and `prompt_v2` parameters. The `prompt` parameter is required when you provide images. - `prompt` (string, optional) — Natural-language instructions for analyzing the video or one or more images. Required for general analysis (prompt-based text generation). Not supported when `analysis_mode` is `time_based_metadata`. To include reference images in your prompt, use the `prompt_v2` parameter instead. Mutually exclusive with the `prompt_v2` parameter. Your prompts can be instructive or descriptive, or you can phrase them as questions. This text counts toward the [context window](/v1.3/docs/concepts/models/pegasus/pegasus-1-6#context-window). **Examples**: - Based on this video, I want to generate five keywords for SEO (Search Engine Optimization). - I want to generate a description for my video with the following format: Title of the video, followed by a summary in 2-3 sentences, highlighting the main topic, key events, and concluding remarks. - `prompt_v2` (AnalyzePromptV2, optional) — A structured prompt that uses `<@name>` placeholders to reference images. Mutually exclusive with the `prompt` and `image` parameters. The prompt text and reference images count toward the [context window](/v1.3/docs/concepts/models/pegasus/pegasus-1-6#context-window). - `analysis_mode` (enum, optional, default: general) — The analysis mode for this task. - `general`: Analyze the video and generate a response based on your prompt. Supports both free-form text and structured output via `response_format`. - `time_based_metadata`: Segment the video into time-based intervals and extract custom metadata for each segment. Requires `response_format.type` set to `segment_definitions`. **Default:** `general` - Allowed values: `general`, `time_based_metadata` - `temperature` (double, optional) — Controls the randomness of the text output. **Default:** 0.2 **Min:** 0 **Max:** 1 - `max_tokens` (CreateAsyncAnalyzeRequestMaxTokens, optional) — The maximum response length, in tokens. Provide an integer or the `unlimited` value. The allowed integer range and default depend on the analysis mode: | Mode | Min | Max | Default | |------|-----|-----|---------| | `general` | 512 | 98,304 | 4,096 | | `time_based_metadata` | 2,048 | 98,304 | 32,768 | - **Integer**: With video segmentation, the task fails if the output exceeds the limit. No partial output is returned. - **Unlimited**: Removes the token limit from video segmentation. It requires `analysis_mode` set to `time_based_metadata` and `model_name` set to `pegasus1.6`. Mutually exclusive with `response_format.segment_definitions[].time_ranges`. The platform extracts as many segments as it can. If it stops before extracting every segment, the task still completes with the `status` field set to `ready`. The `error` field contains the warning that the results may be incomplete. - `response_format` (AsyncResponseFormat, optional) — Controls the response format. When you omit this parameter, you receive unstructured text. - `json_schema`: Return structured JSON that conforms to your schema. - `segment_definitions`: Extract timestamped metadata with custom fields from your video. Requires `analysis_mode` set to `time_based_metadata`. - `min_segment_duration` (double, optional) — Minimum duration for each extracted segment, in seconds. Set this value to enforce a minimum segment length. Requires `analysis_mode` set to `time_based_metadata`. Mutually exclusive with `response_format.segment_definitions[].time_ranges`. **Min:** 2 - `max_segment_duration` (double, optional) — Maximum duration for each extracted segment, in seconds. Set this value to split long continuous sections into shorter segments. Must be greater than or equal to `min_segment_duration`. Requires `analysis_mode` set to `time_based_metadata`. Mutually exclusive with `response_format.segment_definitions[].time_ranges`. **Min:** 2 - `start_time` (double, optional) — Start of the analysis window, as an absolute timestamp in seconds, based on the internal metadata of the video. Use with `end_time` to analyze only a portion of the video. * If omitted, defaults to the internal start time of the video. * Most videos start at 0, but some (for example, from cameras or broadcast recordings) may have a non-zero start time. To find the value, run `ffprobe -v error -show_entries format=start_time,duration -of default=noprint_wrappers=1 your_video.mp4`. * Must be less than `end_time` and the video duration. * The window (`end_time - start_time`) must be at least 1 second and at most 2 hours. The video may be up to 4 hours as long as the window stays within that limit. * Mutually exclusive with `response_format.segment_definitions[].time_ranges`. * Together with `end_time`, this parameter determines the billable video duration. If you omit both, billing uses the full video duration. For details, see the [Frequently asked questions](/v1.3/docs/resources/frequently-asked-questions#how-is-video-segmentation-priced) page. - `end_time` (double, optional) — End of the analysis window, as an absolute timestamp in seconds, based on the internal metadata of the video. Use with `start_time` to analyze only a portion of the video. * If omitted, defaults to the internal start time of the video plus its duration. * Most videos start at 0, but some (for example, from cameras or broadcast recordings) may have a non-zero start time. To find the value, run `ffprobe -v error -show_entries format=start_time,duration -of default=noprint_wrappers=1 your_video.mp4`. * Must be greater than `start_time` and less than or equal to the video duration. * The window (`end_time - start_time`) must be at least 1 second and at most 2 hours. The video may be up to 4 hours as long as the window stays within that limit. * Mutually exclusive with `response_format.segment_definitions[].time_ranges`. * Together with `start_time`, this parameter determines the billable video duration. If you omit both, billing uses the full video duration. For details, see the [Frequently asked questions](/v1.3/docs/resources/frequently-asked-questions#how-is-video-segmentation-priced) page. ## Response ### 202 Analysis task created successfully. - `task_id` (string, required) — The unique identifier of the analysis task. - `status` (enum, required) — The current status of the analysis task. The `ready`, `failed`, and `canceled` statuses are final. - Allowed values: `queued`, `pending`, `processing`, `ready`, `failed`, `canceled` ## Errors ### 400 Bad Request Error Validation failure or inaccessible video - `error` (ErrorResponseError, required) ### 500 Internal Server Error Internal server error - `error` (ErrorResponseError, required) ## Types ### VideoContext An object specifying the source of the video content. Include exactly one source. Mutually exclusive with the `image` parameter. - `type`: `url` (url) - `url` (string, required) — The publicly accessible URL of the video file or HLS manifest. Use direct links to raw media files, or the URL of a VOD HLS manifest. Live video streams are rejected with a `400` error. Video hosting platforms and cloud storage sharing links are not supported. For HLS sources, if the duration cannot be determined from the media, the platform calculates it from the manifest and the duration limits apply to that value. - `type`: `asset_id` (asset_id) - `asset_id` (string, required) — The unique identifier of an asset from a [direct](/v1.3/api-reference/upload-content/direct-uploads) or [multipart](/v1.3/api-reference/upload-content/multipart-uploads) upload. The asset status must be `ready`. Use the [Retrieve an asset](/v1.3/api-reference/upload-content/direct-uploads/retrieve) method to check the status. - `type`: `base64_string` (base64_string) - `base64_string` (string, required) — The base64-encoded video data. The maximum size is 30MB. ### AnalyzeImageInput An image to analyze. Provide exactly one of `url`, `asset_id`, or `base64_string`. Each image can be up to 20 MB and 16,777,216 pixels (width × height). Supported formats: JPEG, PNG, WebP, GIF, and BMP. An image over either limit, or in another format, returns a `400` error. - `name` (string, required) — The name of this image. Must be unique within the `image` array. Can contain only letters, digits, and underscores. To refer to a specific image in your prompt, use its name as the `<@name>` placeholder. Reference every image or none of them: * If the prompt references every image, the platform analyzes each one (Example: `Compare <@before> with <@after> and describe what changed.`). * If the prompt references none of them, the platform analyzes every image you provided (Example: `List every visible feature in these images.`). * If the prompt references only some of the images, the platform returns a `400` error. - `url` (string, optional) — A publicly accessible HTTPS URL of the image. The platform rejects HTTP URLs. When you send the request, the platform verifies that the URL is reachable and that the image is within the [size, pixel count, and format limits](/v1.3/docs/concepts/models/pegasus/pegasus-1-6#image-file-requirements). If either check fails, the platform returns a `400` error. - `asset_id` (string, optional) — The unique identifier of an uploaded asset. - `base64_string` (string, optional) — Base64-encoded image data. The maximum decoded size is 20 MB. Supported image types: `image/jpeg`, `image/png`, `image/webp`, `image/gif`, and `image/bmp`. Standard and unpadded base64 encodings are both accepted. ### AnalyzePromptV2 A structured prompt that uses `<@name>` placeholders to reference images. Not supported when the `analysis_mode` parameter is `time_based_metadata`. Mutually exclusive with the `prompt` and `image` parameters. - `input_text` (string, required) — The text of the prompt. Use `<@name>` placeholders to reference images declared in `media_sources` (Example: `"Is there a <@tiger-1> in the video?"`). This text counts toward the [context window](/v1.3/docs/concepts/models/pegasus/pegasus-1-6#context-window). - `media_sources` (list of SMEMediaSource, optional) — Reference images for the `<@name>` placeholders in the prompt. Maximum 4 sources. ### CreateAsyncAnalyzeRequestMaxTokens The maximum response length, in tokens. Provide an integer or the `unlimited` value. The allowed integer range and default depend on the analysis mode: | Mode | Min | Max | Default | |------|-----|-----|---------| | `general` | 512 | 98,304 | 4,096 | | `time_based_metadata` | 2,048 | 98,304 | 32,768 | - **Integer**: With video segmentation, the task fails if the output exceeds the limit. No partial output is returned. - **Unlimited**: Removes the token limit from video segmentation. It requires `analysis_mode` set to `time_based_metadata` and `model_name` set to `pegasus1.6`. Mutually exclusive with `response_format.segment_definitions[].time_ranges`. The platform extracts as many segments as it can. If it stops before extracting every segment, the task still completes with the `status` field set to `ready`. The `error` field contains the warning that the results may be incomplete. ### AsyncResponseFormat Controls the response format. When you omit this parameter, you receive unstructured text. - `json_schema`: Return structured JSON that conforms to your schema. - `segment_definitions`: Extract timestamped metadata with custom fields from your video. Requires `analysis_mode` set to `time_based_metadata`. - `type` (enum, required) — The response format to use. - `json_schema`: Return structured JSON that conforms to your schema. - `segment_definitions`: Extract timestamped metadata with custom fields from your video. Requires `analysis_mode` set to `time_based_metadata`. - Allowed values: `json_schema`, `segment_definitions` - `json_schema` (AsyncResponseFormatJsonSchema, optional) — Contains the JSON schema that defines the response structure. The schema must adhere to the [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) specification. **Supported types** * `array` * `boolean` * `integer` * `null` * `number` * `object` * `string` * `timestamp` **Supported constraints** | Type | Supported keywords | Notes | | ----------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `integer` | `maximum`, `exclusiveMaximum`, `minimum`, `exclusiveMinimum`. | - `maximum`: Sets the highest allowed value (inclusive).- `exclusiveMaximum`: Sets the highest allowed value (exclusive).- `minimum`: Sets the lowest allowed value (inclusive).- `exclusiveMinimum`: Sets the lowest allowed value (exclusive).These constraints are supported only for the `integer` type. | | `string` | `pattern`, `format` | - `pattern`: A regular expression that the string must match.- `format`: Validates predefined formats. It accepts the following values: `uuid`, `date-time`, `date`, and `time`.See string limitations below. | | `object` | `properties`, `required` | - `properties`: Defines object properties and their schemas.- `required`: Specifies mandatory properties.See object limitations below. | | `array` | `items`, `minItems` | `minItems` accepts only `0` or `1`.See array limitations below. | | `timestamp` | `format` | `format` (required): Sets the output format. Accepted values: `seconds`, `hh:mm:ss`, `hh:mm:ss.fff`.See the **Timestamp type** section below. | **String limitations** When you use the `string` type: * The platform validates strings using only `pattern` and `format`. Including `minLength` or `maxLength` causes a 422 error: "String length constraints (minLength) are not supported." Remove these keywords from your schema. **Object limitations** When you use the `object` type: * The platform does not support the `additionalProperties` keyword. Including it causes a 422 error. Remove it from your schema. * The platform returns properties in declaration order. * Make the first property required. If the first property is optional, the platform moves the first required property to the beginning. **Array limitations** When you use the `array` type: * The platform does not support `uniqueItems` or `maxItems`. Including either keyword causes a 422 error. Remove them from your schema. **Constant and enumerated values** The `const` and `enum` keywords support the following types: * `boolean` * `null` * `number` * `string` **Schema composition** The platform supports only `anyOf` for [schema composition](https://json-schema.org/understanding-json-schema/reference/combining). **Annotations** The platform accepts but ignores JSON schema annotations like `title`, `$comments`, and `description`. **Subschema references** You can reference subschemas using `$ref` with these requirements: * Define subschemas within `$defs`. * Use valid URIs that point to the internal subschema. For details, see the [JSON Schema documentation on \$defs](https://json-schema.org/understanding-json-schema/structuring#defs). **Timestamp type** Declare a property as `{"type": "timestamp", "format": ""}` to control the format of the returned value. The `format` field accepts the following values: | `format` | Example output | Notes | | -------------- | ---------------- | ----------------------------------------------------------------------------- | | `seconds` | `10.5` | Returns a JSON number in seconds. | | `hh:mm:ss` | `"00:01:23"` | Rounded to the nearest second. Negative values are converted to `"00:00:00"`. | | `hh:mm:ss.fff` | `"00:01:23.500"` | Millisecond precision. | The type of the response depends on the value of the `format` field: `seconds` returns a JSON number, while `hh:mm:ss` and `hh:mm:ss.fff` return a JSON string. *Supported positions* You can declare `timestamp` fields at the top level of your schema or inside objects nested one level within an array: * Top level: `properties.` * Inside an array: `properties..items.properties.` Declaring `timestamp` outside these positions — deeper nesting, inside `oneOf` / `anyOf` / `allOf`, or inside `$ref` — is not supported and is rejected with HTTP 400. *Validation errors* When `format` is missing or invalid, the platform returns `400 parameter_invalid`: ``` response_format.json_schema.properties..format: format is required for timestamp type; allowed values: seconds, hh:mm:ss, hh:mm:ss.fff ``` **Reserved property names (`start_time` / `end_time`)** The `start_time` and `end_time` properties in your response schema receive special type handling at any nesting depth (including inside array `items`). These are unrelated to the top-level `start_time` / `end_time` request parameters or `time_ranges`. The platform returns the value in a format determined by the declared type: *Allowed declarations:* | Declared type | Platform behavior | | ------------------------- | ------------------------------------------------------------------- | | `number` | Passes the value through without conversion. | | `integer` | Rounds the value to the nearest integer. | | `string` (no `format`) | Converts the value to the `hh:mm:ss.fff` format. | | `timestamp` with `format` | See the **Timestamp type** section above for the available formats. | *Rejected declarations (returns `400` error):* * `string` with any `format` keyword (`time`, `date-time`, `email`, `uri`, etc.) * `boolean` * `object` * `array` * `null` All other property names in your schema remain unconstrained by these rules. For other field names, use the `timestamp` type described above. **Response validation** Check the `finish_reason` field to verify your JSON response is complete: * When `finish_reason` is `stop`, the generation completed normally, and the JSON is valid and complete. * When `finish_reason` is `length`, the generation reached the maximum response length or the context window. The output may be truncated and fail to parse. - `segment_definitions` (list of SegmentDefinition, optional) — Define the types of segments to extract from your video. Minimum 1, maximum 20 definitions. The number of segment definitions affects billing. For details, see the [Frequently asked questions](/v1.3/docs/resources/frequently-asked-questions#how-is-video-segmentation-priced) page. - `segment_time_format` (enum, optional, default: seconds) — Set the output format for the automatic `start_time` and `end_time` keys returned on each segment. Requires the `type` parameter set to `segment_definitions`. Omitting this parameter is equivalent to setting it to `seconds`. Both return JSON numbers in seconds. | `segment_time_format` | Auto boundary output | |-----------------------|----------------------| | `seconds` (default) | JSON number in seconds (Example: `12.5`) | | `hh:mm:ss` | JSON string (Example: `"00:00:13"`) — rounded to the nearest second | | `hh:mm:ss.fff` | JSON string (Example: `"00:00:12.500"`) — millisecond precision | This parameter applies only to the automatic segment boundaries (`start_time` and `end_time`). For a `time_array` field, it also formats the `start_time` and `end_time` of each event inside the segment. Custom `timestamp` fields always use their own format, regardless of the value of this field. - Allowed values: `seconds`, `hh:mm:ss`, `hh:mm:ss.fff` ### ErrorResponseError - `code` (string, required) — A string representing the code associated with the error. See the [Error codes](/v1.3/api-reference/error-codes) page for details. - `message` (string, required) — A human-readable string describing the error, intended to be suitable for display in a user interface. - `details` (map from string to any, optional) — Additional error details (optional) ### SMEMediaSource A reference image that provides visual context for segment identification. Provide exactly one of `url`, `asset_id`, or `base64_string`. - `name` (string, required) — A descriptive name for this media source. - `media_type` (enum, required) — The media type. Only `image` is available. - Allowed values: `image` - `url` (string, optional) — A publicly accessible HTTPS URL of the image. - `asset_id` (string, optional) — The unique identifier of an uploaded asset. - `base64_string` (string, optional) — Base64-encoded image data. The maximum decoded size is 32 MB. ### AsyncResponseFormatJsonSchema Contains the JSON schema that defines the response structure. The schema must adhere to the [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12) specification. **Supported types** * `array` * `boolean` * `integer` * `null` * `number` * `object` * `string` * `timestamp` **Supported constraints** | Type | Supported keywords | Notes | | ----------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `integer` | `maximum`, `exclusiveMaximum`, `minimum`, `exclusiveMinimum`. | - `maximum`: Sets the highest allowed value (inclusive).- `exclusiveMaximum`: Sets the highest allowed value (exclusive).- `minimum`: Sets the lowest allowed value (inclusive).- `exclusiveMinimum`: Sets the lowest allowed value (exclusive).These constraints are supported only for the `integer` type. | | `string` | `pattern`, `format` | - `pattern`: A regular expression that the string must match.- `format`: Validates predefined formats. It accepts the following values: `uuid`, `date-time`, `date`, and `time`.See string limitations below. | | `object` | `properties`, `required` | - `properties`: Defines object properties and their schemas.- `required`: Specifies mandatory properties.See object limitations below. | | `array` | `items`, `minItems` | `minItems` accepts only `0` or `1`.See array limitations below. | | `timestamp` | `format` | `format` (required): Sets the output format. Accepted values: `seconds`, `hh:mm:ss`, `hh:mm:ss.fff`.See the **Timestamp type** section below. | **String limitations** When you use the `string` type: * The platform validates strings using only `pattern` and `format`. Including `minLength` or `maxLength` causes a 422 error: "String length constraints (minLength) are not supported." Remove these keywords from your schema. **Object limitations** When you use the `object` type: * The platform does not support the `additionalProperties` keyword. Including it causes a 422 error. Remove it from your schema. * The platform returns properties in declaration order. * Make the first property required. If the first property is optional, the platform moves the first required property to the beginning. **Array limitations** When you use the `array` type: * The platform does not support `uniqueItems` or `maxItems`. Including either keyword causes a 422 error. Remove them from your schema. **Constant and enumerated values** The `const` and `enum` keywords support the following types: * `boolean` * `null` * `number` * `string` **Schema composition** The platform supports only `anyOf` for [schema composition](https://json-schema.org/understanding-json-schema/reference/combining). **Annotations** The platform accepts but ignores JSON schema annotations like `title`, `$comments`, and `description`. **Subschema references** You can reference subschemas using `$ref` with these requirements: * Define subschemas within `$defs`. * Use valid URIs that point to the internal subschema. For details, see the [JSON Schema documentation on \$defs](https://json-schema.org/understanding-json-schema/structuring#defs). **Timestamp type** Declare a property as `{"type": "timestamp", "format": ""}` to control the format of the returned value. The `format` field accepts the following values: | `format` | Example output | Notes | | -------------- | ---------------- | ----------------------------------------------------------------------------- | | `seconds` | `10.5` | Returns a JSON number in seconds. | | `hh:mm:ss` | `"00:01:23"` | Rounded to the nearest second. Negative values are converted to `"00:00:00"`. | | `hh:mm:ss.fff` | `"00:01:23.500"` | Millisecond precision. | The type of the response depends on the value of the `format` field: `seconds` returns a JSON number, while `hh:mm:ss` and `hh:mm:ss.fff` return a JSON string. *Supported positions* You can declare `timestamp` fields at the top level of your schema or inside objects nested one level within an array: * Top level: `properties.` * Inside an array: `properties..items.properties.` Declaring `timestamp` outside these positions — deeper nesting, inside `oneOf` / `anyOf` / `allOf`, or inside `$ref` — is not supported and is rejected with HTTP 400. *Validation errors* When `format` is missing or invalid, the platform returns `400 parameter_invalid`: ``` response_format.json_schema.properties..format: format is required for timestamp type; allowed values: seconds, hh:mm:ss, hh:mm:ss.fff ``` **Reserved property names (`start_time` / `end_time`)** The `start_time` and `end_time` properties in your response schema receive special type handling at any nesting depth (including inside array `items`). These are unrelated to the top-level `start_time` / `end_time` request parameters or `time_ranges`. The platform returns the value in a format determined by the declared type: *Allowed declarations:* | Declared type | Platform behavior | | ------------------------- | ------------------------------------------------------------------- | | `number` | Passes the value through without conversion. | | `integer` | Rounds the value to the nearest integer. | | `string` (no `format`) | Converts the value to the `hh:mm:ss.fff` format. | | `timestamp` with `format` | See the **Timestamp type** section above for the available formats. | *Rejected declarations (returns `400` error):* * `string` with any `format` keyword (`time`, `date-time`, `email`, `uri`, etc.) * `boolean` * `object` * `array` * `null` All other property names in your schema remain unconstrained by these rules. For other field names, use the `timestamp` type described above. **Response validation** Check the `finish_reason` field to verify your JSON response is complete: * When `finish_reason` is `stop`, the generation completed normally, and the JSON is valid and complete. * When `finish_reason` is `length`, the generation reached the maximum response length or the context window. The output may be truncated and fail to parse. ### SegmentDefinition Defines a type of segment to extract from the video. - `id` (string, required) — A unique identifier for this segment definition. - `description` (string, required) — Describe what this type of segment looks like in the video. The model uses this text to identify matching segments. - `fields` (list of SegmentField, optional) — Custom fields to extract for each segment instance. Set the `type` field of an item to the `time_array` value to extract a list of events inside each segment. - `media_sources` (list of SMEMediaSource, optional) — Reference images that help the model identify segments. Maximum 4 sources. - `time_ranges` (list of AnalyzeTimeRange, optional) — Time windows that limit segment extraction to specific parts of the video. Only supported when `analysis_mode` is set to `time_based_metadata`. * Each range must satisfy `end_time > start_time` with a minimum duration of `2` seconds. Both values must fall within the video duration. * Ranges within a single definition must not overlap. Touching boundaries are allowed (Example: `[0, 5]` and `[5, 10]`). * Mutually exclusive with the top-level `start_time` / `end_time` fields. * Mutually exclusive with `min_segment_duration` and `max_segment_duration`. * These time ranges control which portions of the `start_time`–`end_time` window are analyzed; the billable duration is always the full `start_time`–`end_time` span. ### SegmentField A custom field to extract for each segment. **Timestamp fields** Set `type` to `timestamp` and provide a `format` to control the format of the returned value on each segment. See the `format` property for supported values. Each segment includes automatic `start_time` and `end_time` keys (floats in seconds) that mark the segment boundary. These names, along with `metadata`, are reserved and cannot be used as the name of a segment field or as the name of a field inside a `time_array` item. - `name` (string, required) — The name of the field. - `type` (enum, required) — The data type of the field. When set to `timestamp`, the `format` property is required and controls the format of the returned value. When set to the `time_array` value, the field extracts a list of events inside each segment. Requires the `items.type` field to be `object` and a non-empty `items.fields` list. Requires Pegasus 1.6. Any other model rejects `time_array` as an invalid type and lists only the types it accepts. The `/analyze/batches` endpoint does not accept Pegasus 1.6, so it does not accept `time_array` either. - Allowed values: `string`, `boolean`, `number`, `integer`, `array`, `timestamp`, `time_array` - `description` (string, required) — Instructions that guide the model on what this field should contain and how to extract it from the video. - `format` (enum, optional) — The output format for `timestamp` fields. Required when `type` is `timestamp`. Must be omitted for any other type. | `format` | Example output | |----------|----------------| | `seconds` | `10.5` (JSON number in seconds) | | `hh:mm:ss` | `"00:01:23"` (rounded to the nearest second; negative values are converted to `"00:00:00"`) | | `hh:mm:ss.fff` | `"00:01:23.500"` (millisecond precision) | *Validation errors* The platform returns `400 parameter_invalid` (with field path `response_format.segment_definitions.fields.format`) when: - `type` is `timestamp` and `format` is missing, empty, or not one of the supported values. - `type` is not `timestamp` and `format` is set. - Allowed values: `seconds`, `hh:mm:ss`, `hh:mm:ss.fff` - `enum` (list of string, optional) — Allowed values for this field. Maximum 100 values. Not supported when `type` is `timestamp`. - `items` (SegmentFieldItems, optional) — Required when `type` is `array` or `time_array`. For `array`, specifies the type of array elements. For `time_array`, set it to `{"type": "object", "fields": [...]}`. Not supported when `type` is `timestamp`. ### AnalyzeTimeRange A time window within the video, expressed in seconds. - `start_time` (double, required) — The start of the window, as an absolute timestamp in seconds, based on the internal metadata of the video. Must be less than `end_time` and within the video duration. Most videos start at 0, but some (for example, from cameras or broadcast recordings) may have a non-zero start time. To find the value, run `ffprobe -v error -show_entries format=start_time,duration -of default=noprint_wrappers=1 your_video.mp4`. - `end_time` (double, required) — The end of the window, as an absolute timestamp in seconds, based on the internal metadata of the video. Must be greater than `start_time` by at least 2 seconds and within the video duration. Most videos start at 0, but some (for example, from cameras or broadcast recordings) may have a non-zero start time. To find the value, run `ffprobe -v error -show_entries format=start_time,duration -of default=noprint_wrappers=1 your_video.mp4`. ### SegmentFieldItems The per-item structure of a segment field whose `type` is `array` or `time_array`. For an `array` field, the `type` value specifies the scalar element type. For a `time_array` field, use `object` as the `type` value and declare the fields of each event in the `fields` array. - `type` (enum, required) — The item type. For an `array` field, use one of the scalar types (`string`, `number`, `boolean`, `integer`). For a `time_array` field, use `object` and declare the per-event schema in the `fields` array. Setting `object` as the `type` value on a plain `array` field returns a `400` error. Setting the `fields` array on a plain `array` field also returns a `400` error. - Allowed values: `string`, `number`, `boolean`, `integer`, `object` - `fields` (list of TimeArrayItemField, optional) — The fields to extract from each event. Required when the `type` of the parent field is `time_array`, with at least one entry. Not permitted on any other field type. ### TimeArrayItemField A field inside a `time_array` item. The platform adds `start_time` and `end_time` keys to every event automatically. You cannot declare fields with these names. A `time_array` field cannot contain another `time_array` field. The `format` property is not accepted on per-event fields. - `name` (string, required) — The name of the field inside each event object. Must be unique within the item. The names `start_time`, `end_time`, and `metadata` are reserved and rejected with a `400` error. - `type` (enum, required) — The data type of this field. It does not accept `timestamp` or `time_array`. When set to the `array` value, the `items` object is required. - Allowed values: `string`, `boolean`, `number`, `integer`, `array` - `description` (string, required) — Instructions that guide the model on what this field should contain in each event. - `enum` (list of string, optional) — Allowed values for this field. Maximum 100 values. Only supported when `type` is `string`; any other type returns a `400` error. - `items` (TimeArrayItemFieldItems, optional) — The element type for an `array` field. Required when `type` is `array`; not supported on any other type. ### TimeArrayItemFieldItems The element type for an `array` field. Required when `type` is `array`; not supported on any other type. - `type` (enum, required) — The type of the array elements. It must be a scalar (`string`, `number`, `boolean`, or `integer`). - Allowed values: `string`, `number`, `boolean`, `integer` ## Examples ### analyzeAsync_tasks_create_example **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python analyzeAsync_tasks_create_example import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" headers = {"x-api-key": ""} response = requests.post(url, headers=headers) print(response.json()) ``` ```javascript analyzeAsync_tasks_create_example const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = {method: 'POST', headers: {'x-api-key': ''}}; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go analyzeAsync_tasks_create_example package main import ( "fmt" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" req, _ := http.NewRequest("POST", url, nil) req.Header.Add("x-api-key", "") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby analyzeAsync_tasks_create_example require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' response = http.request(request) puts response.read_body ``` ```java analyzeAsync_tasks_create_example import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .asString(); ``` ```php analyzeAsync_tasks_create_example request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'headers' => [ 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp analyzeAsync_tasks_create_example using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); IRestResponse response = client.Execute(request); ``` ```swift analyzeAsync_tasks_create_example import Foundation let headers = ["x-api-key": ""] let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Analyze video from URL **Request** ```json { "custom_id": "prod-segment-analysis-42", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Generate a detailed summary of this video in 3-4 sentences", "temperature": 0.2, "max_tokens": 1000 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Analyze video from URL import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "custom_id": "prod-segment-analysis-42", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Generate a detailed summary of this video in 3-4 sentences", "temperature": 0.2, "max_tokens": 1000 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Analyze video from URL const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"custom_id":"prod-segment-analysis-42","video":{"type":"url","url":"https://example.com/video.mp4"},"prompt":"Generate a detailed summary of this video in 3-4 sentences","temperature":0.2,"max_tokens":1000}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Analyze video from URL package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"custom_id\": \"prod-segment-analysis-42\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Generate a detailed summary of this video in 3-4 sentences\",\n \"temperature\": 0.2,\n \"max_tokens\": 1000\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Analyze video from URL require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"custom_id\": \"prod-segment-analysis-42\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Generate a detailed summary of this video in 3-4 sentences\",\n \"temperature\": 0.2,\n \"max_tokens\": 1000\n}" response = http.request(request) puts response.read_body ``` ```java Analyze video from URL import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"custom_id\": \"prod-segment-analysis-42\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Generate a detailed summary of this video in 3-4 sentences\",\n \"temperature\": 0.2,\n \"max_tokens\": 1000\n}") .asString(); ``` ```php Analyze video from URL request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "custom_id": "prod-segment-analysis-42", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Generate a detailed summary of this video in 3-4 sentences", "temperature": 0.2, "max_tokens": 1000 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Analyze video from URL using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"custom_id\": \"prod-segment-analysis-42\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Generate a detailed summary of this video in 3-4 sentences\",\n \"temperature\": 0.2,\n \"max_tokens\": 1000\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Analyze video from URL import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "custom_id": "prod-segment-analysis-42", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "prompt": "Generate a detailed summary of this video in 3-4 sentences", "temperature": 0.2, "max_tokens": 1000 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Extract timestamped metadata with custom fields **Request** ```json { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scene", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "sentiment", "type": "string", "description": "The emotional tone of this segment", "enum": [ "positive", "negative", "neutral" ] }, { "name": "key_objects", "type": "array", "description": "Notable objects visible in this segment", "items": { "type": "string" } } ] } ] }, "min_segment_duration": 5, "max_segment_duration": 30 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Extract timestamped metadata with custom fields import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scene", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "sentiment", "type": "string", "description": "The emotional tone of this segment", "enum": ["positive", "negative", "neutral"] }, { "name": "key_objects", "type": "array", "description": "Notable objects visible in this segment", "items": { "type": "string" } } ] } ] }, "min_segment_duration": 5, "max_segment_duration": 30 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Extract timestamped metadata with custom fields const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.5","video":{"type":"url","url":"https://example.com/video.mp4"},"analysis_mode":"time_based_metadata","response_format":{"type":"segment_definitions","segment_definitions":[{"id":"scene","description":"A distinct scene or setting change in the video","fields":[{"name":"sentiment","type":"string","description":"The emotional tone of this segment","enum":["positive","negative","neutral"]},{"name":"key_objects","type":"array","description":"Notable objects visible in this segment","items":{"type":"string"}}]}]},"min_segment_duration":5,"max_segment_duration":30}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Extract timestamped metadata with custom fields package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scene\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"sentiment\",\n \"type\": \"string\",\n \"description\": \"The emotional tone of this segment\",\n \"enum\": [\n \"positive\",\n \"negative\",\n \"neutral\"\n ]\n },\n {\n \"name\": \"key_objects\",\n \"type\": \"array\",\n \"description\": \"Notable objects visible in this segment\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n ]\n },\n \"min_segment_duration\": 5,\n \"max_segment_duration\": 30\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Extract timestamped metadata with custom fields require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scene\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"sentiment\",\n \"type\": \"string\",\n \"description\": \"The emotional tone of this segment\",\n \"enum\": [\n \"positive\",\n \"negative\",\n \"neutral\"\n ]\n },\n {\n \"name\": \"key_objects\",\n \"type\": \"array\",\n \"description\": \"Notable objects visible in this segment\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n ]\n },\n \"min_segment_duration\": 5,\n \"max_segment_duration\": 30\n}" response = http.request(request) puts response.read_body ``` ```java Extract timestamped metadata with custom fields import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scene\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"sentiment\",\n \"type\": \"string\",\n \"description\": \"The emotional tone of this segment\",\n \"enum\": [\n \"positive\",\n \"negative\",\n \"neutral\"\n ]\n },\n {\n \"name\": \"key_objects\",\n \"type\": \"array\",\n \"description\": \"Notable objects visible in this segment\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n ]\n },\n \"min_segment_duration\": 5,\n \"max_segment_duration\": 30\n}") .asString(); ``` ```php Extract timestamped metadata with custom fields request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scene", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "sentiment", "type": "string", "description": "The emotional tone of this segment", "enum": [ "positive", "negative", "neutral" ] }, { "name": "key_objects", "type": "array", "description": "Notable objects visible in this segment", "items": { "type": "string" } } ] } ] }, "min_segment_duration": 5, "max_segment_duration": 30 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Extract timestamped metadata with custom fields using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scene\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"sentiment\",\n \"type\": \"string\",\n \"description\": \"The emotional tone of this segment\",\n \"enum\": [\n \"positive\",\n \"negative\",\n \"neutral\"\n ]\n },\n {\n \"name\": \"key_objects\",\n \"type\": \"array\",\n \"description\": \"Notable objects visible in this segment\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n ]\n },\n \"min_segment_duration\": 5,\n \"max_segment_duration\": 30\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Extract timestamped metadata with custom fields import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.5", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "analysis_mode": "time_based_metadata", "response_format": [ "type": "segment_definitions", "segment_definitions": [ [ "id": "scene", "description": "A distinct scene or setting change in the video", "fields": [ [ "name": "sentiment", "type": "string", "description": "The emotional tone of this segment", "enum": ["positive", "negative", "neutral"] ], [ "name": "key_objects", "type": "array", "description": "Notable objects visible in this segment", "items": ["type": "string"] ] ] ] ] ], "min_segment_duration": 5, "max_segment_duration": 30 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Clip the video before analysis **Request** ```json { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Summarize the key events in this clip.", "max_tokens": 4096, "start_time": 10, "end_time": 60 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Clip the video before analysis import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Summarize the key events in this clip.", "max_tokens": 4096, "start_time": 10, "end_time": 60 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Clip the video before analysis const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.5","video":{"type":"url","url":"https://example.com/video.mp4"},"prompt":"Summarize the key events in this clip.","max_tokens":4096,"start_time":10,"end_time":60}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Clip the video before analysis package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Summarize the key events in this clip.\",\n \"max_tokens\": 4096,\n \"start_time\": 10,\n \"end_time\": 60\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Clip the video before analysis require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Summarize the key events in this clip.\",\n \"max_tokens\": 4096,\n \"start_time\": 10,\n \"end_time\": 60\n}" response = http.request(request) puts response.read_body ``` ```java Clip the video before analysis import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Summarize the key events in this clip.\",\n \"max_tokens\": 4096,\n \"start_time\": 10,\n \"end_time\": 60\n}") .asString(); ``` ```php Clip the video before analysis request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Summarize the key events in this clip.", "max_tokens": 4096, "start_time": 10, "end_time": 60 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Clip the video before analysis using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Summarize the key events in this clip.\",\n \"max_tokens\": 4096,\n \"start_time\": 10,\n \"end_time\": 60\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Clip the video before analysis import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.5", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "prompt": "Summarize the key events in this clip.", "max_tokens": 4096, "start_time": 10, "end_time": 60 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Per-definition time ranges for segmentation **Request** ```json { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "Scene changes.", "time_ranges": [ { "start_time": 0, "end_time": 4 }, { "start_time": 10, "end_time": 14 } ] } ] } } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Per-definition time ranges for segmentation import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "Scene changes.", "time_ranges": [ { "start_time": 0, "end_time": 4 }, { "start_time": 10, "end_time": 14 } ] } ] } } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Per-definition time ranges for segmentation const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.5","video":{"type":"url","url":"https://example.com/video.mp4"},"analysis_mode":"time_based_metadata","response_format":{"type":"segment_definitions","segment_definitions":[{"id":"scenes","description":"Scene changes.","time_ranges":[{"start_time":0,"end_time":4},{"start_time":10,"end_time":14}]}]}}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Per-definition time ranges for segmentation package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Scene changes.\",\n \"time_ranges\": [\n {\n \"start_time\": 0,\n \"end_time\": 4\n },\n {\n \"start_time\": 10,\n \"end_time\": 14\n }\n ]\n }\n ]\n }\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Per-definition time ranges for segmentation require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Scene changes.\",\n \"time_ranges\": [\n {\n \"start_time\": 0,\n \"end_time\": 4\n },\n {\n \"start_time\": 10,\n \"end_time\": 14\n }\n ]\n }\n ]\n }\n}" response = http.request(request) puts response.read_body ``` ```java Per-definition time ranges for segmentation import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Scene changes.\",\n \"time_ranges\": [\n {\n \"start_time\": 0,\n \"end_time\": 4\n },\n {\n \"start_time\": 10,\n \"end_time\": 14\n }\n ]\n }\n ]\n }\n}") .asString(); ``` ```php Per-definition time ranges for segmentation request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "Scene changes.", "time_ranges": [ { "start_time": 0, "end_time": 4 }, { "start_time": 10, "end_time": 14 } ] } ] } }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Per-definition time ranges for segmentation using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Scene changes.\",\n \"time_ranges\": [\n {\n \"start_time\": 0,\n \"end_time\": 4\n },\n {\n \"start_time\": 10,\n \"end_time\": 14\n }\n ]\n }\n ]\n }\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Per-definition time ranges for segmentation import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.5", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "analysis_mode": "time_based_metadata", "response_format": [ "type": "segment_definitions", "segment_definitions": [ [ "id": "scenes", "description": "Scene changes.", "time_ranges": [ [ "start_time": 0, "end_time": 4 ], [ "start_time": 10, "end_time": 14 ] ] ] ] ] ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Segment definitions with `segment_time_format` and a `timestamp` field **Request** ```json { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "Detect scenes and label them.", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." }, { "name": "highlight_at", "type": "timestamp", "description": "The single most representative moment in this scene.", "format": "seconds" } ] } ], "segment_time_format": "hh:mm:ss" } } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Segment definitions with `segment_time_format` and a `timestamp` field import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "Detect scenes and label them.", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." }, { "name": "highlight_at", "type": "timestamp", "description": "The single most representative moment in this scene.", "format": "seconds" } ] } ], "segment_time_format": "hh:mm:ss" } } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Segment definitions with `segment_time_format` and a `timestamp` field const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.5","video":{"type":"url","url":"https://example.com/video.mp4"},"analysis_mode":"time_based_metadata","response_format":{"type":"segment_definitions","segment_definitions":[{"id":"scenes","description":"Detect scenes and label them.","fields":[{"name":"label","type":"string","description":"A short label for this scene."},{"name":"highlight_at","type":"timestamp","description":"The single most representative moment in this scene.","format":"seconds"}]}],"segment_time_format":"hh:mm:ss"}}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Segment definitions with `segment_time_format` and a `timestamp` field package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Detect scenes and label them.\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"highlight_at\",\n \"type\": \"timestamp\",\n \"description\": \"The single most representative moment in this scene.\",\n \"format\": \"seconds\"\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Segment definitions with `segment_time_format` and a `timestamp` field require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Detect scenes and label them.\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"highlight_at\",\n \"type\": \"timestamp\",\n \"description\": \"The single most representative moment in this scene.\",\n \"format\": \"seconds\"\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}" response = http.request(request) puts response.read_body ``` ```java Segment definitions with `segment_time_format` and a `timestamp` field import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Detect scenes and label them.\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"highlight_at\",\n \"type\": \"timestamp\",\n \"description\": \"The single most representative moment in this scene.\",\n \"format\": \"seconds\"\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}") .asString(); ``` ```php Segment definitions with `segment_time_format` and a `timestamp` field request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "Detect scenes and label them.", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." }, { "name": "highlight_at", "type": "timestamp", "description": "The single most representative moment in this scene.", "format": "seconds" } ] } ], "segment_time_format": "hh:mm:ss" } }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Segment definitions with `segment_time_format` and a `timestamp` field using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"Detect scenes and label them.\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"highlight_at\",\n \"type\": \"timestamp\",\n \"description\": \"The single most representative moment in this scene.\",\n \"format\": \"seconds\"\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Segment definitions with `segment_time_format` and a `timestamp` field import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.5", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "analysis_mode": "time_based_metadata", "response_format": [ "type": "segment_definitions", "segment_definitions": [ [ "id": "scenes", "description": "Detect scenes and label them.", "fields": [ [ "name": "label", "type": "string", "description": "A short label for this scene." ], [ "name": "highlight_at", "type": "timestamp", "description": "The single most representative moment in this scene.", "format": "seconds" ] ] ] ], "segment_time_format": "hh:mm:ss" ] ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Segment definitions with a `time_array` field **Request** ```json { "model_name": "pegasus1.6", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "max_tokens": 32768, "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." }, { "name": "dialogue", "type": "time_array", "description": "Each spoken line in this scene.", "items": { "type": "object", "fields": [ { "name": "speaker", "type": "string", "description": "The name of the person speaking." }, { "name": "line", "type": "string", "description": "What the speaker says." }, { "name": "topics", "type": "array", "description": "The topics mentioned in this line.", "items": { "type": "string" } } ] } } ] } ], "segment_time_format": "hh:mm:ss" } } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Segment definitions with a `time_array` field import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.6", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "max_tokens": 32768, "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." }, { "name": "dialogue", "type": "time_array", "description": "Each spoken line in this scene.", "items": { "type": "object", "fields": [ { "name": "speaker", "type": "string", "description": "The name of the person speaking." }, { "name": "line", "type": "string", "description": "What the speaker says." }, { "name": "topics", "type": "array", "description": "The topics mentioned in this line.", "items": { "type": "string" } } ] } } ] } ], "segment_time_format": "hh:mm:ss" } } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Segment definitions with a `time_array` field const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.6","video":{"type":"url","url":"https://example.com/video.mp4"},"analysis_mode":"time_based_metadata","max_tokens":32768,"response_format":{"type":"segment_definitions","segment_definitions":[{"id":"scenes","description":"A distinct scene or setting change in the video","fields":[{"name":"label","type":"string","description":"A short label for this scene."},{"name":"dialogue","type":"time_array","description":"Each spoken line in this scene.","items":{"type":"object","fields":[{"name":"speaker","type":"string","description":"The name of the person speaking."},{"name":"line","type":"string","description":"What the speaker says."},{"name":"topics","type":"array","description":"The topics mentioned in this line.","items":{"type":"string"}}]}}]}],"segment_time_format":"hh:mm:ss"}}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Segment definitions with a `time_array` field package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": 32768,\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"dialogue\",\n \"type\": \"time_array\",\n \"description\": \"Each spoken line in this scene.\",\n \"items\": {\n \"type\": \"object\",\n \"fields\": [\n {\n \"name\": \"speaker\",\n \"type\": \"string\",\n \"description\": \"The name of the person speaking.\"\n },\n {\n \"name\": \"line\",\n \"type\": \"string\",\n \"description\": \"What the speaker says.\"\n },\n {\n \"name\": \"topics\",\n \"type\": \"array\",\n \"description\": \"The topics mentioned in this line.\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Segment definitions with a `time_array` field require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": 32768,\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"dialogue\",\n \"type\": \"time_array\",\n \"description\": \"Each spoken line in this scene.\",\n \"items\": {\n \"type\": \"object\",\n \"fields\": [\n {\n \"name\": \"speaker\",\n \"type\": \"string\",\n \"description\": \"The name of the person speaking.\"\n },\n {\n \"name\": \"line\",\n \"type\": \"string\",\n \"description\": \"What the speaker says.\"\n },\n {\n \"name\": \"topics\",\n \"type\": \"array\",\n \"description\": \"The topics mentioned in this line.\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}" response = http.request(request) puts response.read_body ``` ```java Segment definitions with a `time_array` field import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": 32768,\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"dialogue\",\n \"type\": \"time_array\",\n \"description\": \"Each spoken line in this scene.\",\n \"items\": {\n \"type\": \"object\",\n \"fields\": [\n {\n \"name\": \"speaker\",\n \"type\": \"string\",\n \"description\": \"The name of the person speaking.\"\n },\n {\n \"name\": \"line\",\n \"type\": \"string\",\n \"description\": \"What the speaker says.\"\n },\n {\n \"name\": \"topics\",\n \"type\": \"array\",\n \"description\": \"The topics mentioned in this line.\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}") .asString(); ``` ```php Segment definitions with a `time_array` field request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.6", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "max_tokens": 32768, "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." }, { "name": "dialogue", "type": "time_array", "description": "Each spoken line in this scene.", "items": { "type": "object", "fields": [ { "name": "speaker", "type": "string", "description": "The name of the person speaking." }, { "name": "line", "type": "string", "description": "What the speaker says." }, { "name": "topics", "type": "array", "description": "The topics mentioned in this line.", "items": { "type": "string" } } ] } } ] } ], "segment_time_format": "hh:mm:ss" } }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Segment definitions with a `time_array` field using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": 32768,\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n },\n {\n \"name\": \"dialogue\",\n \"type\": \"time_array\",\n \"description\": \"Each spoken line in this scene.\",\n \"items\": {\n \"type\": \"object\",\n \"fields\": [\n {\n \"name\": \"speaker\",\n \"type\": \"string\",\n \"description\": \"The name of the person speaking.\"\n },\n {\n \"name\": \"line\",\n \"type\": \"string\",\n \"description\": \"What the speaker says.\"\n },\n {\n \"name\": \"topics\",\n \"type\": \"array\",\n \"description\": \"The topics mentioned in this line.\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n ]\n }\n }\n ]\n }\n ],\n \"segment_time_format\": \"hh:mm:ss\"\n }\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Segment definitions with a `time_array` field import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.6", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "analysis_mode": "time_based_metadata", "max_tokens": 32768, "response_format": [ "type": "segment_definitions", "segment_definitions": [ [ "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ [ "name": "label", "type": "string", "description": "A short label for this scene." ], [ "name": "dialogue", "type": "time_array", "description": "Each spoken line in this scene.", "items": [ "type": "object", "fields": [ [ "name": "speaker", "type": "string", "description": "The name of the person speaking." ], [ "name": "line", "type": "string", "description": "What the speaker says." ], [ "name": "topics", "type": "array", "description": "The topics mentioned in this line.", "items": ["type": "string"] ] ] ] ] ] ] ], "segment_time_format": "hh:mm:ss" ] ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Segment the whole video without a token ceiling **Request** ```json { "model_name": "pegasus1.6", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "max_tokens": "unlimited", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." } ] } ] } } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Segment the whole video without a token ceiling import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.6", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "max_tokens": "unlimited", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." } ] } ] } } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Segment the whole video without a token ceiling const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.6","video":{"type":"url","url":"https://example.com/video.mp4"},"analysis_mode":"time_based_metadata","max_tokens":"unlimited","response_format":{"type":"segment_definitions","segment_definitions":[{"id":"scenes","description":"A distinct scene or setting change in the video","fields":[{"name":"label","type":"string","description":"A short label for this scene."}]}]}}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Segment the whole video without a token ceiling package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": \"unlimited\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n }\n ]\n }\n ]\n }\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Segment the whole video without a token ceiling require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": \"unlimited\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n }\n ]\n }\n ]\n }\n}" response = http.request(request) puts response.read_body ``` ```java Segment the whole video without a token ceiling import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": \"unlimited\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n }\n ]\n }\n ]\n }\n}") .asString(); ``` ```php Segment the whole video without a token ceiling request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.6", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "analysis_mode": "time_based_metadata", "max_tokens": "unlimited", "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ { "name": "label", "type": "string", "description": "A short label for this scene." } ] } ] } }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Segment the whole video without a token ceiling using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.6\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"analysis_mode\": \"time_based_metadata\",\n \"max_tokens\": \"unlimited\",\n \"response_format\": {\n \"type\": \"segment_definitions\",\n \"segment_definitions\": [\n {\n \"id\": \"scenes\",\n \"description\": \"A distinct scene or setting change in the video\",\n \"fields\": [\n {\n \"name\": \"label\",\n \"type\": \"string\",\n \"description\": \"A short label for this scene.\"\n }\n ]\n }\n ]\n }\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Segment the whole video without a token ceiling import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.6", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "analysis_mode": "time_based_metadata", "max_tokens": "unlimited", "response_format": [ "type": "segment_definitions", "segment_definitions": [ [ "id": "scenes", "description": "A distinct scene or setting change in the video", "fields": [ [ "name": "label", "type": "string", "description": "A short label for this scene." ] ] ] ] ] ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Structured prompt with reference images **Request** ```json { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt_v2": { "input_text": "Is there a <@tiger-1> in the video?", "media_sources": [ { "name": "tiger-1", "media_type": "image", "url": "https://example.com/tiger.jpg" } ] }, "max_tokens": 4096 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Structured prompt with reference images import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt_v2": { "input_text": "Is there a <@tiger-1> in the video?", "media_sources": [ { "name": "tiger-1", "media_type": "image", "url": "https://example.com/tiger.jpg" } ] }, "max_tokens": 4096 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Structured prompt with reference images const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.5","video":{"type":"url","url":"https://example.com/video.mp4"},"prompt_v2":{"input_text":"Is there a <@tiger-1> in the video?","media_sources":[{"name":"tiger-1","media_type":"image","url":"https://example.com/tiger.jpg"}]},"max_tokens":4096}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Structured prompt with reference images package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt_v2\": {\n \"input_text\": \"Is there a <@tiger-1> in the video?\",\n \"media_sources\": [\n {\n \"name\": \"tiger-1\",\n \"media_type\": \"image\",\n \"url\": \"https://example.com/tiger.jpg\"\n }\n ]\n },\n \"max_tokens\": 4096\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Structured prompt with reference images require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt_v2\": {\n \"input_text\": \"Is there a <@tiger-1> in the video?\",\n \"media_sources\": [\n {\n \"name\": \"tiger-1\",\n \"media_type\": \"image\",\n \"url\": \"https://example.com/tiger.jpg\"\n }\n ]\n },\n \"max_tokens\": 4096\n}" response = http.request(request) puts response.read_body ``` ```java Structured prompt with reference images import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt_v2\": {\n \"input_text\": \"Is there a <@tiger-1> in the video?\",\n \"media_sources\": [\n {\n \"name\": \"tiger-1\",\n \"media_type\": \"image\",\n \"url\": \"https://example.com/tiger.jpg\"\n }\n ]\n },\n \"max_tokens\": 4096\n}") .asString(); ``` ```php Structured prompt with reference images request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt_v2": { "input_text": "Is there a <@tiger-1> in the video?", "media_sources": [ { "name": "tiger-1", "media_type": "image", "url": "https://example.com/tiger.jpg" } ] }, "max_tokens": 4096 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Structured prompt with reference images using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt_v2\": {\n \"input_text\": \"Is there a <@tiger-1> in the video?\",\n \"media_sources\": [\n {\n \"name\": \"tiger-1\",\n \"media_type\": \"image\",\n \"url\": \"https://example.com/tiger.jpg\"\n }\n ]\n },\n \"max_tokens\": 4096\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Structured prompt with reference images import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.5", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "prompt_v2": [ "input_text": "Is there a <@tiger-1> in the video?", "media_sources": [ [ "name": "tiger-1", "media_type": "image", "url": "https://example.com/tiger.jpg" ] ] ], "max_tokens": 4096 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Analyze an image from a URL **Request** ```json { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "url": "https://example.com/product.jpg" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Analyze an image from a URL import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "url": "https://example.com/product.jpg" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Analyze an image from a URL const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.6","image":[{"name":"product_shot","url":"https://example.com/product.jpg"}],"prompt":"Describe the product in <@product_shot> and list every visible feature.","max_tokens":1024}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Analyze an image from a URL package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Analyze an image from a URL require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}" response = http.request(request) puts response.read_body ``` ```java Analyze an image from a URL import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}") .asString(); ``` ```php Analyze an image from a URL request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "url": "https://example.com/product.jpg" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Analyze an image from a URL using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Analyze an image from a URL import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.6", "image": [ [ "name": "product_shot", "url": "https://example.com/product.jpg" ] ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Analyze an image from an uploaded asset **Request** ```json { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "asset_id": "698ae3f3f4a786d35a6303e1" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Analyze an image from an uploaded asset import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "asset_id": "698ae3f3f4a786d35a6303e1" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Analyze an image from an uploaded asset const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.6","image":[{"name":"product_shot","asset_id":"698ae3f3f4a786d35a6303e1"}],"prompt":"Describe the product in <@product_shot> and list every visible feature.","max_tokens":1024}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Analyze an image from an uploaded asset package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"asset_id\": \"698ae3f3f4a786d35a6303e1\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Analyze an image from an uploaded asset require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"asset_id\": \"698ae3f3f4a786d35a6303e1\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}" response = http.request(request) puts response.read_body ``` ```java Analyze an image from an uploaded asset import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"asset_id\": \"698ae3f3f4a786d35a6303e1\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}") .asString(); ``` ```php Analyze an image from an uploaded asset request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "asset_id": "698ae3f3f4a786d35a6303e1" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Analyze an image from an uploaded asset using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"asset_id\": \"698ae3f3f4a786d35a6303e1\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Analyze an image from an uploaded asset import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.6", "image": [ [ "name": "product_shot", "asset_id": "698ae3f3f4a786d35a6303e1" ] ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Analyze a base64-encoded image **Request** ```json { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "base64_string": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Analyze a base64-encoded image import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "base64_string": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Analyze a base64-encoded image const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.6","image":[{"name":"product_shot","base64_string":"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg=="}],"prompt":"Describe the product in <@product_shot> and list every visible feature.","max_tokens":1024}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Analyze a base64-encoded image package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"base64_string\": \"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Analyze a base64-encoded image require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"base64_string\": \"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}" response = http.request(request) puts response.read_body ``` ```java Analyze a base64-encoded image import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"base64_string\": \"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}") .asString(); ``` ```php Analyze a base64-encoded image request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "base64_string": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==" } ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Analyze a base64-encoded image using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"base64_string\": \"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==\"\n }\n ],\n \"prompt\": \"Describe the product in <@product_shot> and list every visible feature.\",\n \"max_tokens\": 1024\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Analyze a base64-encoded image import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.6", "image": [ [ "name": "product_shot", "base64_string": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFAAH/q842iQAAAABJRU5ErkJggg==" ] ], "prompt": "Describe the product in <@product_shot> and list every visible feature.", "max_tokens": 1024 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Analyze two images in one prompt **Request** ```json { "model_name": "pegasus1.6", "image": [ { "name": "before", "url": "https://example.com/before.jpg" }, { "name": "after", "url": "https://example.com/after.jpg" } ], "prompt": "Compare <@before> with <@after> and describe what changed.", "max_tokens": 1024 } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Analyze two images in one prompt import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.6", "image": [ { "name": "before", "url": "https://example.com/before.jpg" }, { "name": "after", "url": "https://example.com/after.jpg" } ], "prompt": "Compare <@before> with <@after> and describe what changed.", "max_tokens": 1024 } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Analyze two images in one prompt const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.6","image":[{"name":"before","url":"https://example.com/before.jpg"},{"name":"after","url":"https://example.com/after.jpg"}],"prompt":"Compare <@before> with <@after> and describe what changed.","max_tokens":1024}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Analyze two images in one prompt package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"before\",\n \"url\": \"https://example.com/before.jpg\"\n },\n {\n \"name\": \"after\",\n \"url\": \"https://example.com/after.jpg\"\n }\n ],\n \"prompt\": \"Compare <@before> with <@after> and describe what changed.\",\n \"max_tokens\": 1024\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Analyze two images in one prompt require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"before\",\n \"url\": \"https://example.com/before.jpg\"\n },\n {\n \"name\": \"after\",\n \"url\": \"https://example.com/after.jpg\"\n }\n ],\n \"prompt\": \"Compare <@before> with <@after> and describe what changed.\",\n \"max_tokens\": 1024\n}" response = http.request(request) puts response.read_body ``` ```java Analyze two images in one prompt import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"before\",\n \"url\": \"https://example.com/before.jpg\"\n },\n {\n \"name\": \"after\",\n \"url\": \"https://example.com/after.jpg\"\n }\n ],\n \"prompt\": \"Compare <@before> with <@after> and describe what changed.\",\n \"max_tokens\": 1024\n}") .asString(); ``` ```php Analyze two images in one prompt request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.6", "image": [ { "name": "before", "url": "https://example.com/before.jpg" }, { "name": "after", "url": "https://example.com/after.jpg" } ], "prompt": "Compare <@before> with <@after> and describe what changed.", "max_tokens": 1024 }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Analyze two images in one prompt using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"before\",\n \"url\": \"https://example.com/before.jpg\"\n },\n {\n \"name\": \"after\",\n \"url\": \"https://example.com/after.jpg\"\n }\n ],\n \"prompt\": \"Compare <@before> with <@after> and describe what changed.\",\n \"max_tokens\": 1024\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Analyze two images in one prompt import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.6", "image": [ [ "name": "before", "url": "https://example.com/before.jpg" ], [ "name": "after", "url": "https://example.com/after.jpg" ] ], "prompt": "Compare <@before> with <@after> and describe what changed.", "max_tokens": 1024 ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Analyze an image with a JSON schema response **Request** ```json { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "url": "https://example.com/product.jpg" } ], "prompt": "Extract the product name and the visible colors from <@product_shot>.", "max_tokens": 1024, "response_format": { "type": "json_schema", "json_schema": { "type": "object", "properties": { "product_name": { "type": "string" }, "colors": { "type": "array", "items": { "type": "string" } } }, "required": [ "product_name", "colors" ] } } } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python Analyze an image with a JSON schema response import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "url": "https://example.com/product.jpg" } ], "prompt": "Extract the product name and the visible colors from <@product_shot>.", "max_tokens": 1024, "response_format": { "type": "json_schema", "json_schema": { "type": "object", "properties": { "product_name": { "type": "string" }, "colors": { "type": "array", "items": { "type": "string" } } }, "required": ["product_name", "colors"] } } } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript Analyze an image with a JSON schema response const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.6","image":[{"name":"product_shot","url":"https://example.com/product.jpg"}],"prompt":"Extract the product name and the visible colors from <@product_shot>.","max_tokens":1024,"response_format":{"type":"json_schema","json_schema":{"type":"object","properties":{"product_name":{"type":"string"},"colors":{"type":"array","items":{"type":"string"}}},"required":["product_name","colors"]}}}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go Analyze an image with a JSON schema response package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Extract the product name and the visible colors from <@product_shot>.\",\n \"max_tokens\": 1024,\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"product_name\": {\n \"type\": \"string\"\n },\n \"colors\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n },\n \"required\": [\n \"product_name\",\n \"colors\"\n ]\n }\n }\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby Analyze an image with a JSON schema response require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Extract the product name and the visible colors from <@product_shot>.\",\n \"max_tokens\": 1024,\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"product_name\": {\n \"type\": \"string\"\n },\n \"colors\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n },\n \"required\": [\n \"product_name\",\n \"colors\"\n ]\n }\n }\n}" response = http.request(request) puts response.read_body ``` ```java Analyze an image with a JSON schema response import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Extract the product name and the visible colors from <@product_shot>.\",\n \"max_tokens\": 1024,\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"product_name\": {\n \"type\": \"string\"\n },\n \"colors\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n },\n \"required\": [\n \"product_name\",\n \"colors\"\n ]\n }\n }\n}") .asString(); ``` ```php Analyze an image with a JSON schema response request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.6", "image": [ { "name": "product_shot", "url": "https://example.com/product.jpg" } ], "prompt": "Extract the product name and the visible colors from <@product_shot>.", "max_tokens": 1024, "response_format": { "type": "json_schema", "json_schema": { "type": "object", "properties": { "product_name": { "type": "string" }, "colors": { "type": "array", "items": { "type": "string" } } }, "required": [ "product_name", "colors" ] } } }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp Analyze an image with a JSON schema response using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.6\",\n \"image\": [\n {\n \"name\": \"product_shot\",\n \"url\": \"https://example.com/product.jpg\"\n }\n ],\n \"prompt\": \"Extract the product name and the visible colors from <@product_shot>.\",\n \"max_tokens\": 1024,\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"product_name\": {\n \"type\": \"string\"\n },\n \"colors\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n },\n \"required\": [\n \"product_name\",\n \"colors\"\n ]\n }\n }\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift Analyze an image with a JSON schema response import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.6", "image": [ [ "name": "product_shot", "url": "https://example.com/product.jpg" ] ], "prompt": "Extract the product name and the visible colors from <@product_shot>.", "max_tokens": 1024, "response_format": [ "type": "json_schema", "json_schema": [ "type": "object", "properties": [ "product_name": ["type": "string"], "colors": [ "type": "array", "items": ["type": "string"] ] ], "required": ["product_name", "colors"] ] ] ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### JSON schema with `timestamp` fields **Request** ```json { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Identify intro, main content, and outro clips.", "response_format": { "type": "json_schema", "json_schema": { "type": "object", "properties": { "clip_start": { "type": "timestamp", "format": "seconds" }, "events": { "type": "array", "items": { "type": "object", "properties": { "intro_end": { "type": "timestamp", "format": "hh:mm:ss" }, "outro_start": { "type": "timestamp", "format": "hh:mm:ss.fff" }, "label": { "type": "string" } } } } } } } } ``` **Response** ```json { "task_id": "64f8d2c7e4a1b37f8a9c5d12", "status": "pending" } ``` **SDK Code** ```python JSON schema with `timestamp` fields import requests url = "https://api.twelvelabs.io/v1.3/analyze/tasks" payload = { "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Identify intro, main content, and outro clips.", "response_format": { "type": "json_schema", "json_schema": { "type": "object", "properties": { "clip_start": { "type": "timestamp", "format": "seconds" }, "events": { "type": "array", "items": { "type": "object", "properties": { "intro_end": { "type": "timestamp", "format": "hh:mm:ss" }, "outro_start": { "type": "timestamp", "format": "hh:mm:ss.fff" }, "label": { "type": "string" } } } } } } } } headers = { "x-api-key": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` ```javascript JSON schema with `timestamp` fields const url = 'https://api.twelvelabs.io/v1.3/analyze/tasks'; const options = { method: 'POST', headers: {'x-api-key': '', 'Content-Type': 'application/json'}, body: '{"model_name":"pegasus1.5","video":{"type":"url","url":"https://example.com/video.mp4"},"prompt":"Identify intro, main content, and outro clips.","response_format":{"type":"json_schema","json_schema":{"type":"object","properties":{"clip_start":{"type":"timestamp","format":"seconds"},"events":{"type":"array","items":{"type":"object","properties":{"intro_end":{"type":"timestamp","format":"hh:mm:ss"},"outro_start":{"type":"timestamp","format":"hh:mm:ss.fff"},"label":{"type":"string"}}}}}}}}' }; try { const response = await fetch(url, options); const data = await response.json(); console.log(data); } catch (error) { console.error(error); } ``` ```go JSON schema with `timestamp` fields package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.twelvelabs.io/v1.3/analyze/tasks" payload := strings.NewReader("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Identify intro, main content, and outro clips.\",\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"clip_start\": {\n \"type\": \"timestamp\",\n \"format\": \"seconds\"\n },\n \"events\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"intro_end\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss\"\n },\n \"outro_start\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss.fff\"\n },\n \"label\": {\n \"type\": \"string\"\n }\n }\n }\n }\n }\n }\n }\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("x-api-key", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` ```ruby JSON schema with `timestamp` fields require 'uri' require 'net/http' url = URI("https://api.twelvelabs.io/v1.3/analyze/tasks") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["x-api-key"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Identify intro, main content, and outro clips.\",\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"clip_start\": {\n \"type\": \"timestamp\",\n \"format\": \"seconds\"\n },\n \"events\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"intro_end\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss\"\n },\n \"outro_start\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss.fff\"\n },\n \"label\": {\n \"type\": \"string\"\n }\n }\n }\n }\n }\n }\n }\n}" response = http.request(request) puts response.read_body ``` ```java JSON schema with `timestamp` fields import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.twelvelabs.io/v1.3/analyze/tasks") .header("x-api-key", "") .header("Content-Type", "application/json") .body("{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Identify intro, main content, and outro clips.\",\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"clip_start\": {\n \"type\": \"timestamp\",\n \"format\": \"seconds\"\n },\n \"events\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"intro_end\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss\"\n },\n \"outro_start\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss.fff\"\n },\n \"label\": {\n \"type\": \"string\"\n }\n }\n }\n }\n }\n }\n }\n}") .asString(); ``` ```php JSON schema with `timestamp` fields request('POST', 'https://api.twelvelabs.io/v1.3/analyze/tasks', [ 'body' => '{ "model_name": "pegasus1.5", "video": { "type": "url", "url": "https://example.com/video.mp4" }, "prompt": "Identify intro, main content, and outro clips.", "response_format": { "type": "json_schema", "json_schema": { "type": "object", "properties": { "clip_start": { "type": "timestamp", "format": "seconds" }, "events": { "type": "array", "items": { "type": "object", "properties": { "intro_end": { "type": "timestamp", "format": "hh:mm:ss" }, "outro_start": { "type": "timestamp", "format": "hh:mm:ss.fff" }, "label": { "type": "string" } } } } } } } }', 'headers' => [ 'Content-Type' => 'application/json', 'x-api-key' => '', ], ]); echo $response->getBody(); ``` ```csharp JSON schema with `timestamp` fields using RestSharp; var client = new RestClient("https://api.twelvelabs.io/v1.3/analyze/tasks"); var request = new RestRequest(Method.POST); request.AddHeader("x-api-key", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"model_name\": \"pegasus1.5\",\n \"video\": {\n \"type\": \"url\",\n \"url\": \"https://example.com/video.mp4\"\n },\n \"prompt\": \"Identify intro, main content, and outro clips.\",\n \"response_format\": {\n \"type\": \"json_schema\",\n \"json_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"clip_start\": {\n \"type\": \"timestamp\",\n \"format\": \"seconds\"\n },\n \"events\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"intro_end\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss\"\n },\n \"outro_start\": {\n \"type\": \"timestamp\",\n \"format\": \"hh:mm:ss.fff\"\n },\n \"label\": {\n \"type\": \"string\"\n }\n }\n }\n }\n }\n }\n }\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` ```swift JSON schema with `timestamp` fields import Foundation let headers = [ "x-api-key": "", "Content-Type": "application/json" ] let parameters = [ "model_name": "pegasus1.5", "video": [ "type": "url", "url": "https://example.com/video.mp4" ], "prompt": "Identify intro, main content, and outro clips.", "response_format": [ "type": "json_schema", "json_schema": [ "type": "object", "properties": [ "clip_start": [ "type": "timestamp", "format": "seconds" ], "events": [ "type": "array", "items": [ "type": "object", "properties": [ "intro_end": [ "type": "timestamp", "format": "hh:mm:ss" ], "outro_start": [ "type": "timestamp", "format": "hh:mm:ss.fff" ], "label": ["type": "string"] ] ] ] ] ] ] ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.twelvelabs.io/v1.3/analyze/tasks")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ```