Create video embeddings

The Resources.EmbedTask class provides methods to create embeddings for your videos.

To create video embeddings:

  1. Create a video embedding task that uploads and processes a video.
  2. Monitor the status of your task.
  3. Retrieve the embeddings once the task is completed.

Methods

Create a video embedding task

Description: This method creates a new video embedding task that uploads a video to the platform.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

Function signature and example:

1create(
2 request: TwelvelabsApi.embed.TasksCreateRequest,
3 requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.embed.TasksCreateResponse>

Parameters:

NameTypeRequiredDescription
modelNamestringYesThe name of the model you want to use. Available models: Marengo-retrieval-2.7.
videoFileFile | fs.ReadStream | BlobNoThe video file to upload.
videoUrlstringNoSpecify this parameter to upload a video from a publicly accessible URL.
videoStartOffsetSecnumberNoThe start offset in seconds from the beginning of the video where processing should begin. Specifying 0 means starting from the beginning of the video. Default: 0, Min: 0, Max: Duration of the video minus videoClipLength.
videoEndOffsetSecnumberNoThe end offset in seconds from the beginning of the video where processing should stop. You must set both start and end offsets when using this parameter. Min: videoStartOffset + videoClipLength, Max: Duration of the video file.
videoClipLengthnumberNoThe desired duration in seconds for each clip for which the platform generates an embedding. Ensure that the clip length does not exceed the interval between the start and end offsets. Default: 6, Min: 2, Max: 10.
videoEmbeddingScopeTwelvelabsApi.
embed.
TasksCreateRequestVideoEmbeddingScopeItem[]
NoDefines the scope of video embedding generation. Valid values are: ["clip"] and ["clip", "video"]. Default: clip.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns a Promise that resolves to a TasksCreateResponse object representing the new video embedding task.

The TasksCreateResponse interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the video embedding task. You can use the identifier to retrieve the status of your task or retrieve the embedding.

API Reference: Create a video embedding task.

Related guide: Create video embeddings.

Retrieve the status of a video embedding task

Description: This method retrieves the status of a video embedding task.

Function signature and example:

1status(
2 taskId: string,
3 requestOptions?: Tasks.RequestOptions
4): core.HttpResponsePromise<TwelvelabsApi.embed.TasksStatusResponse>
NameTypeRequiredDescription
taskIdstringYesThe unique identifier of the video embedding task.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns a Promise that resolves to a TasksStatusResponse object containing the current status of the embedding task.

The TasksStatusResponse interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the video embedding task.
statusstringThe status of the video indexing task. It can take one of the following values: processing, ready or failed.
modelNamestringThe name of the video understanding model the platform used to create the embedding.
videoEmbeddingTasksStatusResponseVideoEmbeddingAn object containing the metadata associated with the embedding.

The TasksStatusResponseVideoEmbedding interface contains the following properties:

NameTypeDescription
metadataVideoEmbeddingMetadataMetadata associated with the embedding.

The VideoEmbeddingMetadata interface extends BaseEmbeddingMetadata and contains the following properties:

NameTypeDescription
videoClipLengthnumberThe duration for each clip in seconds. Note that the platform automatically truncates video segments shorter than 2 seconds.
videoEmbeddingScopestring[]The scope you’ve specified in the request. It can take one of the following values: ['clip'] or ['clip', 'video'].
durationnumberThe total duration of the video in seconds.
inputUrlstringThe URL of the media file used to generate the embedding. Present if a URL was provided in the request.
inputFilenamestringThe name of the media file used to generate the embedding. Present if a file was provided in the request.

API Reference: Retrieve the status of a video embedding task.

Related guide: Create video embeddings.

Wait for a video embedding task to complete

Description: This method waits until a video embedding task is completed by periodically checking its status. If you provide a callback function, it calls the function after each status update with the current task object, allowing you to monitor progress.

Function signature and example:

1waitForDone(
2 taskId: string,
3 options?: {
4 sleepInterval?: number;
5 callback?: (task: TwelvelabsApi.embed.TasksStatusResponse) => void | Promise<void>;
6 },
7 requestOptions?: Tasks.RequestOptions
8): Promise<TwelvelabsApi.embed.TasksStatusResponse>

Parameters

NameTypeRequiredDescription
taskIdstringYesThe unique identifier of the task to wait for.
sleepIntervalnumberNoSets the time in seconds to wait between status checks. Must be greater than 0. Default: 5.0.
callback(task: TwelvelabsApi.embed.TasksStatusResponse) => void | Promise<void>NoProvides an optional function to call after each status check. The function receives the current task response. Use this to monitor progress.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns a Promise that resolves to a TasksStatusResponse object containing the status of your completed task. See the Retrieve the status of a video embedding task section above for complete property details.

Retrieve video embeddings

Description: This method retrieves embeddings for a specific video embedding task. Ensure the task status is ready before retrieving your embeddings.

Function signature and example:

1retrieve(
2 taskId: string,
3 request?: TwelvelabsApi.embed.TasksRetrieveRequest,
4 requestOptions?: Tasks.RequestOptions
5): core.HttpResponsePromise<TwelvelabsApi.embed.TasksRetrieveResponse>

Parameters:

NameTypeRequiredDescription
taskIdstringYesThe unique identifier of your video embedding task.
embeddingOptionTwelvelabsApi.
embed.
TasksRetrieveRequestEmbeddingOptionItem | TwelvelabsApi.
embed.
TasksRetrieveRequestEmbeddingOptionItem[]
NoSpecifies which types of embeddings to retrieve. You can include one or more of the following values: visual-text (returns visual embeddings optimized for text search) or audio (returns audio embeddings). The platform returns all available embeddings if you don’t provide this parameter.
requestOptionsTasks.RequestOptionsNoRequest-specific configuration.

Return value: Returns a Promise that resolves to a TasksRetrieveResponse object containing your embeddings.

The TasksRetrieveResponse interface contains the following properties:

NameTypeDescription
idstringThe unique identifier of the video embedding task.
modelNamestringThe name of the video understanding model the platform used to create the embedding.
statusstringThe status of the video indexing task. It can take one of the following values: processing, ready or failed.
createdAtDateThe date and time, in the RFC 3339 format, that the video embedding task was created.
videoEmbeddingTasksRetrieveResponseVideoEmbeddingAn object containing the embeddings and metadata.

The TasksRetrieveResponseVideoEmbedding interface contains the following properties:

NameTypeDescription
metadataVideoEmbeddingMetadataMetadata associated with the embedding.
segmentsVideoSegment[]An array of objects containing the embeddings for each video segment and the associated information.

The VideoEmbeddingMetadata interface extends BaseEmbeddingMetadata and contains the following properties:

NameTypeDescription
videoClipLengthnumberThe duration for each clip in seconds. Note that the platform automatically truncates video segments shorter than 2 seconds.
videoEmbeddingScopestring[]The scope you’ve specified in the request. It can take one of the following values: ['clip'] or ['clip', 'video'].
durationnumberThe total duration of the video in seconds.
inputUrlstringThe URL of the media file used to generate the embedding. Present if a URL was provided in the request.
inputFilenamestringThe name of the media file used to generate the embedding. Present if a file was provided in the request.

The VideoSegment interface extends AudioSegment and contains the following properties:

NameTypeDescription
startOffsetSecnumberThe start time, in seconds, from which the platform generated the embedding.
endOffsetSecnumberThe end time, in seconds, of the video segment for this embedding.
embeddingOptionstringThe type of the embedding.
embeddingScopestringThe scope of the video embedding.
floatnumber[]An array of floating point numbers representing the embedding. You can use this array with cosine similarity for various downstream tasks.

API Reference: Retrieve video embeddings.

Related guide: Create video embeddings.

Error codes

This section lists the most common error messages you may encounter while creating video embeddings.

  • parameter_invalid
    • The video_clip_length parameter is invalid. video_clip_length should be within 2-10 seconds long
    • The video_end_offset_sec parameter is invalid. video_end_offset_sec should be greater than video_start_offset_sec