Create video embeddings

The TasksClient class provides methods to create embeddings for your videos.

To create video embeddings:

  1. Create a video embedding task that uploads and processes a video.
  2. Monitor the status of your task.
  3. Retrieve the embeddings once the task is completed.

Related quickstart notebook

Methods

Create a video embedding task

Description: This method creates a new video embedding task that uploads a video to the platform.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

The videos you wish to upload must meet the following requirements:

  • Video resolution: Must be at least 360x360 and must not exceed 3840x2160.
  • Aspect ratio: Must be one of 1:1, 4:3, 4:5, 5:4, 16:9, 9:16, or 17:9.
  • Video and audio formats: Your video files must be encoded in the video and audio formats listed on the FFmpeg Formats Documentation page. For videos in other formats, contact us at support@twelvelabs.io.
  • Duration: Must be between 4 seconds and 2 hours (7,200s).
  • File size: Must not exceed 2 GB. If you require different options, contact us at support@twelvelabs.io.

Function signature and example:

1def create(
2 self,
3 *,
4 model_name: str,
5 video_file: typing.Optional[core.File] = OMIT,
6 video_url: typing.Optional[str] = OMIT,
7 video_start_offset_sec: typing.Optional[float] = OMIT,
8 video_end_offset_sec: typing.Optional[float] = OMIT,
9 video_clip_length: typing.Optional[float] = OMIT,
10 video_embedding_scope: typing.Optional[typing.List[TasksCreateRequestVideoEmbeddingScopeItem]] = OMIT,
11 request_options: typing.Optional[RequestOptions] = None,
12) -> TasksCreateResponse

Parameters:

NameTypeRequiredDescription
model_namestrYesThe name of the model you want to use. Available models: Marengo-retrieval-2.7.
video_filetyping.Optional[core.File]NoThe video file to upload.
video_urltyping.Optional[str]NoSpecify this parameter to upload a video from a publicly accessible URL.
video_start_offset_sectyping.Optional[float]NoThe start offset in seconds from the beginning of the video where processing should begin. Specifying 0 means starting from the beginning of the video. Default: 0, Min: 0, Max: Duration of the video minus video_clip_length.
video_end_offset_sectyping.Optional[float]NoThe end offset in seconds from the beginning of the video where processing should stop. You must set both start and end offsets when using this parameter. Min: video_start_offset + video_clip_length, Max: Duration of the video file.
video_clip_lengthtyping.Optional[float]NoThe desired duration in seconds for each clip for which the platform generates an embedding. Ensure that the clip length does not exceed the interval between the start and end offsets. Default: 6, Min: 2, Max: 10.
video_embedding_scopetyping.
Optional
[typing.List
[TasksCreateRequestVideoEmbeddingScopeItem]]
NoDefines the scope of video embedding generation. Valid values are: ["clip"] and ["clip", "video"]. Default: clip.
request_optionstyping.Optional[RequestOptions]NoRequest-specific configuration.

Return value: Return value: Returns a TasksCreateResponse object representing the new video embedding task.

The TasksCreateResponse class contains the following properties:

NameTypeDescription
idOptional[str]The unique identifier of the video embedding task. You can use the identifier to retrieve the status of your task or retrieve the embedding.

API Reference: Create a video embedding task.

Related guide: Create video embeddings.

Retrieve the status of a video embedding task

Description: This method retrieves the status of a video embedding task.

Function signature and example:

1def status(self, task_id: str, *, request_options: typing.Optional[RequestOptions] = None) -> TasksStatusResponse

Parameters:

NameTypeRequiredDescription
task_idstringYesThe unique identifier of the video embedding task.
request_optionstyping.Optional[RequestOptions]NoRequest-specific configuration.

Return value: Returns a TasksStatusResponse object containing the current status of the embedding task.

The TasksStatusResponse class contains the following properties:

NameTypeDescription
idOptional[str]The unique identifier of the video embedding task.
statusOptional[str]The status of the video indexing task. It can take one of the following values: processing, ready or failed.
model_nameOptional[str]The name of the video understanding model the platform used to create the embedding.
video_embeddingOptional[TasksStatusResponseVideoEmbedding]An object containing the metadata associated with the embedding.

The TasksStatusResponseVideoEmbedding class contains the following properties:

NameTypeDescription
metadataOptional[VideoEmbeddingMetadata]Metadata associated with the embedding.

The VideoEmbeddingMetadata class extends BaseEmbeddingMetadata and contains the following properties:

NameTypeDescription
video_clip_lengthOptional[float]The duration for each clip in seconds, as specified by the video_clip_length parameter. Note that the platform automatically truncates video segments shorter than 2 seconds.
video_embedding_scopeOptional[List[str]]The scope you’ve specified in the request. It can take one of the following values: ['clip'] or ['clip', 'video'].
durationOptional[float]The total duration of the video in seconds.
input_urlOptional[str]The URL of the media file used to generate the embedding. Present if a URL was provided in the request.
input_filenameOptional[str]The name of the media file used to generate the embedding. Present if a file was provided in the request.

API Reference: Retrieve the status of a video embedding task.

Related guide: Create video embeddings.

Wait for a video embedding task to complete

Description: This method waits until a video embedding task is completed by periodically checking its status. If you provide a callback function, it calls the function after each status update with the current task object, allowing you to monitor progress.

1def wait_for_done(
2 self,
3 task_id: str,
4 *,
5 sleep_interval: float = 5.0,
6 callback: typing.Optional[typing.Callable[[TasksStatusResponse], None]] = None,
7 request_options: typing.Optional[RequestOptions] = None,
8) -> TasksStatusResponse

Parameters

NameTypeRequiredDescription
task_idstrYesThe unique identifier of the task to wait for.
sleep_intervalfloatNoSets the time in seconds to wait between status checks. Must be greater than 0. Default: 5.0.
callbackOptional[Callable[[TasksStatusResponse], None]]NoProvides an optional function to call after each status check. The function receives the current task response. Use this to monitor progress.
request_optionsOptional[RequestOptions]NoRequest-specific configuration.

Return value: Returns a TasksStatusResponse object containing the status of your completed task. See the Retrieve the status of a video embedding task section above for complete property details.

Retrieve video embeddings

Description: This method retrieves embeddings for a specific video embedding task. Ensure the task status is ready before retrieving your embeddings.

Function signature and example:

1def retrieve(
2 self,
3 task_id: str,
4 *,
5 embedding_option: typing.Optional[
6 typing.Union[
7 TasksRetrieveRequestEmbeddingOptionItem, typing.Sequence[TasksRetrieveRequestEmbeddingOptionItem]
8 ]
9 ] = None,
10 request_options: typing.Optional[RequestOptions] = None,
11) -> TasksRetrieveResponse

Parameters:

NameTypeRequiredDescription
task_idstrYesThe unique identifier of your video embedding task.
embedding_optiontyping.
Optional
[typing.Union
[TasksRetrieveRequestEmbeddingOptionItem, typing.
Sequence
[TasksRetrieveRequestEmbeddingOptionItem]]]
NoSpecifies which types of embeddings to retrieve. You can include one or more of the following values: visual-text (returns visual embeddings optimized for text search) or audio (returns audio embeddings). The platform returns all available embeddings if you don’t provide this parameter.
request_optionstyping.Optional[RequestOptions]NoRequest-specific configuration.

Return value: Returns a TasksRetrieveResponse object containing your embeddings.

The TasksRetrieveResponse class contains the following properties:

NameTypeDescription
idOptional[str]The unique identifier of the video embedding task.
model_nameOptional[str]The name of the video understanding model the platform used to create the embedding.
statusOptional[str]The status of the video indexing task. It can take one of the following values: processing, ready or failed.
created_atOptional[datetime]The date and time, in the RFC 3339 format, that the video embedding task was created.
video_embeddingOptional[TasksRetrieveResponseVideoEmbedding]An object containing the embeddings and metadata.

The TasksRetrieveResponseVideoEmbedding class contains the following properties:

NameTypeDescription
metadataOptional[VideoEmbeddingMetadata]Metadata associated with the embedding.
segmentsOptional[List[VideoSegment]]An array of objects containing the embeddings for each video segment and the associated information.

The VideoEmbeddingMetadata class extends BaseEmbeddingMetadata and contains the following properties:

NameTypeDescription
video_clip_lengthOptional[float]The duration for each clip in seconds. Note that the platform automatically truncates video segments shorter than 2 seconds.
video_embedding_scopeOptional[List[str]]The scope you’ve specified in the request. It can take one of the following values: ['clip'] or ['clip', 'video'].
durationOptional[float]The total duration of the video in seconds.
input_urlOptional[str]The URL of the media file used to generate the embedding. Present if a URL was provided in the request.
input_filenameOptional[str]The name of the media file used to generate the embedding. Present if a file was provided in the request.

The VideoSegment class extends AudioSegment and contains the following properties:

NameTypeDescription
start_offset_secOptional[float]The start time, in seconds, from which the platform generated the embedding.
end_offset_secOptional[float]The end time, in seconds, of the video segment for this embedding.
embedding_optionOptional[str]The type of the embedding.
embedding_scopeOptional[str]The scope of the video embedding.
float_Optional[List[float]]An array of floating point numbers representing the embedding. You can use this array with cosine similarity for various downstream tasks.

API Reference: Retrieve video embeddings.

Related guide: Create video embeddings.

Error codes

This section lists the most common error messages you may encounter while creating video embeddings.

  • parameter_invalid
    • The video_clip_length parameter is invalid. video_clip_length should be within 2-10 seconds long
    • The video_end_offset_sec parameter is invalid. video_end_offset_sec should be greater than video_start_offset_sec