Use the Marengo video understanding model to generate embeddings from video, audio, text, and image inputs. These embeddings enable similarity search, content clustering, recommendation systems, and other machine learning applications.

Model specification

The model has two types of limits: the maximum input size you can submit and the portion of content that it embeds.

Input requirements

This table shows the maximum size for each type of input:

Embedding coverage per input type

This table shows what portion of your input the model processes into embeddings:

Pricing

For details on pricing, see the Amazon Bedrock pricing page.

Choose the processing method

Select the processing method based on your use case and performance requirements. Synchronous processing returns embeddings immediately in the API response, while asynchronous processing handles larger files and batch operations by saving results to S3.

Use synchronous processing to:

• Build real-time applications like chatbots, search, and recommendation systems.
• Enable interactive features that require immediate results.

Use asynchronous processing to:

• Build applications that process video, audio, and large-scale image files.
• Run batch operations and background workflows.

Prerequisites

Before you start, ensure you have the following:

• An AWS account with access to a region where the TwelveLabs models are supported.
• An AWS IAM principal with sufficient Amazon Bedrock permissions. For details on setting permissions, see the Identity and access management for Amazon Bedrock page.
• S3 permissions to read input files and write output files for Marengo operations.
• The AWS CLI and configured with your credentials.
• Python 3.7 or later with the boto3 library.
• Access to the model you want to use. Navigate to the AWS Console > Bedrock > Model Access page and request access. Note that the availability of the models varies by region.

Create embeddings

Marengo supports base64 encoded strings and S3 URIs for media input. Note that the base64 method has a 36MB file size limit. This guide uses S3 URIs.

To generate embeddings from your content, you use one of two Amazon Bedrock APIs, depending on your processing needs.

Synchronous processing

The InvokeModel API processes your request synchronously and returns embeddings directly in the response.

The InvokeModel API requires two parameters:

• modelId: The inference profile ID for the model.
• body: A JSON-encoded string containing your input parameters.

The request body contains the following fields:

• inputType: The type of content. Values: “text” or “image”.
• inputText: The text to embed. This parameter is required for text inputs.
• mediaSource: The image source, which contains either:
  • base64String: Your base64-encoded image for inline processing
  • s3Location: The S3 location for images stored in S3.
  This parameter is required for image inputs.

Example

1
import boto3
2
import json
3
4
# Replace the `us` prefix depending on your region
5
INFERENCE_PROFILE_ID = "us.twelvelabs.marengo-embed-2-7-v1:0"
6
INPUT_TEXT="<YOUR_TEXT>"
7
                            
8
model_input = {
9
    "inputType": "text",
10
    "inputText": INPUT_TEXT
11
}
12
13
# Initialize the Bedrock Runtime client
14
client = boto3.client('bedrock-runtime')
15
16
# Make the request
17
response = client.invoke_model(
18
    modelId=INFERENCE_PROFILE_ID,
19
    body=json.dumps(model_input)
20
)
21
22
# Print the response body
23
response_body = json.loads(response['body'].read().decode('utf-8'))
24
print(response_body)

Asynchronous processing

The StartAsyncInvoke API processes your request asynchronously, storing the results in your S3 bucket.

To create embeddings asynchronously, you must complete the following steps:

The StartAsyncInvoke API requires three parameters:

• modelId: The model ID.
• modelInput: A dictionary containing your input parameters.
• outputDataConfig: A dictionary specifying where to save the results

The modelInput dictionary contains the following fields:

• inputType: The type of content you’re embedding (“video”, “audio”, “image”, or “text”)
• mediaSource: The S3 location of your input file (for video, audio, and image)
• inputText: The text content (for text inputs only)

S3 output structure

Each invocation creates a unique directory in your S3 bucket with two files:

• manifest.json: Contains metadata including the request ID.
• output.json: Contains the actual embeddings.

Example

1
import boto3
2
import time
3
4
REGION = "<YOUR_REGION>"
5
MODEL_ID = "twelvelabs.marengo-embed-2-7-v1:0"
6
ACCOUNT_ID = "<YOUR_ACCOUNT_ID>"
7
BUCKET = "<YOUR_BUCKET_NAME>"
8
FILE_NAME = "<YOUR_FILE>"
9
INPUT_TYPE = "<YOUR_INPUT_TYPE>"
10
11
bedrock_client = boto3.client(service_name="bedrock-runtime", region_name=REGION)
12
13
# Start async video embedding
14
model_input = {
15
    "mediaSource": {
16
        "s3Location": {
17
            "uri": f"s3://{BUCKET}/{FILE_NAME}",
18
            "bucketOwner": ACCOUNT_ID
19
        }
20
    },
21
    "inputType": INPUT_TYPE
22
}
23
24
async_request_response = bedrock_client.start_async_invoke(
25
    modelId=MODEL_ID,
26
    modelInput=model_input,
27
    outputDataConfig={
28
        "s3OutputDataConfig": {
29
            "s3Uri": f"s3://{BUCKET}",
30
            "bucketOwner": ACCOUNT_ID
31
        }
32
    }
33
)
34
35
print("async_request_response: ", async_request_response)
36
37
# get the invocation arn
38
invocation_arn = async_request_response.get("invocationArn")
39
40
# wait for the async job to complete
41
max_retries = 60
42
retries = 0
43
while True:
44
    response = bedrock_client.get_async_invoke(
45
        invocationArn=invocation_arn
46
    )
47
    print(f"status: {response.get('status')}")
48
    if response.get("status") == "Completed":
49
        break
50
    time.sleep(1)
51
    retries += 1
52
    if retries > max_retries:
53
        break
54
55
print(response)
56
57
# Extract the S3 URI where results are stored
58
output_s3_uri = response.get("outputDataConfig", {}).get("s3OutputDataConfig", {}).get("s3Uri")
59
print(f"Results stored at: {output_s3_uri}")

Use embeddings

After generating embeddings, you can store them in a vector database for efficient similarity search and retrieval.

The typical workflow is as follows:

Request parameters and response fields

For a complete list of request parameters and response fields, see the TwelveLabs Marengo Embed 2.7 page in the Amazon Bedrock documentation.