The Entity Search feature is currently in beta.

Use the Entity Search API to identify and locate specific people in your video content. You can search for players in sports footage, track characters in movies, or locate persons in security footage.

Key features:

Simple syntax: Search for specific people in your videos using straightforward queries.
Built-in organization: Group related entities into logical groups for efficient management and improved accuracy.
Multiple reference images: Upload multiple images per person to improve identification across different lighting conditions and angles.

Use cases:

Create sports highlights: Find specific players in game footage to generate highlight reels or analyze performance statistics.
Track movie characters: Locate characters across scenes to measure screen time or create promotional content.
Analyze security footage: Identify personss in surveillance videos to investigate incidents or locate missing persons.

Key concepts:

Entity collections: Organize related entities into logical groups. Create separate collections for different projects - for example, one collection for the players in a soccer team.
Entities: Represent the specific people you want to find in videos. Each entity belongs to a collection and requires reference images for identification.
Assets: Are the reference images that help identify each person. Upload multiple images of each person in different outfits, lighting conditions, and angles to improve search accuracy.

Benefits of separate collections

Better search accuracy: When grouping related persons in a collection, the platform analyzes their similarities and differences. When you search for a specific person, the platform uses reference images from other persons in the collection to better differentiate them. This produces more relevant results than extensive mixed collections.

Improved organization: Find and manage entities faster within focused collections.

When to create separate collections

✅ Create separate collections for:

Sports teams
Movies or TV shows
Security locations or time periods
Departments or organizational units

❌ Avoid mixing unrelated entities:

Players from different teams
Movie characters with sports players
People from different projects or contexts

Examples of good collection strategies

🏀 Sports:
- “Team_A_Roster” (Team A players)
- “Team_B_Roster” (Team B players)
- “Referees” (game officials)
🎬 Entertainment:
- “Movie_Main_Characters” (protagonists and key characters)
- “Movie_Supporting_Cast” (secondary characters)
- “Movie_Extras” (background actors, if needed)
👮 Security:
- “Building_A_Employees” (authorized personnel)
- “Building_A_Visitors” (temporary access)
- “Security_Personnel” (guards and staff)

Note

The platform automatically creates a sample entity collection when you create your account. Users on the Free plan can have a total of one entity collection with up to fifteen entities. The default collection is included in this limit. To create more collections, upgrade to the Developer plan. For instructions, see the Upgrade your plan section.

Prerequisites

You already know how to perform search requests. For instructions, see the Search page.
The Marengo 3.0 video understanding engine must be enabled in your index.
Before uploading assets, ensure your image files meet the format requirements.

Complete example

This complete example shows how to search for entities in your video content. You need an index with uploaded videos to complete these steps. Ensure you replace the placeholders surrounded by <> with your values.

1 from twelvelabs import TwelveLabs
2 
3 # Initialize the TwelveLabs client with your API key, create an index, and upload videos.
4 
5 #1. Create an entity collection
6 entity_collection = client.entity_collections.create(
7     name="<YOUR_ENTITY_COLLECTION_NAME>",
8 )
9 print(f"Created entity collection: id={entity_collection.id}")
10 
11 
12 #2. Upload reference images
13 sample_image_urls = [
14     "<YOUR_IMAGE_URL_1>",
15     "<YOUR_IMAGE_URL_2>",
16     "<YOUR_IMAGE_URL_3>",
17     "<YOUR_IMAGE_URL_4>",
18     "<YOUR_IMAGE_URL_5>"
19 ]
20 asset_ids= []
21 for i, url in enumerate(sample_image_urls, 1):
22     asset= client.assets.create(
23         method="url",
24         url=url,
25     )
26     asset_ids.append(asset.id)
27     print(f"Created asset {i}/{len(sample_image_urls)}: id={asset.id}")
28 
29 
30 #3. Create an entity
31 entity = client.entity_collections.entities.create(
32     entity_collection_id=entity_collection.id,
33     name="<YOUR_ENTITY_NAME>",
34     asset_ids=asset_ids,
35 )
36 print(f"Created entity: id={entity.id}")
37 
38 
39 #4. Perform an entity search
40 search_pager = client.search.query(
41     index_id="<YOUR_INDEX_ID>",
42     search_options=["visual", "audio"],
43     query_text=f"<@{entity.id}> is walking",
44 )
45 
46 5. Process the search results
47 for clip in search_pager:
48     print(
49         f"  video_id={clip.video_id} start={clip.start} end={clip.end} rank={clip.rank}"
50     )

Step-by-step guide

Python

Node.js

Create an entity collection

Create a collection to organize your entities into a logical group.
Function call: You call the entity_collections.create method.
Parameters:

name: The name of your new entity collection.

Return value: An object representing the newly created entity collection. It contains, among other information, a field named id representing the unique identifier of the newly created entity collection.

Upload reference images

Upload image files that show the people you want to identify. Choose clear, high-quality images that show each person’s face clearly.
Function call: You call the assets.create method.
Parameters:

method: Specifies the upload method for the asset. Use direct to upload a local image file or url for a publicly accessible URL. This example uses url.
url or file: Specifies the publicly accessible URL of your image or a file from your local file system. This example uses url.

Return value: An object representing the newly created asset. It contains, among other information, a field named id representing the unique identifier of the newly created asset.

Create an entity

Define each person you want to search for by creating an entity within your collection. Link your uploaded images to the entity to help the platform identify that person. More reference images improve accuracy because they show the person in different conditions and angles.
Function call: You call the entity_collections.entities.create method.
Parameters:

entity_collection_id: The unique identifier of the entity collection in which to create the entity.
name: The name of the entity. Make sure you use a succinct and descriptive name.
asset_ids: An array of asset IDs to associate with the entity. You must provide at least one value.

Return value: An object representing the newly created entity. It contains, among other information, a field named id representing the unique identifier of the newly created entity.

Perform an entity search

Search for your entities in your video content.

Function call: You call the search.query method.
Parameters:

index_id: The unique identifier of the index to search.
search_options: An array specifying the types of search to perform. Use visual to search based on visual content and audio to search based on spoken words. This example uses both options.
query_text: The search query that includes the identifier of your entity. You must enclose it between the <@ and > markers. For example, to search for an entity with the ID entity123, use <@entity123> is walking as your query.

Return value: An object of type SyncPager[SearchItem] that can be iterated to access search results. Each item contains the following fields, among other information:

video_id: The unique identifier of the video that matched your search terms.
start: The start time of the matching video clip, expressed in seconds.
end: The end time of the matching video clip, expressed in seconds.
rank: The relevance ranking assigned by the model. Lower numbers indicate higher relevance, starting with 1 for the most relevant result.
Only Marengo 3.0 and newer versions return this field. Earlier versions return score and confidence instead.

Process the search results

This example iterates over the results using a for loop to display the search results to the standard output.
.