This recipe shows you how to extract every distinct entity from your videos and images. The response includes each entity’s name, type, frequency, and the items it appears in, ready for a content index, tagging system, or downstream pipeline.

Use cases:

• Content indexing: Build a searchable catalog of who and what appears across your videos and images
• Classification pipelines: Feed entity lists to downstream tagging or categorization systems
• Cross-video analysis: Find which entities appear most frequently or span multiple videos

Key concepts

• Knowledge store: A persistent store of your videos and images plus the understanding the platform derives from them — spatiotemporal context, a typed ontology, and embeddings — that together enable corpus-level reasoning.
• Structured output: A JSON schema you provide with your request. Jockey constrains its output to match your schema, so you retrieve machine-readable results instead of plain text. For a guide on designing schemas and parsing responses, see the Structured output page.

Workflow

This recipe combines two elements: a JSON schema that defines the entity data structure, and a prompt that describes what to extract. Jockey reasons across your knowledge store, identifies every distinct entity, and returns structured results that match your schema. You can adapt this recipe to extract different entity types or apply different extraction criteria.

Prerequisites

• You’ve already created a knowledge store with at least one item in ready status. See the Quickstart page for details.
• You’re familiar with the request and response format. See the Create a response page for details.

Complete example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values.

1
import json
2
from twelvelabs import TwelveLabs, TextParam
3
from twelvelabs.types.text_param_format import TextParamFormat_JsonSchema
4
5
client = TwelveLabs(api_key="<YOUR_API_KEY>")
6
STORE_ID = "<YOUR_KNOWLEDGE_STORE_ID>"
7
8
# Define a JSON schema for the entity list
9
entity_schema = {
10
    "type": "object",
11
    "properties": {
12
        "entities": {
13
            "type": "array",
14
            "items": {
15
                "type": "object",
16
                "properties": {
17
                    "name": {"type": "string"},            # Entity name (person, place, brand, etc.)
18
                    "type": {"type": "string"},            # Category: person, place, object, brand, concept
19
                    "frequency": {"type": "string"},       # How often the entity appears
20
                    "appears_in": {                        # Items containing this entity
21
                        "type": "array",
22
                        "items": {"type": "string"},
23
                    },
24
                },
25
            },
26
        },
27
        "entity_count": {"type": "integer"},                # Total number of distinct entities
28
    },
29
}
30
31
# Extract the entities
32
response = client.responses.create(
33
    knowledge_store_id=STORE_ID,
34
    input=[{"type": "message", "role": "user", "content": "List every distinct entity across all videos and images - people, places, objects, brands, and concepts. Include how frequently each appears."}],
35
    text=TextParam(format=TextParamFormat_JsonSchema(name="entity_list", schema_=entity_schema)),
36
)
37
38
# Parse and read the entities
39
for output in response.output:
40
    if output.type == "message":
41
        for content in output.content:
42
            data = json.loads(content.text)
43
            print(f"Found {data['entity_count']} entities:\n")
44
            for entity in data["entities"]:
45
                print(f"  [{entity['type']}] {entity['name']} - {entity['frequency']}")
46
                if entity.get("appears_in"):
47
                    print(f"    Videos: {', '.join(entity['appears_in'])}")

Code explanation

Example response

The text field inside each content part is a JSON string that matches your schema. After parsing, a typical result looks like this:

1
{
2
  "entities": [
3
    {
4
      "name": "Sarah Chen",
5
      "type": "person",
6
      "frequency": "42 spans",
7
      "appears_in": ["069eb4e8-aeb0-7e83-8000-86413fcc296a", "069e1e97-27f8-7a8e-8000-bf32ecd7fc8c"]
8
    },
9
    {
10
      "name": "Dashboard Analytics",
11
      "type": "concept",
12
      "frequency": "18 spans",
13
      "appears_in": ["069eb4e8-aeb0-7e83-8000-86413fcc296a"]
14
    },
15
    {
16
      "name": "San Francisco",
17
      "type": "place",
18
      "frequency": "5 spans",
19
      "appears_in": ["069eb4e8-aeb0-7e83-8000-86413fcc296a", "069e1e97-27f8-7a8e-8000-bf32ecd7fc8c"]
20
    }
21
  ],
22
  "entity_count": 3
23
}

Variations

Change the prompt to adapt this recipe for different extraction needs.

• Filter by entity type: Change the prompt to “List only the people who appear in these videos and images.”
• Cross-video presence: Change the prompt to “Which entities appear in more than one video?”
• Include relationships: Change the prompt to “List entities and describe how they relate to each other.”

Jupyter notebook

See also

• Track entities across videos - follow a specific entity across multiple videos
• Organize a video library - use entity extraction for categorization
• Structured output - more on JSON Schema responses
• Create a response - API reference for the Responses endpoint