This guide shows you how to search a knowledge store with a natural-language query. The search matches both videos and images: a video result is a clip with a time range, and an image result is the whole image. Each result is ranked by relevance. Use the results in a search interface or a processing pipeline.

Use direct search when a literal query describes what you need and you want the matching clips and images to display or process. For interpretive or subjective queries, or to have Jockey explain and refine the results, see agentic search instead.

Key features:

• Natural-language queries: Searches videos and images with a single natural-language query.
• Video and image results: Returns a clip with a time range for each video match and the whole image for each image match.
• Relevance ranking: Ranks every result by relevance.
• Modality control: Matches videos by visual content, audio, or both. Images always match on their visual content.

Use cases:

• In-app search features: Build content search into your application.
• Content retrieval: Find specific moments and images across a large collection without manual review.
• Targeted lookups: Filter to a type of item or specific items, then page through the results.

Key concepts

• Knowledge store: A persistent store of your videos and images plus the understanding the platform derives from them — spatiotemporal context, a typed ontology, and embeddings — that together enable corpus-level reasoning.
• Modalities: The sources of information the platform searches within a video: visual content and audio (both speech and non-speech sounds). The platform searches images by their visual content. For guidance, see the Modalities page.

Prerequisites

You’ve already added at least one asset to a knowledge store, and the knowledge store item has reached the ready status. See the Add assets to a knowledge store page for details.

Example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values. The example searches both videos and images with a single query, then pages through the results and prints every match.

1
from twelvelabs import TwelveLabs
2
3
client = TwelveLabs(api_key="<YOUR_API_KEY>")
4
STORE_ID = "<YOUR_KNOWLEDGE_STORE_ID>"
5
6
# Step 1: Send the request
7
result = client.knowledge_stores.search(
8
    knowledge_store_id=STORE_ID,
9
    query={"text": "<YOUR_QUERY>"},
10
    search_options={
11
        "video": {
12
            "modalities": ["visual", "audio"],
13
        }
14
    },
15
    # filter={"asset_type": {"eq": "image"}},  # Optional. Return only one type, or restrict to specific items.
16
    # group_by="item",  # Optional. Return one result per item instead of per clip.
17
    # page_size=10,  # Optional. Results per page. Default 10, maximum 50.
18
    # include_metadata=True,  # Optional. Return the system and user metadata of each item.
19
)
20
21
while True:
22
    # Step 2: Process the results
23
    for hit in result.data:
24
        if hit.asset_type == "video":
25
            clip = hit.matches[0]
26
            print(f"Rank {hit.rank}: video {hit.item_id}  {clip.start_sec}s-{clip.end_sec}s")
27
            if clip.transcription:
28
                print(f"  transcription: {clip.transcription}")
29
        elif hit.asset_type == "image":
30
            print(f"Rank {hit.rank}: image {hit.item_id}")
31
32
    # Step 3: Request the next page
33
    if not result.next_page_token:
34
        break
35
    result = client.knowledge_stores.search(
36
        knowledge_store_id=STORE_ID,
37
        query={"text": "<YOUR_QUERY>"},
38
        search_options={"video": {"modalities": ["visual", "audio"]}},
39
        page_token=result.next_page_token,
40
    )

Code explanation

Example response

For a query such as “a rocket lifting off the launch pad”, a typical response with a video result and an image result looks like this:

1
{
2
  "data": [
3
    {
4
      "asset_type": "video",
5
      "rank": 1,
6
      "item_id": "ksi_069e9870-3c4d-7abc-9012-3456789abcde",
7
      "matches": [
8
        {
9
          "start_sec": 22.5,
10
          "end_sec": 29.75,
11
          "modalities": ["visual", "audio"],
12
          "transcription": "We have a liftoff. Liftoff on Apollo 11."
13
        }
14
      ]
15
    },
16
    {
17
      "asset_type": "image",
18
      "rank": 2,
19
      "item_id": "ksi_069e1e97-27f8-7a8e-8000-bf32ecd7fc8c"
20
    }
21
  ]
22
}