Build a content agent

This recipe shows you how to build a multi-step agent that analyzes a video collection and produces structured creative output. The example constructs a micro-drama outline with scene breakdowns, character arcs, and clip references.

Use cases:

  • Micro-drama outlines: Extract narrative elements and construct scene breakdowns from video footage
  • Training curricula: Identify learning objectives and demonstrations for instructional design
  • Compliance audits: Find claims, disclosures, and regulation references for review
  • Content plans: Discover topics, audience signals, and gaps for editorial strategy

Key concepts

  • Knowledge store: A persistent store of your videos and images plus the understanding the platform derives from them — spatiotemporal context, a typed ontology, and embeddings — that together enable corpus-level reasoning.
  • Ingestion configuration: A configuration that controls how the platform processes content added to a knowledge store. You set the ingestion_config parameter at creation time and cannot change it afterward.
  • Instructions: Additional guidance that shapes Jockey’s behavior for a specific domain or task.
  • Structured output: A JSON schema you provide with your request. Jockey constrains its output to match your schema, so you retrieve machine-readable results instead of plain text. For a guide on designing schemas and parsing responses, see the Structured output page.
  • Multi-turn session: A conversation where each request builds on previous turns. The first request creates a session, and follow-up requests reference that session to continue the conversation. For details, see the Multi-turn sessions page.

Workflow

This recipe combines three capabilities into a reusable agent pattern. First, you create a knowledge store with a domain-specific ingestion configuration. This configuration controls what the platform extracts from your videos during processing. Then, you call the Responses API with a JSON schema, instructions that give Jockey a specific role and domain guidance, and a prompt. Jockey reasons across your knowledge store and returns structured output that matches your schema. Finally, you use the session identifier to refine the results in a follow-up turn. You can adapt this pattern to any creative or analytical domain.

Prerequisites

  • You’ve already created a knowledge store with at least one item in ready status. See the Quickstart page for details.
  • You’re familiar with the request and response format. See the Create a response page for details.

Complete example

Copy and paste the code below, replacing the placeholders surrounded by <> with your values.

1import json
2from twelvelabs import TwelveLabs, IngestionConfig, EnrichmentConfig_Description, TextParam
3from twelvelabs.types.text_param_format import TextParamFormat_JsonSchema
4
5client = TwelveLabs(api_key="<YOUR_API_KEY>")
6
7# Create a knowledge store with a domain-specific ingestion configuration
8store = client.knowledge_stores.create(
9 name="Microdrama Source Material",
10 ingestion_config=IngestionConfig(
11 enrichment_config=EnrichmentConfig_Description(
12 description="Focus on characters and their emotions, interpersonal dynamics, conflicts, tension points, visual mood shifts, dialogue tone, and dramatic turning points. Track recurring characters across videos."
13 )
14 ),
15)
16store_id = store.id
17
18# Before continuing, add video assets to your knowledge store and wait
19# for processing to complete. See the Quickstart for details.
20
21# Define a JSON schema for the narrative output
22drama_schema = {
23 "type": "object",
24 "properties": {
25 "title": {"type": "string"}, # Title for the micro-drama
26 "logline": {"type": "string"}, # One-sentence summary
27 "characters": {
28 "type": "array",
29 "items": {
30 "type": "object",
31 "properties": {
32 "name": {"type": "string"}, # Character name
33 "role": {"type": "string"}, # Protagonist, antagonist, etc.
34 "arc": {"type": "string"}, # Character development summary
35 },
36 },
37 },
38 "scenes": {
39 "type": "array",
40 "items": {
41 "type": "object",
42 "properties": {
43 "scene_number": {"type": "integer"}, # Position in sequence
44 "description": {"type": "string"}, # What happens in the scene
45 "video_reference": {"type": "string"}, # Source video identifier
46 "timestamp": {"type": "string"}, # Clip timestamp
47 "dramatic_function": {"type": "string"}, # Role in the narrative arc
48 },
49 },
50 },
51 "central_conflict": {"type": "string"}, # Main dramatic tension
52 "resolution": {"type": "string"}, # How the conflict resolves
53 },
54}
55
56# Request a narrative outline
57response = client.responses.create(
58 knowledge_store_id=store_id,
59 instructions="You are a micro-drama creator. Analyze the video collection for narrative potential. Identify characters, conflicts, and dramatic moments. Construct a compelling 60-second micro-drama outline.",
60 input=[{"type": "message", "role": "user", "content": "Create a micro-drama from these videos. Focus on the strongest emotional arc you can find."}],
61 text=TextParam(format=TextParamFormat_JsonSchema(name="micro_drama", schema_=drama_schema)),
62)
63
64# Parse and read the outline
65session_id = response.session_id
66for output in response.output:
67 if output.type == "message":
68 for content in output.content:
69 drama = json.loads(content.text)
70 print(f"Title: {drama['title']}")
71 print(f"Logline: {drama['logline']}")
72 print(f"Conflict: {drama['central_conflict']}")
73 for scene in drama["scenes"]:
74 print(f" Scene {scene['scene_number']}: {scene['description']}")
75
76# Refine the output in a follow-up turn
77response = client.responses.create(
78 knowledge_store_id=store_id,
79 session_id=session_id,
80 input=[{"type": "message", "role": "user", "content": "Make the conflict sharper. Can you find a stronger turning point moment?"}],
81 text=TextParam(format=TextParamFormat_JsonSchema(name="micro_drama", schema_=drama_schema)),
82)
83
84# Process the refined output
85for output in response.output:
86 if output.type == "message":
87 for content in output.content:
88 drama = json.loads(content.text)
89 print(f"\nRefined Title: {drama['title']}")
90 print(f"Logline: {drama['logline']}")
91 print(f"Conflict: {drama['central_conflict']}")
92 for scene in drama["scenes"]:
93 print(f" Scene {scene['scene_number']}: {scene['description']}")

Code explanation

1

Create a domain-focused knowledge store

Create a knowledge store with an ingestion configuration tailored to your domain, so the platform extracts the signals your task needs.
Function call: You call the knowledge_stores.create method.
Parameters:

  • name: The name of the knowledge store.
  • ingestion_config: An object that controls what the platform extracts during processing. You cannot change it after creation. It contains an enrichment_config object:
    • type: The enrichment type. Set to "description" for a natural-language description, or "json_schema" for precise typed fields. For details, see the Configure ingestion page.
    • description: A natural-language description of what to extract. This example focuses on narrative elements: characters, emotions, conflicts, and dramatic turning points.

Return value: An object of type KnowledgeStore with a field named id representing the unique identifier of the newly created knowledge store. Pass it to the Responses API calls that follow.

After creating the store, add video assets and wait for processing to complete. See the Quickstart page for the full upload and processing workflow.

2

Request a narrative outline

Generate the structured outline from the collection.
Function call: You call the responses.create method with domain instructions, a prompt, and a text parameter that describes your narrative schema.
Parameters:

  • knowledge_store_id: The unique identifier of the knowledge store to reason over.
  • input: An array of input items. Each item is a message you send to Jockey. This example asks Jockey to create a micro-drama, focusing on the strongest emotional arc.
  • instructions: A per-request system prompt that shapes Jockey’s behavior for a domain or task. This example uses a micro-drama creator role that identifies narrative elements.
  • text: An object that specifies the response format. To return structured JSON, set the format field: its schema_ field defines the structure the output must match, and its name field identifies the schema. Design the schema to fit your use case; the inline comments in the example describe each field. For the schema rules, see the Structured output page.

Return value: An object of type ResponseObject. Read the session_id field to continue in the follow-up turn, and the output items for the outline. The response also includes id, status, and usage.

3

Process the results

The text field of each content part in a message item of the output array is a JSON string that matches your schema. Parse it with json.loads(), then iterate the scenes array to read each scene. Keep the session_id from the response for the next step.

4

Refine the output in a follow-up turn

Continue the session with a new prompt that adjusts the result. Jockey retains the context from the first turn.
Function call: You call the responses.create method again with the session_id and the same schema.
Parameters:

  • knowledge_store_id: The unique identifier of the knowledge store. Must match the store from the first request.
  • session_id: The session identifier from the first response.
  • input: An array of input items. Each item is a message you send to Jockey. This example asks Jockey to sharpen the conflict and find a stronger turning point.
  • text: The same schema as the first request.

Return value: An object of type ResponseObject with the same structure as the first outline. Parse it the same way.

Example response

The text field inside each content part is a JSON string that matches your schema. After parsing, a typical result looks like this:

1{
2 "title": "The Unspoken Divide",
3 "logline": "Two colleagues navigate a silent rivalry that threatens to unravel their shared project.",
4 "characters": [
5 {
6 "name": "Maya",
7 "role": "Protagonist",
8 "arc": "Moves from passive avoidance to direct confrontation"
9 },
10 {
11 "name": "Daniel",
12 "role": "Antagonist",
13 "arc": "Shifts from confident control to vulnerability when challenged"
14 }
15 ],
16 "scenes": [
17 {
18 "scene_number": 1,
19 "description": "Maya and Daniel present their project update, tension visible in their body language",
20 "video_reference": "069eb4e8-aeb0-7e83-8000-86413fcc296a",
21 "timestamp": "02:15-02:42",
22 "dramatic_function": "Establishes the core tension between the two characters"
23 },
24 {
25 "scene_number": 2,
26 "description": "Maya reviews documents alone, frustration building as she discovers discrepancies",
27 "video_reference": "069e1e97-27f8-7a8e-8000-bf32ecd7fc8c",
28 "timestamp": "08:30-08:55",
29 "dramatic_function": "Rising action - reveals the source of the conflict"
30 },
31 {
32 "scene_number": 3,
33 "description": "Maya confronts Daniel in the hallway, his composure breaks",
34 "video_reference": "069eb4e8-aeb0-7e83-8000-86413fcc296a",
35 "timestamp": "14:02-14:28",
36 "dramatic_function": "Climax - the unspoken divide becomes spoken"
37 }
38 ],
39 "central_conflict": "Maya suspects Daniel has been taking credit for her contributions, but neither acknowledges it directly",
40 "resolution": "Maya presents her work independently, forcing a public acknowledgment of her contributions"
41}
Notes
  • The video_reference field contains the asset identifier of the knowledge store item.
  • The timestamp field is a range in MM:SS-MM:SS format indicating the clip boundaries.
  • Jockey returns narrative outlines with clip references (video identifiers and timestamps), not rendered video files. Use these references with your editing tool or pipeline to assemble the final content.

Variations

Change the ingestion configuration, instructions, or prompt to adapt this pattern for different domains and creative directions.

  • Change the genre: Set the instructions parameter to “horror micro-drama creator”, “comedy sketch writer”, or “documentary short producer.”
  • Focus on a character: Change the prompt to “Build the drama around the most frequently appearing person.”
  • Create a series: Use a multi-turn session to extend the story: “Create episode 2, continuing from this ending.”
  • Change the domain: Replace the ingestion configuration, instructions, and schema to build a training curriculum designer, compliance auditor, or content planner.

Jupyter notebook

Open In Colab

See also