> This page is for version v1.3 (default). > For other versions, use one of these documentation indexes: > - v1.3 (default): https://beta.docs.twelvelabs.io/v1.3/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://beta.docs.twelvelabs.io/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://beta.docs.twelvelabs.io/_mcp/server. # Pegasus > Pegasus is a generative model. It analyzes multiple modalities to generate contextually relevant text. Pegasus is a generative model for video-to-text generation. Pegasus analyzes multiple modalities to generate contextually relevant text based on the content of your videos. # Available versions #### [Pegasus 1.6](/v1.3/docs/concepts/models/pegasus/pegasus-1-6) Analyze videos or images to generate text, with egocentric video understanding, improved entity recognition, and in-segment events. #### [Pegasus 1.5](/v1.3/docs/concepts/models/pegasus/pegasus-1-5) Analyze videos to generate summaries, detailed descriptions, and structured responses. # Examples ## Segmenting egocentric video into atomic actions This example request segments first-person factory footage into atomic actions and labels the hand used for each one. The output is truncated to the first three segments. **`Request`** ```json title="Request" showLineNumbers={false} wordWrap maxLines=20 { "model_name": "pegasus1.6", "analysis_mode": "time_based_metadata", "video": { "type": "asset_id", "asset_id": "" }, "temperature": 0, "min_segment_duration": 2, "max_tokens": 98304, "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "atomic_action", "description": "Cut this egocentric video into atomic actions. One segment is ONE uninterrupted motion by one hand, or by both hands when they act on the same object together. Cut a NEW segment at EVERY change in what a hand is doing — a new contact, a release, a re-grasp, a change of direction, a change of object, a change of which hand is acting, a correction, a nudge, a pause. Small and incidental motions count: a two-second re-grip is its own segment. Never summarise several motions into one segment, and never skip a motion because it is brief. A repeated activity is one segment per repetition, never one segment for the whole run. Cover the clip end to end — consecutive segments touch, each starting where the previous one ended, with no gaps and no overlaps — and hold the same level of detail in the last minute as in the first.", "fields": [ { "name": "action", "type": "string", "description": "Exactly ONE action: the single motion this segment covers, as verb + object + hand. Never a list and never a sequence — if you are about to join two verbs with 'and' or a comma, that is two segments, not one. Name the motion, not its purpose." }, { "name": "hands_used", "type": "string", "description": "Which hand performs THIS action. Use 'both' only when the two hands act on the same object at the same moment and neither could do it alone. If one hand acts while the other only holds, steadies, supports or waits, the value is the ACTING hand alone, not 'both'. Use 'none' when neither hand is manipulating anything. Asymmetric work is the norm, so do not default to 'both' because two hands are visible.", "enum": ["left", "right", "both", "none"] }, { "name": "description", "type": "string", "description": "One sentence describing this single motion in full." } ] } ] } } ``` **`Output`** ```json title="Output" showLineNumbers={false} wordWrap maxLines=20 { "atomic_action": [ { "start_time": 0, "end_time": 2, "metadata": { "action": "hold screwdriver", "hands_used": "right", "description": "The right hand holds the pneumatic screwdriver steady while the left hand holds a small screw on the workbench." } }, { "start_time": 2, "end_time": 5, "metadata": { "action": "hold screwdriver", "hands_used": "right", "description": "The right hand holds the pneumatic screwdriver steady while the left hand holds a small screw on the workbench." } }, { "start_time": 5, "end_time": 7, "metadata": { "action": "reach for motor", "hands_used": "left", "description": "The left hand reaches forward and grasps the motor assembly to pull it closer to the center of the workspace." } } ] } ``` ## Labeling task attempts in an egocentric cooking video This example request segments a first-person cooking video into task attempts and states the goal of each attempt. **`Request`** ```json title="Request" showLineNumbers={false} wordWrap maxLines=20 { "model_name": "pegasus1.6", "analysis_mode": "time_based_metadata", "video": { "type": "asset_id", "asset_id": "" }, "temperature": 0, "min_segment_duration": 10, "max_tokens": 98304, "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "task", "description": "A coherent task attempt by the camera-wearer, from the first action that begins it to the point where its goal is achieved or abandoned. Do not split a task into its constituent motions — one segment per task attempt. Example boundaries: the whole of 'pack one shirt into a box', not the reach, the fold and the place separately.", "fields": [ { "name": "goal", "type": "string", "description": "Imperative phrasing of the task goal, as a language command would be given to a robot, e.g. 'fold the shirt and place it in the box'. Keep it short — a command, not a narration. Under 60 characters, no 'the person…', no account of what happened." }, { "name": "description", "type": "string", "description": "What the camera-wearer actually did, in one sentence: the objects involved and the order of the steps. Say what the goal does not already say — never restate the goal in the third person. Maximum 200 characters." } ] } ] } } ``` **`Output`** ```json title="Output" showLineNumbers={false} wordWrap maxLines=20 { "task": [ { "start_time": 0, "end_time": 38, "metadata": { "goal": "Add honey to the sauce in the bowl", "description": "The person retrieves a honey bottle from the cabinet, opens it, and pours honey into a measuring spoon before adding it to the sauce in the bowl." } }, { "start_time": 38, "end_time": 59, "metadata": { "goal": "Check the recipe on the laptop", "description": "The person walks to the laptop, points at the screen, and uses the trackpad to scroll through the recipe instructions." } }, { "start_time": 59, "end_time": 75, "metadata": { "goal": "Add broccoli and mushrooms to the pan", "description": "The person picks up chopped broccoli and mushrooms from the cutting board and places them into the pan on the stove." } }, { "start_time": 75, "end_time": 97, "metadata": { "goal": "Stir the vegetables in the pan", "description": "The person retrieves a spatula, stirs the vegetables in the pan, and adjusts the stove's heat setting." } }, { "start_time": 97, "end_time": 118, "metadata": { "goal": "Add green pepper to the pan", "description": "The person picks up chopped green pepper from the cutting board and adds it to the pan, then stirs the mixture." } }, { "start_time": 118, "end_time": 130, "metadata": { "goal": "Check the recipe on the laptop again", "description": "The person returns to the laptop and uses the trackpad to scroll through the recipe instructions." } }, { "start_time": 130, "end_time": 150, "metadata": { "goal": "Stir the sauce in the bowl", "description": "The person picks up the bowl containing the sauce and stirs it with a spoon." } } ] } ``` ## Auditing privacy exposure in egocentric video This example request flags faces, bystanders, documents, and screens in first-person factory footage for a privacy review. The output is truncated to one segment per definition. **`Request`** ```json title="Request" showLineNumbers={false} wordWrap maxLines=20 { "model_name": "pegasus1.6", "analysis_mode": "time_based_metadata", "video": { "type": "asset_id", "asset_id": "" }, "temperature": 0, "min_segment_duration": 2, "max_tokens": 98304, "response_format": { "type": "segment_definitions", "segment_definitions": [ { "id": "face", "description": "A stretch of time in which a human face is visible in frame. Start a new segment each time a face enters frame and end it when no face remains. Do not create a segment when no face is visible.", "fields": [ { "name": "identifiable", "type": "string", "description": "Whether the face is visible clearly enough and long enough that the person could be recognised from this footage. Use 'partially' when the face is turned, blurred by motion, or only briefly in frame.", "enum": ["yes", "partially", "no"] }, { "name": "description", "type": "string", "description": "One sentence stating where in the frame the face appears and what makes it identifiable or not." } ] }, { "id": "bystander", "description": "A stretch of time in which a person other than the camera-wearer is visible in frame, whether or not their face can be seen. Start a new segment each time such a person enters frame. Do not create a segment for the camera-wearer's own hands or body.", "fields": [ { "name": "position", "type": "string", "description": "Where in the frame the person is, relative to the work the camera-wearer is doing.", "enum": ["foreground", "background", "edge-of-frame"] }, { "name": "description", "type": "string", "description": "One sentence stating what is visible of this person and what they appear to be doing." } ] }, { "id": "document", "description": "A stretch of time in which printed or handwritten text on paper, card, or a label is visible in frame. Do not create a segment for text on an electronic display.", "fields": [ { "name": "legible", "type": "string", "description": "Whether the text could actually be read from this footage, as opposed to being recognisable as text.", "enum": ["yes", "partially", "no"] }, { "name": "description", "type": "string", "description": "One sentence stating what the document appears to be and whether any of its text can be made out." } ] }, { "id": "screen", "description": "A stretch of time in which an electronic display is visible in frame. Start a new segment each time a display enters frame.", "fields": [ { "name": "legible", "type": "string", "description": "Whether what the display is showing could actually be read from this footage.", "enum": ["yes", "partially", "no"] }, { "name": "description", "type": "string", "description": "One sentence stating what kind of display it is and what, if anything, can be made out on it." } ] } ] } } ``` **`Output`** ```json title="Output" showLineNumbers={false} wordWrap maxLines=20 { "face": [ { "start_time": 27, "end_time": 29, "metadata": { "identifiable": "partially", "description": "A face is visible in the upper left of the frame, seen in profile looking down at the workbench, but it is partially obscured by the angle and motion blur." } } ], "bystander": [ { "start_time": 27, "end_time": 30, "metadata": { "position": "foreground", "description": "A person wearing a black shirt and a head-mounted camera is visible on the left side of the frame, leaning over the workspace and reaching toward the phone with their right hand." } } ], "screen": [ { "start_time": 28, "end_time": 30, "metadata": { "legible": "no", "description": "A smartphone screen displaying a chat interface with green message bubbles, but the text is too blurry to read." } } ], "document": [ { "start_time": 42, "end_time": 44, "metadata": { "legible": "no", "description": "A white paper label or document is visible on the right side of the frame, but the text is too blurry and out of focus to be read." } } ] } ``` ## Summarizing educational videos This example prompt summarizes an educational video without any customization. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Summarize this video ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 This video presents an educational overview of advanced techniques for adapting pre-trained large language models (LLMs) to specialized tasks, with a focus on prompt tuning as a more efficient alternative to traditional fine-tuning and manual prompt engineering. The speaker, Martin Kevner, Master Inventor at IBM, begins by explaining that foundation models like ChatGPT are highly flexible but may need customization for specific applications. He contrasts three approaches: fine-tuning, which requires large amounts of labeled data and retraining; prompt engineering, where humans craft explicit instructions (hard prompts) to guide model behavior; and prompt tuning, which uses AI-generated, tunable soft prompts—numerical embeddings that are invisible to humans but highly effective. The video highlights that soft prompts outperform human-designed ones, enable faster adaptation, and are particularly valuable in multi-task and continual learning scenarios, where models must switch tasks or learn new concepts without forgetting prior knowledge. Despite the efficiency and power of prompt tuning, the speaker notes its lack of interpretability, as the AI cannot explain why certain embeddings were chosen. The video concludes with a reflective tone, acknowledging that while prompt engineering may become obsolete, the future lies in leveraging the underlying embedding layers, where "a string of numbers is worth a thousand words." The presentation is visually enhanced with dynamic digital annotations, including animated text, virtual writing, and diagrams illustrating model architectures and data flow, reinforcing the technical concepts through interactive visualization. ``` ## Generating captions for social media This example prompt generates a caption for a social media post. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Generate an attention-grabbing caption for a social media post. Keep it shorter than 200 characters. ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 Master the basics of golf in just 4 weeks! From alignment to grip, swing mechanics to mindset—learn how to play like a pro. Hit 30 balls this week, stay consistent, and have fun. #HowToPlayGolf #GolfTips ``` ## Table of contents This example prompt creates a table of contents detailing the main sections. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Provide a table of contents detailing the main sections of this video. ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 - Planning - Execution - Review & Improvement - Checklist: Enthusiast, Brand Message, Cheese - Pre-Armed Model (Pre-trained Model) - Model Architecture: Input, Player, Target, Output, Softmax - Feedback Loop and Data Flow - Prompt Tuning vs. Fine Tuning vs. Prompt Engineering - Applications in Multi-task and Continual Learning - Conclusion: The Future of Prompt Engineering ``` ## Company-wide memo This example prompt generates a company-wide memo. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Generate a company-wide memo based on the content of this video. ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 Subject: Enhancing Our Understanding of Machine Learning and Its Applications Dear Team, I hope this message finds you well. I’m sharing a concise yet comprehensive overview of a recent video that explores the fundamentals and real-world applications of machine learning—a field increasingly central to innovation across industries. Machine learning enables computers to learn from data and improve over time without being explicitly programmed for each task. First introduced in 1959 by Arthur Samuel at IBM, who developed a checkers-playing program that learned from experience, the concept has evolved dramatically. Today, predictive models powered by machine learning are embedded in everyday technologies—from self-driving cars and smart thermostats to personalized content recommendations and medical diagnostics. The core process begins with acquiring and cleaning large volumes of high-quality data, as the principle “garbage in, garbage out” underscores the importance of data integrity. Data scientists then perform feature engineering to transform raw data into meaningful inputs that better represent the underlying problem. The dataset is split into training and testing sets: the model learns from the training data, and its performance is validated using the test data. A wide range of algorithms can be used, from simple models like linear regression to complex ones such as convolutional neural networks (CNNs), which automatically generate features—especially valuable for image and natural language processing. These models improve by minimizing error through feedback mechanisms, using metrics like accuracy for classification or mean absolute error for regression. Python remains the dominant language in the field, supported by powerful frameworks like TensorFlow, PyTorch, and Scikit-learn, which streamline development and deployment. The final output is a trained model—a compact file that takes input data and produces predictions, ready to be deployed on devices or in the cloud to power real-world applications. This video not only demystifies the technical process but also highlights the transformative impact of machine learning across sectors. I encourage everyone to watch it to deepen their understanding of how data-driven intelligence is shaping the future of technology. Best regards, [Your Name] [Your Position] ``` ## Video annotations This example prompt identifies and lists key visual elements, scene changes, and notable events, briefly describing each. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Identify and list key visual elements, scene changes, and notable events in the video, briefly describing each. ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 The video opens with the Universal Sports logo (00:00–00:03), setting the stage for a high-profile athletics broadcast. It quickly transitions to a scoreboard displaying the results of the Men’s 100-meter final at the IAAF World Championships, highlighting Usain Bolt in first place (00:03–00:11). The scene shifts to the starting line, where the sprinters are lined up, and the camera pans across the athletes, emphasizing the elite field of runners who have all broken the 10-second barrier (00:11–00:16). At the start of the race (00:16), the runners explode from the blocks, with Daniel Bailey making a strong initial move, but Usain Bolt quickly takes control of the race, accelerating into his stride (00:16–00:26). The camera follows the race from a side-on perspective, capturing Bolt’s dominant lead as he pulls away from the field. As Bolt crosses the finish line (00:26–00:32), the camera shifts to a higher vantage point, emphasizing his commanding margin of victory—two full meters ahead of Tyson Gay, who finishes in second (00:32–00:45). The video then replays the race from a different angle, focusing on Bolt’s consistent speed and the growing gap between him and the rest of the field (00:45–01:17). A close-up shot captures Bolt’s reaction upon realizing his performance, showing a mix of disbelief and elation (01:17–01:28). The camera then switches to a top-down view, visually reinforcing Bolt’s lead throughout the race (01:28–01:42). The climax comes at 01:42, when the electronic timer displays the new world record: 9.58 seconds. The screen flashes the iconic image of Bolt’s victory, accompanied by the announcement of the record-breaking time (01:49–02:01). The moment is underscored by dramatic commentary, noting the significance of the performance and the slight tailwind of +0.9 m/s that aided the time but remained within the allowable limit (01:38–01:44). The video concludes with a final shot of Bolt celebrating, followed by the Universal Sports logo reappearing (02:03–02:09), marking the end of the broadcast. The entire sequence captures not only the athletic brilliance of Usain Bolt but also the global significance of his record-breaking run. ``` ## Video question answering This example prompt identifies the key takeaways. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 What are the key takeaways of this video? ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 The video explains the transformative power of transformer models in artificial intelligence, emphasizing their ability to process and generate human-like text and perform diverse tasks beyond language translation. Key takeaways include: transformers are sequence-to-sequence models composed of an encoder and decoder, which work together to understand context and generate outputs; they leverage an attention mechanism that allows parallel processing of input sequences, significantly speeding up training compared to older models like RNNs; they are pre-trained on large unlabeled datasets and fine-tuned with supervised learning, enabling versatility across applications such as joke generation, document summarization, blog writing, and even image processing and game playing; the video concludes by highlighting that while current outputs—like a banana joke—may not always be funny, the underlying technology is rapidly advancing and holds immense potential for future applications. ``` ## Timestamp breakdown This example prompt lists all timestamps in an advertisement where a close-up of the product appears. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Tell me all the timestamps in the advertisement where a closeup of the product appears. ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 The advertisement features a close-up of the product, Gatorade, at the following timestamps: - [00:55] – A young man is shown drinking directly from a Gatorade bottle, captured in a clear close-up shot that highlights the product. ``` ## Police report This example prompt creates a police report using a specific template for a video showing a robbery. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Create a police report based on what happened in the video. Provide the exact time range where the suspect appears in the video ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 Police Report Incident Description: A suspicious individual was observed approaching a package located on a wooden deck, picking it up, and departing through a white gate. The incident was captured on video from a fixed camera angle, providing a clear visual of the suspect’s actions and appearance. Suspect Description: - Gender: Male - Clothing: Blue hoodie, grey pants - Additional Features: None observed Time Range of Suspect’s Appearance: 00:02 to 00:10 Video Context: The video footage begins at 00:00 with the package resting on the deck. At 00:02, the suspect enters the frame, walks toward the package, and takes it. The individual is seen moving steadily and deliberately, exiting the scene through the white gate by 00:10. The audio clip from 00:07 to 00:10 includes the phrase “so you,” which may be part of a conversation unrelated to the incident or possibly a fragment of the suspect’s speech, though no further context is available. Additional Information: The video provides a continuous and unobstructed view of the suspect’s movements. No other individuals are visible during the incident. The package was unattended and left on the deck prior to the suspect’s arrival. Please contact the police department if you have any further information or if you are the owner of the stolen package. ``` ## Using different languages ### Spanish This example prompt summarizes a video and indicates that the response should be in Spanish. Note that the prompt is in English, and the output is in Spanish. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Write a summary in Spanish. ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 Este video presenta a Tim Shields, fotógrafo profesional, quien comparte una guía práctica para tomar retratos de alta calidad. Comienza con una introducción sobre la importancia de los retratos, el escenario más común en fotografía, y explica cómo aplicar la regla de los tercios para una composición equilibrada: colocar al sujeto en uno de los tercios horizontales o verticales, evitando que mire hacia fuera del cuadro para no desviar la atención del espectador. Destaca que el rostro debe situarse en la línea superior de los tercios, preferiblemente en una intersección, y que elementos interesantes del fondo deben alinearse con los otros tercios. También advierte contra distracciones detrás de la cabeza del sujeto, como postes o ramas. A continuación, revela el "ingrediente secreto" para fotos profesionales: desenfocar el fondo para resaltar al sujeto. Explica cómo lograrlo con cámaras inteligentes (usando el modo retrato) o con cámaras DSLR ajustando el diafragma al valor más bajo (f-stop más pequeño) y usando el zoom al máximo. También aborda la iluminación, recomendando evitar que el sujeto esté de espaldas al sol y proponiendo soluciones como acercarse, usar flash o posicionar al sujeto con luz natural o artificial (clave y relleno) para lograr un efecto suave y profesional. Tim introduce su libro, *The Photo Cookbook*, que contiene 30 recetas fotográficas para dominar técnicas como la profundidad de campo, la composición, el equilibrio, las líneas guía, el enfoque y el uso de la luz. Destaca que no se trata de tener una cámara cara, sino de seguir pasos claros y estructurados. El video concluye con una invitación a adquirir el libro para mejorar las fotos, ganar reconocimiento en redes sociales, participar en concursos y tomar imágenes de las que sentirse orgulloso. ``` ### French This example prompt summarizes the three main takeaways of a video. Note that the prompt and the output are in French. **`Prompt`** ```markdown title="Prompt" showLineNumbers={false} wordWrap maxLines=5 Résumez les trois principaux points à retenir de cette vidéo ``` **`Output`** ```markdown title="Output" showLineNumbers={false} wordWrap maxLines=5 Cette vidéo explique les stratégies de réplication de bases de données essentielles pour les entretiens de conception système. Les trois principaux points à retenir sont : 1. La réplication est cruciale pour la résilience, la scalabilité et la réduction de la latence, notamment dans les systèmes distribués où les réseaux sont instables. 2. La réplication leader-follower, la plus courante, repose sur un nœud principal (leader) qui gère les écritures et les réplique vers des nœuds secondaires (followers), avec deux variantes : synchrones (haute cohérence, mais plus lente) et asynchrones (plus rapide, mais risque d’incohérence en cas de panne du leader). 3. Des alternatives comme la réplication multi-leader (améliore la disponibilité) et la réplication sans leader (comme dans DynamoDB, avec gestion des conflits par des mécanismes comme read-repair) offrent des compromis entre performance, cohérence et résilience, et doivent être choisies selon les besoins spécifiques du système. ``` # Support For support or feedback regarding Pegasus, contact [support@twelvelabs.io](mailto:support@twelvelabs.io).