Node.js SDK

Search

The SearchWrapper class provides methods to perform search requests.

Methods

Make a search request

Description: This method performs a search across a specific index using text, media, or a combination of both as your query and returns a paginated iterator of search results.

Text queries:

  • Use the queryText parameter to specify your query.

Media queries:

  • Set the queryMediaType parameter to the corresponding media type (example: image).
  • For a single image, specify one of the following parameters:
    • queryMediaUrl: Publicly accessible URL of your media file.
    • queryMediaFile: Local media file. If you specify both, queryMediaUrl takes precedence.
  • For multiple images, specify one of the following parameters (Marengo 3.0 only, up to 10 images):
    • queryMediaUrls: Publicly accessible URLs of your media files.
    • queryMediaFiles: Local media files.

Composed text and media queries (Marengo 3.0 only):

  • Use the queryText parameter for your text query.
  • Set queryMediaType to image.
  • Specify your images using queryMediaUrl, queryMediaFile, queryMediaUrls, or queryMediaFiles.
Note

When using images in your search queries (either as media queries or in composed searches), ensure your image files meet the requirements.

Entity search (Marengo 3.0 only):

  • To find a specific person in your videos, enclose the unique identifier of the entity you want to find in the queryText parameter.

For instructions on setting up and using this feature, see the Entity search page.

Note

This method is rate-limited. For details, see the Rate limits page.

Function signature and example:

1query(
2  request: SearchWrapper.QueryRequest,
3  requestOptions?: Search.RequestOptions
4): Promise<core.Page<TwelvelabsApi.SearchItem>>

Parameters

NameTypeRequiredDescription
request[TwelvelabsApi.SearchCreateRequest]YesThe search request parameters.
requestOptions[Search.RequestOptions]NoRequest-specific configuration.

The SearchCreateRequest interface defines the parameters for performing a search:

NameTypeRequiredDescription
indexIdstringYesThe unique identifier of the index to search.
searchOptionsTwelvelabsApi.
SearchCreateRequestSearchOptionsItem[]		YesSpecifies the modalities the video understanding model uses to find relevant information.
Available options:
- visual: Searches visual content.
- audio: Searches non-speech audio (Marengo 3.0) or all audio (Marengo 2.7).
- transcription: Spoken words (Marengo 3.0 only)
You can specify multiple search options in conjunction with the operator parameter to broaden or narrow your search.
For detailed guidance and version-specific behavior, see the Search options section.
transcriptionOptionsTwelvelabsApi.
SearchCreateRequestTranscriptionOptionsItem[]		NoSpecifies how the platform matches your text query with the words spoken in the video. This parameter applies only when using Marengo 3.0 with the search_options parameter containing the transcription value.
Available options:
- lexical: Exact word matching
- semantic: Meaning-based matching
For details on when to use each option, see the Transcription options section.
Default: ["lexical", "semantic"].
queryTextstringNoThe text query to search for. This parameter is required for text queries. The maximum query length varies by model. Marengo 3.0 supports up to 500 tokens per query, while Marengo 2.7 supports up to 77 tokens per query.
queryMediaTypeTwelvelabsApi.SearchCreateRequestQueryMediaTypeNoThe type of media you wish to use. This parameter is required for media queries. For example, to perform an image-based search, set this parameter to image. Use queryText together with this parameter when you want to perform a composed image+text search.
queryMediaFileFile | fs.ReadStream | Blob | (File | fs.ReadStream | Blob)[]NoA local media file to use as a query. Pass a single file or an array of up to 10 files (Marengo 3.0). This parameter is required for media queries if queryMediaUrl is not provided.
queryMediaUrlstringNoThe publicly accessible URL of a media file to use as a query. This parameter is required for media queries if queryMediaFile is not provided.
queryMediaFiles(File | fs.ReadStream | Blob)[]NoA list of local media files to use as a query. Marengo 3.0 only. You can provide up to 10 images in total.
queryMediaUrlsstring[]NoA list of publicly accessible URLs of media files to use as a query. Marengo 3.0 only. You can provide up to 10 images in total.
adjustConfidenceLevelnumberNoThe strictness of the thresholds for assigning the high, medium, or low confidence levels to search results. This parameter is deprecated in Marengo 3.0 and newer versions. Use the rank field in the response instead, which indicates the relevance ranking assigned by the model. Min: 0, Max: 1, Default: 0.5.
groupByTwelvelabsApi.SearchCreateRequestGroupByNoUse this parameter to group or ungroup items in a response. Values: video, clip. Default: clip.
thresholdTwelvelabsApi.ThresholdSearchNoFilter on the level of confidence that the results match your query. This parameter is deprecated in Marengo 3.0 and newer versions. Use the rank field in the response instead, which indicates the relevance ranking assigned by the model. Values: high, medium, low, none.
sortOptionTwelvelabsApi.SearchCreateRequestSortOptionNoThe sort order for the response. Values:
- score: Sorts results by relevance ranking in ascending order (1 = most relevant). When groupBy is video, sorts videos by highest relevance ranking (lowest number) among their clips
- clipCount: Sorts videos by the number of matching clips in descending order. Clips within each video are sorted by relevance ranking in ascending order. Only available when groupBy is set to video.

Default: score.
operatorTwelvelabsApi.SearchCreateRequestOperatorNoLogical operator for combining search options. Values: or, and. Default: or.
pageLimitnumberNoThe number of items to return on each page. Max: 50.
filterstringNoA stringified object to filter search results based on video metadata or custom fields.
includeUserMetadatabooleanNoSpecifies whether to include user-defined metadata in the search results.

Return value

Returns a Promise that resolves to a Page<SearchItem> object that implements AsyncIterable, allowing you to iterate through the paginated search results.

The Page class contains the following properties and methods:

NameTypeDescription
dataT[]An array containing the current page of items.
getNextPage()Promise<this>Retrieves the next page and returns the updated Page object.
hasNextPage()booleanReturns whether there is a next page to load.
Symbol.asyncIteratorAsyncIterator<T>Allows iteration through all items across all pages using for await loops.

The SearchItem interface contains the following properties:

NameTypeDescription
scorenumberNOTE: Marengo 3.0 and newer versions do not return this field. Use the rank field instead.

The score indicating how well the clip matches the search query.
startnumberThe start time of the clip in seconds.
endnumberThe end time of the clip in seconds.
videoIdstringThe unique identifier of the video. Once the platform indexes a video, it assigns a unique identifier.
confidencestringNOTE: Marengo 3.0 and newer versions do not return this field. Use the rank field instead.

The confidence level of the match (high, medium, low).
ranknumberNOTE : Only Marengo 3.0 and newer versions return this field. Earlier versions return score and confidence instead.

The relevance ranking assigned by the model. Lower numbers indicate higher relevance, starting with 1 for the most relevant result.
thumbnailUrlstringThe URL of the thumbnail image for the clip.
transcriptionstringA transcription of the spoken words that are captured in the video.
idstringThe unique identifier of the video. Only appears when the group_by=video parameter is used in the request.
userMetadataRecord<string, unknown>User-defined metadata associated with the video.
clipsSearchItemClipsItem[]An array that contains detailed information about the clips that match your query. The platform returns this array only when the group_by parameter is set to video in the request.

The SearchItemClipsItem interface contains the following properties:

NameTypeDescription
scorenumberNOTE: Marengo 3.0 and newer versions do not return this field. Use the rank field instead.

The score indicating how well the clip matches the search query.
startnumberThe start time of the clip in seconds.
endnumberThe end time of the clip in seconds.
confidencestringNOTE: Marengo 3.0 and newer versions do not return this field. Use the rank field instead.

The confidence level of the match (high, medium, low).
ranknumberNOTE : Only Marengo 3.0 and newer versions return this field. Earlier versions return score and confidence instead.

The relevance ranking assigned by the model. Lower numbers indicate higher relevance, starting with 1 for the most relevant result.
thumbnailUrlstringThe URL of the thumbnail image for the clip.
transcriptionstringA transcription of the spoken words that are captured in the clip.
videoIdstringThe unique identifier of the video for the corresponding clip.
userMetadataRecord<string, unknown>User-defined metadata associated with the video.

API Reference

Any-to-video search.

Error codes

This section lists the most common error messages you may encounter while performing search requests.

  • search_option_not_supported
    • Search option {search_option} is not supported for index {index_id}. Please use one of the following search options: {supported_search_option}.
  • search_option_combination_not_supported
    • Search option {search_option} is not supported with {other_combination}.
  • search_filter_invalid
    • Filter used in search is invalid. Please use the valid filter syntax by following filtering documentation.
  • search_page_token_expired
    • The token that identifies the page to be retrieved is expired or invalid. You must make a new search request. Token: {next_page_token}.
  • index_not_supported_for_search:
    • You can only perform search requests on indexes with an engine from the Marengo family enabled.

For a list of errors specific to this endpoint and general errors that apply to all endpoints, see the Error codes page.