TencentCloud VITA Image and Video Understanding

Problem Solved

In agent workflows, image captioning, video summarization, and content analysis for surveillance, news, product demos, or sports clips often require inconsistent handling: single images, ordered image sets, video URLs, custom prompts, and streaming outputs all need a clear contract. Local videos also must be converted into accessible URLs before analysis. TencentCloud VITA wraps Tencent Cloud's image and video understanding service with explicit script boundaries, preventing an agent from fabricating visual content with a text-only model.

How It Works

The core script scripts/main.py supports three input modes: a single image, multiple images in chronological order, and one video URL. It accepts image formats such as JPG, PNG, SVG, and WEBP, plus video formats such as MP4, MOV, AVI, and WebM; videos are limited to 10 minutes and 100MB. Images can be public URLs or local paths, with local files converted to base64 data URLs before calling the VITA API. Prompt precedence is: --prompt > the persisted file /prompt/vita_prompt.txt > the built-in default prompt. Output can be streamed as Server-Sent Events with --stream, and generation can be shaped with --temperature and --max-tokens.

Boundaries and Notes

--image and --video are mutually exclusive, and remote media must be publicly accessible. Local videos are not uploaded directly; another tool must first produce an accessible URL. The script does not include COS upload capability. When the VITA service fails, the agent should return a clear error instead of inventing results. This skill fits engineering workflows that need image or video understanding inside an agent, but it is not an object storage or upload gateway.