Smart Drawing Image Understanding

Specific Problem Addressed

In image-driven applications like content moderation, data analysis, or automated workflows, extracting structured information or performing semantic understanding from images is essential. Traditional approaches often require complex preprocessing models, while large models offer end-to-end solutions, but integrating them involves API alignment, payment management, and error handling.

The Smart Drawing Image Understanding skill encapsulates synchronous image analysis capabilities, allowing developers to submit image URLs or Base64 data via a simple interface and receive textual understanding results. This reduces the burden of building visual models from scratch, especially suitable for scenarios requiring rapid integration of image analysis features.

Core Capabilities and Workflow

The skill's core is vision capability, processing image understanding requests synchronously. Key steps include:

1. Query Models (optional): Verify model availability via GET /api/v1/llm/models.
2. Submit Request: Use POST /api/v1/llm/invoke with a request body where the messages array contains image and text content. Two image input formats are supported:
   - URL images: Provide a public https link in image_url.url.
   - Base64 images: Use the data:image/...;base64,... format.
3. Handle Payment: If HTTP 402 is returned, invoke the payment interface, then retry with the same JSON body, adding payment-related headers like WeixinPay-Required and X-Payment-Id.
4. Parse Response: Extract model reply text from data.choices[0].message.content.

The request body follows OpenAI-compatible format, with payload fields including:
- messages: Conversation array supporting text and image_url types.
- Optional parameters: system, max_tokens, temperature, etc.

Example request (URL image):

{
  "model": "ixhlink-skills-image-explain",
  "capability": "vision",
  "payload": {
    "messages": [
      {
        "role": "user",
        "content": [
          { "type": "text", "text": "Describe this image" },
          { "type": "image_url", "image_url": { "url": "https://example.com/image.jpg" } }
        ]
      }
    ]
  }
}

Caveats and Boundaries

• Payment Pre-check: Before calling, ensure the Agent has payment capabilities installed (e.g., weixinpay extension), otherwise prompt and terminate. Service pricing is 0.3 yuan per call, requiring user confirmation before payment flow.
• Error Handling: Common HTTP errors include:
• 400: Parameter errors, such as missing model or invalid message format.
• 402: Payment required or payment failure.
• 404: Model not enabled.
• 503: Service unavailable.
  For debugging, use data._billing.call_log_id to check logs.
• Synchronous Limitation: The skill is synchronous, suitable for low-latency scenarios; high-volume requests may need queue management.
• Image Formats: Only publicly accessible URLs or Base64 encoded images are supported; local files must be uploaded first.