Shopping Agent Experience Evaluation

Problem

Shopping assistants can answer product recommendations, budget, brand preference, and feature questions, but single-turn impressions are not enough to compare Doubao and Qwen fairly. This skill provides a repeatable, quantified evaluation workflow for real e-commerce scenarios.

How It Works

• Query generation: Uses DeepSeek to create 30 user queries covering 11 shopping scenarios, including specific products, budget constraints, brand preferences, functional requirements, ambiguous requests, and conflicting needs.
• Model responses: Runs doubao-seed-2-0-pro and qwen-plus as shopping assistants to answer all queries.
• Expert scoring: Uses DeepSeek to score each answer on a 7-point scale across 11 dimensions, such as product relevance, persuasive recommendations, price reasonableness, factual accuracy, and conversational naturalness.
• Deliverables: Produces a DOCX report with donut and radar charts, plus an XLSX file containing queries, responses, dimension scores, and summary statistics.

Boundaries and Notes

It is best used for comparing shopping assistant experience, not for production-scale system load testing. The workflow makes many API calls and may run for 15–30 minutes, so DEEPSEEK_API_KEY, DOUBAO_API_KEY, and QWEN_API_KEY must be configured. Scores reflect the fixed query set and scoring prompts; validate with business-specific data before using them for model selection.