Merchant Data Ingestion

Problem

Collected web data often sits in ad hoc tasks, logs, or intermediate files, while raw html and parsed json lack a consistent persistence target. This makes it hard to identify when a page structure changed, and increases the risk of overwritten records or inconsistent fields. When page structure, API payloads, or merchant data change frequently, temporary files are not a reliable long-term reference.

How It Works

The skill turns ingestion into a concrete write pipeline:
- Database connection: connects to PostgreSQL using provided credentials.
- Data preparation: assembles merchant_id, url, html, json, and a timestamp.
- Targeted writes: stores raw HTML in merchant_raw_pages, and structured JSON in business tables or JSONB fields.
- Versioned snapshots: checks whether a snapshot exists for the day, then creates a new version record when needed for traceability.
- Result feedback: returns success, failure, and error details for downstream task handling.

Boundaries

It does not replace cleaning rules; it persists the cleaned raw page and structured result by merchant dimension. For long HTML or high-volume writes, account for PostgreSQL storage limits, index pressure, and write performance. Sensitive fields should be masked before insertion.