Why manual catalogue work is a bottleneck
Retail teams spend significant time manually writing product descriptions and tagging attributes across large catalogues. For a brand like Meridian Athletic with hundreds of SKUs across apparel, footwear, and gear, this is slow, inconsistent, and error-prone — especially when images are the primary source of truth for new arrivals.
The objective was to build a multimodal agent that accepts product images and returns structured, retailer-ready output — not a generic image caption, but a validated JSON object with named attributes, quality flags, and an overall quality verdict that a merchandiser or catalogue manager can act on immediately.
The Problem
Product catalogues require consistent descriptions, accurate attribute tagging, and quality control at scale. Manual processes create bottlenecks and inconsistency — especially in fast-moving retail environments where new inventory arrives constantly.
The Objective
A single-call vision agent that returns a strict JSON schema — product name, category, description, five named attributes, typed quality flags, and an overall quality verdict — from a product image with optional category and brand context.
What the agent returns for every image
Rather than free-form text, the agent is constrained to return a defined JSON schema — enforced through the system prompt. Each field serves a specific downstream use case: descriptions feed PDP copy, attributes populate filter systems, quality flags route images to review queues.
A concise, retailer-appropriate product name derived from visual analysis — type, style, and key distinguishing feature.
The model's visual classification of product type — confirmed or corrected against the optional category hint provided at input.
A product description written for a PDP or catalogue — factual, benefit-led, and ready to use with minimal editing.
Color, material, style, condition, and target audience — extracted from the image and structured for faceted search and filter systems.
Observations about image or product quality — well-lit background, visible defect, poor contrast, strong lifestyle context. Each flag is typed for colour-coded UI rendering.
A three-state verdict for routing — pass goes live, review routes to a merchandiser, fail triggers a reshoot request. Automates QA triage at catalogue scale.
How the agent was architected
Several deliberate constraints shape how Agent 06 behaves — each one a product decision with a clear rationale, not a default technical choice.
Each image is analysed in one API call. No multi-step chains, no pre-processing calls. This keeps latency low, cost predictable, and the architecture auditable. The tradeoff is that the system prompt must do more work to constrain output quality.
The system prompt instructs the model to return only a raw JSON object — no markdown fences, no preamble, no explanation. The front end strips any accidental fences before parsing and surfaces a legible error if the model goes off-schema.
Category and brand context are appended to the user message at runtime — not hardcoded into the system prompt. This keeps the system prompt under 300 tokens and makes the agent reusable across brands and categories without re-prompting.
Images are encoded to base64 in the browser before being sent to the Cloudflare Worker. No server-side file storage, no upload pipeline, no signed URL management. The Worker passes the image content block directly to the Anthropic API.
Each quality flag carries a type: positive, caution, or negative. This isn't cosmetic — it's designed so that a downstream system can parse the JSON and route images to the correct workflow without reading the flag text. The UI colour-coding is a visible demonstration of that logic.
How structured output is enforced
The system prompt is under 300 tokens — deliberately compact. It defines the output schema inline, prohibits any non-JSON response, and leaves no ambiguity about field types. The model has no discretion over format; only content is variable.
// System prompt enforces schema — user turn carries context SYSTEM: You are a product image analyst. Given a product image, return ONLY valid JSON (no markdown, no prose) with this schema: { "product_name": string, "category": string, "description": string, // 2-3 sentences, retailer-ready "attributes": { "color": string, "material": string, "style": string, "condition": string, "target_audience": string }, "quality_flags": [ { "flag": string, "type": "positive"|"caution"|"negative" } ], "overall_quality": "pass"|"review"|"fail" } // USER TURN appends runtime context: // "Analyse this product. Category hint: apparel. Context: Meridian Athletic, premium tier." // Context in user turn → system prompt stays reusable across brands
What this agent teaches
Demonstrates how to pass image content blocks — base64-encoded JPG, PNG, or WEBP — to the Anthropic API alongside text instructions in a single messages array call.
The system prompt constrains the model to return a defined schema with no markdown, no preamble, and no freeform deviation — a production pattern for any structured output use case.
Five named attributes — color, material, style, condition, target audience — are extracted from visual analysis and structured for faceted search, filter systems, and PIM ingestion.
Quality observations are typed — positive, caution, or negative — so downstream systems can route images programmatically without parsing natural language. The UI colour-coding makes this logic visible.
System prompt under 300 tokens, max output 800 tokens, single call per interaction. Demonstrates deliberate cost and latency discipline — important for any agent operating at catalogue scale.
Brand and category context is injected at runtime via the user turn — not hardcoded into the system prompt. The same agent works across multiple brands and product categories without modification.
What I would do differently
JSON enforcement requires redundancy in the system prompt. Instructing the model to "return only JSON" once is not enough — models occasionally add a preamble or wrap output in markdown fences, especially when uncertain. The prompt needs to state the constraint multiple ways, and the front end needs a fence-stripping fallback before attempting to parse. In production, a JSON mode API parameter (where available) would remove this fragility entirely.
Image quality directly affects output quality. The model performs significantly better on clean, well-lit product images against neutral backgrounds. Lifestyle shots with complex backgrounds reduce attribute accuracy — particularly for material and condition fields. In a production catalogue pipeline, image pre-screening (resolution, background complexity) before sending to the vision model would improve overall accuracy.
The overall_quality verdict needs calibration, not just instruction. Telling the model to return "pass", "review", or "fail" produces consistent formatting but inconsistent thresholds. Without explicit definitions of what constitutes each verdict — image resolution, background cleanliness, product visibility — the model applies its own judgement. In production, these thresholds should be defined explicitly in the system prompt with examples.
Sequential processing is a bottleneck for large batches. Agent 06 processes images one at a time to stay within the single-call-per-interaction constraint. For a catalogue of 50+ images, parallel processing with promise batching would be necessary. The current architecture is correct for a demo and for small batches; production scale requires a queue-based approach with rate limit management.
Context hints have a measurable impact on output relevance. Providing a category hint ("apparel") and brand context ("Meridian Athletic, premium tier") in the user turn noticeably improves description tone and attribute precision. Without context, descriptions are generic. This confirms that even single-call vision agents benefit from runtime context injection — a pattern worth standardising across the portfolio.