AI-Powered Supply Chain Marketplace

The client, Diagon, founded by ex-Tesla and Rivian veterans, set out to revolutionize the industrial machinery market by allowing manufacturers to source equipment "in minutes, not months".

While they had successfully validated their concept with a prototype, they faced a critical execution gap in scaling. The goal was to index and structure product data from over 32,000 disparate supplier websites—a task requiring massive scale and "human-like" reasoning to interpret unstructured technical specifications.

The existing infrastructure faced resource limits when handling heavy AI extraction workloads. To meet investor expectations, Diagon needed to transition from a lightweight MVP to a robust, enterprise-grade cloud architecture capable of autonomous decision-making to keep operational costs viable.

The partnership successfully evolved Diagon’s platform into a robust, investor-ready data engine capable of handling massive data volume with high precision.

• Intelligent AI Pipeline (The "Brain"): We moved beyond simple scraping to a cognitive agentic workflow. An AI Agent intelligently scans sitemaps to identify "Product Pages," reducing processing volume by ~80% and significantly cutting token costs.
• Adaptive Extraction & 95% Recall: The system uses a tiered approach (cost-effective parsers first, auto-healing with tools like Firecrawler second). The pipeline achieves a 95% success rate in identifying and extracting valid product pages.
• Infrastructure & Performance: We migrated heavy extraction workers to AWS ECS, eliminating timeouts and allowing for massive parallel processing. Terraform was implemented to ensure identical Staging and Production environments.
• Human-in-the-Loop Validation: We decoupled AI logic using LangSmith, allowing domain experts to adjust prompts ("Prompt-Ops") without needing developer deployment.
• Analytics & BI: We deployed a Dockerized Metabase instance to give leadership a real-time view of data coverage and supplier indexing progress.