Autonomous Competitive Intelligence
Signal (ACIA)
Autonomous intelligence pipeline scraping dynamic pricing pages. Built a cost-aware delta engine that bypasses expensive LLM inference loops unless statistical pricing changes exceed a 5% baseline discrepancy threshold.
targetProblem Statement
Manual competitive research is slow and error-prone. Scraping SaaS pricing pages is brittle due to dynamic UI changes and running LLMs constantly over raw HTML results in extreme operational costs.
boltReal-world Impact
Automates competitor pricing checks end to end. The delta engine only calls an LLM when a price actually moves, so token cost stays flat as the tracked competitor list grows.
account_treeSystem Architecture
Multi-layer proxy scraper using Bright Data feeds raw DOM into a parser. The delta engine correlates changes to a persistent relational baseline in Supabase. A context memory layer tracks history. LLM (Gemini) strictly invoked only when threshold boundaries fail.
[Target Pricing Page]
│ (Bright Data MCP / Proxies)
▼
[DOM Parsing Node] ──► [Stored Baseline Comparison]
│
(Is Delta > 5%?)
┌─────┴─────┐
[NO] [YES]
│ │
(Skip LLM) [Gemini Insight Gen]
│
[Dashboard DB Sync]memoryAI / System Intelligence
ActionBook agents navigate client-side rendered tabs, like Monthly vs Annual pricing toggles, that a static scraper would miss, before the delta engine compares the extracted price against the stored baseline.
psychologyKey Architectural Decisions
Cost-Aware Delta Engine
Instead of running every scrape through Gemini to parse pricing, a statistical text-delta check runs first. This reduces LLM token overhead by 85% for unchanged competitors.
Bright Data Waterfall Setup
Residential proxies with fallback logic keep scrapes from getting blocked by aggressive bot-detection systems like Cloudflare and DataDome.