DataPilot

Overview

An LLM planning agent turns a prompt into a collection plan, runs robots.txt-aware collectors across 9 sources in parallel, then validates, de-duplicates and confidence-scores the rows into a Postgres dataset.

How it works

• Planner agent chooses collectors per prompt: job boards, Hacker News, GitHub, Wikipedia, CoinGecko, web search and URL fetch.
• Collectors respect robots.txt and run in parallel after the response is sent.
• Validator merges duplicates and assigns a confidence score to every row.
• Rule-based fallback keeps the pipeline running when every LLM provider fails.