SN
All projects
Data Engineering · AI Agents Featured2025

AI-Assisted Job-Scraping Platform

A multi-source job scraper whose "source builder" uses an AI agent to generate a validated scraper config from any URL.

Private repository
200+
Automated tests (live + non-live)
minutes
URL → working scraper
3
Fetch modes (basic / stealth / Playwright)

Tech stack

  • Python
  • FastAPI
  • Playwright
  • Alpine.js
  • SQLite
  • Claude / GPT SDKs
  • Directus
  • pytest

Problem

Onboarding a new job board normally means a developer hand-writing CSS/XPath selectors and pagination logic for each site — slow, brittle, and expensive to maintain as sites change. Scaling ingestion to many sources this way does not work.

Approach

I built a FastAPI + Playwright platform that ingests job boards from YAML / CSS-selector configs with pagination and stealth fetch modes for JavaScript-heavy sites. The standout feature is an agentic source builder: you paste a job-board URL and an LLM (Claude / GPT via SDK) drives a live browser, snapshots the DOM, infers the listing and pagination selectors, and emits a complete, validated SourceConfig that opens in the editor for review. Results are normalized and loaded into a headless CMS (Directus) with dedupe and freshness tracking.

Results

200+ automated tests across non-live and live suites keep the platform reliable as sites change. Onboarding a new source dropped from hours of manual selector-writing to minutes: the agent turns a URL into a working, validated scraper you can review and ship immediately.

Notable engineering

  • Agentic source builder: an LLM (Claude / GPT via SDK) drives a live browser, snapshots the DOM, infers listing & pagination selectors, and emits a complete, validated SourceConfig for review.
  • Multi-source scraping from declarative YAML / CSS-selector configs with pagination and automation steps.
  • Three fetch modes — basic, stealth, and full Playwright — for static and JavaScript-heavy sites.
  • Normalized load into a headless CMS (Directus) with dedupe and freshness tracking (first_seen / last_seen).
  • Bounded per-source health checks and validation, backed by 200+ automated tests.

Screenshots & diagrams

Real UI captures and figures — source is private and client brands are not shown.

The sources dashboard — health, jobs found and freshness per source.
"Generate Source with AI" — an LLM turns a URL into a validated config.
Overview dashboard — run stats, freshness and extraction-drift alerts.
Normalized jobs & companies, ready to push to the CMS.
The ingestion pipeline — config → scrape → normalize → CMS.

Deep dive

The platform treats a scraper as data, not code: every source is a declarative SourceConfig (selectors, pagination, fetch mode, automation steps) that can be version-controlled, validated, and health-checked. That design is what makes the AI builder possible — the agent's job is simply to produce a valid config, which the system already knows how to run and test.

The agent runs a tight perceive→act loop against a real browser session: it navigates, captures the rendered DOM, proposes selectors for the listing and individual fields, verifies them against live results, and iterates until the config validates. A live log surfaces each iteration, the tool calls, and the pages visited, so a human stays in the loop and can accept, tweak, or reject the generated source.