- 100%
- Detection (500 / 500)
- 0.88
- Mean confidence
- 30
- ATS providers identified
- 1,651
- Feature dimensions
Tech stack
- Python
- scikit-learn
- XGBoost
- RandomForest
- SHAP
- Playwright
- Selenium
- BeautifulSoup
- pytest
Problem
Companies run their careers pages on dozens of different ATS platforms — Workday, Greenhouse, Lever, and many more. Identifying which one each company uses, at scale and reliably, is hard: pages are wildly inconsistent, frequently defended against scraping, and provide no standard signal that names the underlying platform.
Approach
I built a 4-tier detection cascade that stops at the first confident match: (1) a curated table of known company→ATS mappings, (2) URL-pattern regex with 75+ provider fingerprints, (3) rule-based analysis of HTML signals (scripts, forms, meta tags, asset hosts), and (4) a RandomForest ML classifier operating over a 1,651-dimension feature union with SHAP explainability. Career pages for all 500 Fortune 500 companies are fetched through a resilient Playwright / Selenium / BeautifulSoup cascade with HTML caching, so a blocked or JS-heavy page falls back gracefully instead of failing.
Results
100% detection (500 / 500) across 30 ATS providers, with a mean confidence of 0.88. The system was validated with 90 passing unit tests, k-fold cross-validation, a ground-truth check, and an active-learning loop that surfaces low-confidence samples for human review. The analysis also produced a clear market finding: Workday and Greenhouse together account for roughly 70% of Fortune-500 ATS usage — a duopoly.
Notable engineering
- 4-tier detection cascade that stops at the first confident match — known-mappings → URL-pattern regex (75+ fingerprints) → rule-based HTML signal analysis → RandomForest ML classifier.
- Resilient fetch layer cascades Playwright → Selenium → BeautifulSoup with HTML caching to survive anti-bot defenses across inconsistent career pages.
- SHAP explainability over a 1,651-dimension feature union makes every ML prediction auditable.
- Validated with 90 passing unit tests, k-fold cross-validation, a ground-truth check, and an active-learning loop that queues uncertain samples for review.
- Market finding: Workday + Greenhouse form a duopoly (~70% of Fortune-500 ATS).
Screenshots & diagrams
Real UI captures and figures — source is private and client brands are not shown.
Deep dive
The thesis treats ATS detection as a layered decision problem rather than a single model. Cheap, high-precision signals run first (exact known mappings and URL fingerprints), and only the genuinely ambiguous pages fall through to the more expensive ML classifier. This keeps the system fast, cheap, and — crucially — explainable: most answers come with a human-readable reason, and the ML tier ships with SHAP attributions for the rest.
The feature union spans URL structure, HTML/DOM signals, embedded script and asset hosts, form endpoints, and text fingerprints, producing 1,651 dimensions that are reduced and fed to a RandomForest classifier. An active-learning loop continually queues the lowest-confidence predictions for review, so the ground-truth set — and the model — improve over time.