All projects

Private, runs daily

Job Market Intelligence Pipeline

A daily crawler that pulls postings from recruiting APIs, scores fit and emails a ranked digest.

A digest email headed '4 new matches', listing job postings as cards with a location, a one-line reason and a match score. Employer names are blurred.
A digest email from September 2026, rebuilt from the saved run. Employer names are blurred.

In brief

The question
Out of thousands of live postings across employer career sites, which few are worth a person's time today?
The approach
A free keyword filter removes the obvious misses, a model scores the survivors against a profile, and a ranked digest goes out by email.
The result
It runs unattended every morning. A typical day of 10 to 30 new postings costs a few cents to score, and nothing is scored twice.

What it does

Every morning at 8:00 New York time the pipeline checks a list of job boards, scores each new posting against my resume, and emails me the ones worth reading. It runs on GitHub Actions, so it works whether or not my laptop is on.

The problem has the same shape as lead scoring: a large stream of records, a cheap filter, a costlier score, and a ranked queue for a person to act on.

The digest has three sections: new today, still open, and closed since the last run. Postings are grouped by employer, employers are ordered by their strongest match, and each job carries its overall rank.

How a run works

  1. Sources
  2. Fetch
  3. Dedupe
  4. Keyword prefilter
  5. Fit scoring
  6. Email digest

Working with messy data

  • Adapters for Greenhouse, Lever, Ashby, Workable, Recruitee, Workday and Oracle Recruiting Cloud, plus generic RSS, HTML and JSON sources configured in YAML rather than code.
  • Large boards cannot be pulled whole. One carries about 7,300 live requisitions, so each query runs as a server-side search sorted newest first, and only the newest results come back.
  • A posting is marked closed only after it is missing from three consecutive healthy runs, and a source that failed to fetch gets no vote. Boards routinely drop a listing from one page of results and bring it back.

Decisions and why

  • The keyword filter runs before the model so that only plausible postings cost anything. Bad titles, wrong cities, old postings and ones already seen are dropped for free.
  • State lives in two files that the run commits each day: everything ever scored, so nothing is paid for twice, and the board of open postings.
  • The resume and scoring instructions are sent as a cached prefix, so every posting after the first in a run is billed at about a tenth of the normal input rate.
  • GitHub's scheduler is UTC-only and ignores daylight saving. The workflow fires at two UTC hours and the job checks the New York clock, so exactly one run proceeds each day and the time never drifts.
  • The crawler identifies itself in its User-Agent, reads only public endpoints and does not log in anywhere. LinkedIn and Indeed are deliberately unsupported, because both prohibit scraping.