Query file in, outreach list out

The pipeline is two commands. The first drives a headless browser over a list of search queries and writes a 36-column CSV; the second merges, deduplicates and filters those CSVs into four outreach lists and one rejection log. Everything below is what had to be learned to make those two commands produce something worth sending to.

Stage one is the scrape. The input is a plain text file of Google Maps search queries — one per line, of the form "cable assembly in Phoenix AZ" — crossed across the US metros worth covering. Twenty-four queries per vertical turned out to be the point of diminishing returns: past that, the same businesses start reappearing under different search terms and the deduplication step absorbs them.

Concurrency is capped at two, and this is the single most expensive thing to get wrong. Google throttles aggressively and silently. Measured on identical query sets: at four concurrent workers the run produced seven HTTP 429 responses and 47 failed jobs across ten queries. At two, there were fewer, and the scraper’s retry logic absorbed them. Raising concurrency does not make the run faster — it makes the run shorter by dropping results, which is a different thing and much harder to notice, because the output CSV looks perfectly healthy. It is just missing rows nobody counted. The only real fix is a rotating proxy pool, which costs money, and the throughput was never the constraint.

Stage two of the scrape is the part that justifies the whole exercise: an email crawl. For every business with a website, the scraper fetches that site and extracts email addresses from it. This is most of the runtime — far slower than the Maps traversal itself — and it is the reason the dataset has 251 verified addresses in it. No official API offers this. Google’s Places API returns a phone number and a website URL and stops there, so a pipeline built on it produces a call list, not an email list.

Stage three is the filter, a single Python script. It merges the raw CSVs per vertical, deduplicates on place_id — Google’s own stable identifier, which survives a business renaming itself or moving — and falls back to a normalised name-plus-address key for the rows where place_id is missing. Then it applies the category rules, validates the email column, and splits the survivors four ways: makers with an email, makers with only a phone or website, once per vertical. Rejections are not discarded. They are written to their own CSV with the reason attached, because a filter you cannot audit is a filter you will eventually stop trusting.

The whole run is idempotent and cheap to repeat. Raw CSVs are kept, so re-running the filter after changing a category rule takes seconds and does not touch Google at all. That matters more than it sounds: it means tuning the allowlist is a fast loop rather than an overnight one, and a fast loop is why the allowlist is any good.