Clean contact, directory and company-name data from discovery

[edmiidz]

I found three data-quality bugs during a first real harvey discover --provider serper run against a local-services vertical. Each one would have ended up in an outgoing email.

1. Team-page parser records marketing copy as people

detect_people produced contacts like:

Recorded contact	Source markup
HOA Boards — partner dedicated to accoun	<li><strong>HOA Boards</strong> — Desiring consistent oversight and a trusted management partner…</li>
Customer care — Responsive Customer Serv	<strong class="main-nav-link-heading">Customer care</strong>
<real name> — CEO <div class="descriptio	<h3>Name</h3><p>CEO</p><p class="description">long bio…

These have three causes:

• A blanket (?is) flag made the [A-Z][a-z]+ [A-Z]… name pattern case-insensitive, so lowercase headings passed. Now only the tag names are case-insensitive ((?i:h[1-5]|strong|b)).
• The title was searched anywhere in the 220-char window. Any sentence containing "partner" or "service" became a title. Now the title is the first block-level text chunk after the name (<p>, <br>, <div>, <li>, …), and it has to read like a title: at most 10 words, mostly capitalised, with a role word in it.
• The 220-char window could end mid-tag. Tag stripping then left <div class="descriptio in the text. That dangling tag is now dropped.

Name tokens also reject vertical nouns ("HOA", "Boards", "Community"), abstract nouns (-tion/-sion/-ness) and all-caps acronyms, checking each part of a hyphenated token ("Process-Driven"). DECISION_MAKER_TITLES gains chief and director. Before, "Chief Executive Officer" written out in full wasn't flagged as a decision-maker.

2. Directory filter misses "… Company List" pages

A page titled " HOA Management Company List" on a clean-looking domain got through. JUNK_NAME_PATTERNS now also matches (company|companies|firms|contractors|businesses|providers) list and list of.

3. SERP providers store a page title as the company name

The Serper and DataForSEO SERP adapters store item["title"] as Business.name. Those are page titles: "Denver, CO | HOA Management Firm", "Home Page - NVCM | Community Association Management". The writer would then say "…at Denver, CO | HOA Management Firm".

• business_name(title, domain) splits the title on |, -, –, —, : and ·, and keeps the segment that shares a distinctive word with the domain. Generic words (management, HOA, company, city/state names, …) don't count, so "HOA Management Firm" does not match managementharbor.com. It also accepts an initialism ("Northvale Association of Community Managers" ↔ nacm.org). If nothing matches, it derives a name from the domain: managementharbor.com → "Management Harbor", kqresidential.com → "KQ Residential".
• The profile stage already fetches each homepage. It now reads og:site_name and <title> (taking the fullest matching segment) and upgrades a name the domain doesn't back up, or one that was only domain-derived. A name that already matches is left alone.
• StateManager.rename_company() updates companies.name and the denormalised prospects.company together, so emails never use a stale name.

The listings adapter is unchanged, because its title is a real business name.

Testing

• 13 new tests across test_collectors_profile.py and test_discover.py, built from the real markup patterns above. All fixtures are fictional businesses and people.
• Full suite: 544 passed, 22 skipped. The one local failure, test_load_env_reads_environment, happens only when a real .env exists. It's fixed separately in Keep a developer's real .env out of test_load_env_reads_environment #22.
• I checked the parser against 4 real association-management team pages: before, it recorded 6 junk contacts and 2 titles containing HTML; after, there are none, and all 11 real people come out with clean titles.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B4tcpnvC8VAkRoRkM7JWNF