fix(web-scraping): certify Phase 4 integration

[rmusser01]

Implementation summary

• keep inferred/default regex extraction as non-terminal enrichment while preserving explicit regex compatibility
• preserve Phase 3 pre-scrape analyzer configuration through raw Web-Scraper section propagation without interpolation or secret-bearing error logs
• regenerate and review the import inventory, document canonical ownership and deferred scope, and finalize Phase 4 certification records

Verification

• latest-dev focused preservation matrix: 276 passed
• broad Web_Scraping matrix before the final no-overlap rebase: 2,362 passed, 4 skipped, 2 failed, 2 errors; all four non-passing cases reproduced on that exact dev base
• selected cross-consumer matrix before the final no-overlap rebase: 2,947 passed, 50 skipped, 1 xpassed; 18 DNS-blocked tiktoken failures reproduced on that exact dev base
• final rebase onto dev b1d0aed was conflict-free; intervening commits did not modify Phase 4D source, documentation, or test paths
• regenerated inventory changed only three shifted line numbers and passes byte-for-byte validation
• compileall and diff checks passed
• plan-scoped Black passed across 47 files
• Ruff and Bandit matched the certified exact base; no new Ruff findings and no new medium/high Bandit findings
• Python 3.10 is unavailable locally and remains a CI gate

Change summary

Tracking

• TASK-12989.4
• TASK-12989.5
• follows merged Phase 4C PR refactor(web-scraping): govern article orchestration #2792

---

Summary by cubic

Certifies the Phase 4 Web_Scraping integration and refreshes certification after the final dev rebase. Default/inferred regex is now non-terminal enrichment: previously it could mark a scrape successful; now it only fills regex_matches and the pipeline continues, while explicit regex still terminates when selected.

• Bug Fixes

  • Isolate config section values via explicit section maps or raw=True reads to avoid interpolation and preserve pre-scrape analyzer settings.
  • Keep regex_matches in results even when later strategies fail.
  • Log only exception types when reading config sections to avoid leaking secrets in debug logs.

• Migration

  • If you relied on implicit regex success, put regex first in strategy_order or consume regex_matches from an unsuccessful result.

^{Written for commit b05d63d. Summary will update on new commits.}

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[rmusser01]

/review

[qodo-code-review]

PR Reviewer Guide 🔍

Warning

/review is deprecated. Use /agentic_review instead (removal date not yet scheduled).

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ No major issues detected

[rmusser01]

/agentic_review

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Config defaults leak into web_scraper ✓ Resolved 🐞 Bug ≡ Correctness

Description

load_and_log_configs() now merges ConfigParser.items('Web-Scraper', raw=True) into the
web_scraper mapping, which (per ConfigParser behavior) includes DEFAULT/top-of-file options; this
unintentionally injects unrelated settings into web_scraper. This can cause surprising key
collisions and increases the risk of exposing unrelated or sensitive default keys wherever
web_scraper is consumed.

Code

tldw_Server_API/app/core/config.py[R5591-5593]

            'web_scraper':{
+                **_section_items_dict('Web-Scraper'),
                'web_scraper_api_key': web_scraper_api_key,

Evidence

The PR now expands all [Web-Scraper] items directly into web_scraper. Since config.txt has
key/value lines before any [Section] header, those are DEFAULT options in ConfigParser and will be
included by .items('Web-Scraper', ...), unintentionally polluting the web_scraper config
mapping.

tldw_Server_API/app/core/config.py[4831-4837]
tldw_Server_API/app/core/config.py[5591-5618]
tldw_Server_API/Config_Files/config.txt[2-11]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`load_and_log_configs()` now seeds the `web_scraper` dict with `dict(config_parser_object.items('Web-Scraper', raw=True))`. In `configparser`, `items(section)` includes options from the DEFAULT section (including any key/value pairs placed before the first `[Section]` header in `config.txt`). This means the returned `web_scraper` mapping can silently accumulate unrelated DEFAULT keys.

### Issue Context
This repo’s `tldw_Server_API/Config_Files/config.txt` contains global (DEFAULT) keys before any section headers, so they will be pulled into `web_scraper` after this change.

### Fix Focus Areas
- tldw_Server_API/app/core/config.py[4831-4837]
- tldw_Server_API/app/core/config.py[5591-5618]

### Suggested fix
Adjust `_section_items_dict()` so it returns only keys explicitly defined in the named section, excluding DEFAULT keys.

Two safe approaches:
1) Use the parser’s internal section store (preferred for correctness here):
```py
section = getattr(config_parser_object, "_sections", {}).get(section_name)
if isinstance(section, dict):
   items = dict(section)
   items.pop("__name__", None)
   return items
```
2) Or compute `items()` and then drop DEFAULT-only keys **without** dropping keys that are explicitly present in the section (requires checking the section’s explicit keys via `_sections[section_name]` or equivalent).

Keep `raw=True` semantics for the values you *do* return.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 74 rules

Review mode: ⏭️ Skipped: The latest push only updates task/backlog markdown metadata and checklist evidence, with no runtime, configuration, test, or build behavior changes.

Tip of the day

💡 Did you know, you can copy the agent prompt from any finding and feed it to your IDE agent

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit b05d63d ⏭️ Skipped

Results up to commit c70544f ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Config defaults leak into web_scraper ✓ Resolved 🐞 Bug ≡ Correctness

Description

load_and_log_configs() now merges ConfigParser.items('Web-Scraper', raw=True) into the
web_scraper mapping, which (per ConfigParser behavior) includes DEFAULT/top-of-file options; this
unintentionally injects unrelated settings into web_scraper. This can cause surprising key
collisions and increases the risk of exposing unrelated or sensitive default keys wherever
web_scraper is consumed.

Code

tldw_Server_API/app/core/config.py[R5591-5593]

            'web_scraper':{
+                **_section_items_dict('Web-Scraper'),
                'web_scraper_api_key': web_scraper_api_key,

Evidence

The PR now expands all [Web-Scraper] items directly into web_scraper. Since config.txt has
key/value lines before any [Section] header, those are DEFAULT options in ConfigParser and will be
included by .items('Web-Scraper', ...), unintentionally polluting the web_scraper config
mapping.

tldw_Server_API/app/core/config.py[4831-4837]
tldw_Server_API/app/core/config.py[5591-5618]
tldw_Server_API/Config_Files/config.txt[2-11]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`load_and_log_configs()` now seeds the `web_scraper` dict with `dict(config_parser_object.items('Web-Scraper', raw=True))`. In `configparser`, `items(section)` includes options from the DEFAULT section (including any key/value pairs placed before the first `[Section]` header in `config.txt`). This means the returned `web_scraper` mapping can silently accumulate unrelated DEFAULT keys.

### Issue Context
This repo’s `tldw_Server_API/Config_Files/config.txt` contains global (DEFAULT) keys before any section headers, so they will be pulled into `web_scraper` after this change.

### Fix Focus Areas
- tldw_Server_API/app/core/config.py[4831-4837]
- tldw_Server_API/app/core/config.py[5591-5618]

### Suggested fix
Adjust `_section_items_dict()` so it returns only keys explicitly defined in the named section, excluding DEFAULT keys.

Two safe approaches:
1) Use the parser’s internal section store (preferred for correctness here):
```py
section = getattr(config_parser_object, "_sections", {}).get(section_name)
if isinstance(section, dict):
   items = dict(section)
   items.pop("__name__", None)
   return items
```
2) Or compute `items()` and then drop DEFAULT-only keys **without** dropping keys that are explicitly present in the section (requires checking the section’s explicit keys via `_sections[section_name]` or equivalent).

Keep `raw=True` semantics for the values you *do* return.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit bb71712 ⚖️ Balanced

No changes from previous review

Results up to commit 56e90b6 ⚖️ Balanced

No changes from previous review

[rmusser01]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit bb71712

[rmusser01]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit 56e90b6

[rmusser01]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit b05d63d