20260930 - Derive the RSPduo notch filters from the capture frequency

README.md

@@ -87,7 +87,52 @@ Config-merger generates config files at startup by merging:
 default.yml → user.yml → forced.yml
 ```
 
-Later files override earlier ones.
+Later files override earlier ones. A few values are then derived from the
+merged result rather than taken from any file. The notch filters below are one.
+
+### RF notch filters
+
+The RSPduo has two hardware notch filters, set by `capture.device.rfNotch` and
+`capture.device.dabNotch`. Config-merger derives both from the final
+`capture.fc` and `capture.fs` on every merge, after `forced.yml` is applied, so
+any value in `default.yml`, `user.yml` or `forced.yml` is overwritten.
+
+| Key | Stopband (SDRplay spec) | Rejection |
+| --- | --- | --- |
+| `rfNotch` (FM) | 77-115 MHz | 30 dB |
+| `dabNotch` | 155-235 MHz | at least 20 dB |
+
+A notch is on unless the capture band `[fc - fs/2, fc + fs/2]` overlaps its
+stopband (touching an edge counts as overlap). With the notch on, strong
+broadcast signals in that band are kept out of the ADC. With the capture band
+inside the stopband, the notch would reject the very signal being received, so
+it is turned off. At the shipped 503 MHz both notches are on.
+
+Why the derived value always wins: on first boot config-merger seeds `user.yml`
+with a full copy of `default.yml`, so every node already persists
+`rfNotch: true` and `dabNotch: true`. Overwriting on every merge means those
+copies need no migration, and a node moved to an FM or DAB illuminator gets the
+right notches with no hand edit. To change a notch, change `fc`.
+
+A node whose `fc` sits inside a stopband has that notch turned off by this rule,
+so its illuminator arrives 20-30 dB stronger than before. Gain set while the
+notch was on may now overload the front end: lower it, or run Auto-Calibrate
+again, after the change lands.
+
+If `fc` or `fs` is missing or not a number, the merged values are left as they
+are (`default.yml` ships both keys as `true`, and blah2 aborts on a missing
+key). blah2 cannot tune without a valid `fc` in any case.
+
+**Known limitation: live retune.** blah2 applies the notches only at
+`sdrplay_api_Init()`. blah2-api's `POST /capture/retune` changes `fc` without
+restarting blah2 and never touches the notches. retina-gui's Auto-Calibrate uses
+it to probe candidate towers, so a candidate in a different notch band from the
+starting `fc` is probed with the wrong notches (for example an FM tower probed
+with the FM notch still on, which costs it up to 30 dB). The persisted result
+is correct, because `/calibrate/apply` runs config-merger and restarts blah2.
+Standalone and staging deployments do not run config-merger: see
+[STANDALONE.md](STANDALONE.md#rf-notch-filters) and
+[staging/README.md](staging/README.md).
 
 ### Editing Config on a Node
 

STANDALONE.md

@@ -142,6 +142,20 @@ For development, uncomment and edit the image versions in `.env` to quickly swit
 
 Edit `config/config.yml` to set your location, frequency, and SDR settings. See [blah2-arm documentation](https://github.com/offworldlabs/blah2-arm) for full config options.
 
+### RF notch filters
+
+Standalone skips config-merger (`SKIP_CONFIG_MERGER=true`), so nothing derives
+`capture.device.rfNotch` and `capture.device.dabNotch` from your frequency. The
+copy of `default.yml` ships both as `true`, which suits UHF. If you change
+`capture.fc` so the capture band `[fc - fs/2, fc + fs/2]` overlaps a stopband,
+set that notch to `false` yourself, or it rejects the signal you are receiving:
+
+- `rfNotch`: 77-115 MHz (FM band)
+- `dabNotch`: 155-235 MHz (DAB band)
+
+Keep both keys present: blah2 aborts if either is missing. See the
+[README](README.md#rf-notch-filters) for the full rule.
+
 ## Commands
 
 From the install location (`/opt/retina-standalone`):

config-merger/script/merge_config.py

@@ -8,6 +8,8 @@
 Generates tar1090.env for tar1090-node if the tar1090 section exists in config.
 Generates retina-tracker.yaml for the retina-tracker sidecar if the
 retina_tracker section exists in config.
+Derives the RSPduo DAB and RF notch filters from the final capture.fc and
+capture.fs, overwriting whatever the merged files held (see derive_notches()).
 
 Takes as inputs:
 1. Defaults directory - holds baked-in default.yml and forced.yml
@@ -51,6 +53,14 @@
 # first-boot copy rather than anyone's choice. See migrate_adsb_truth_server().
 LEGACY_ADSB_TAR1090 = 'sfo1.retnode.com'
 
+# RSPduo notch filter stopbands in Hz, from the SDRplay spec. The RF (FM) notch
+# rejects 30 dB over 77-115 MHz, the DAB notch at least 20 dB over 155-235 MHz.
+# See derive_notches().
+RF_NOTCH_MIN_HZ = 77_000_000
+RF_NOTCH_MAX_HZ = 115_000_000
+DAB_NOTCH_MIN_HZ = 155_000_000
+DAB_NOTCH_MAX_HZ = 235_000_000
+
 
 def get_node_id_from_mender():
     """Read node_id from Mender device identity file (generated by mender-device-identity)"""
@@ -203,6 +213,42 @@ def migrate_gain_reduction(config):
         config['capture']['device']['gainReduction'] = [gain, gain]
 
 
+def derive_notches(config):
+    """Set capture.device.dabNotch and rfNotch from the final capture.fc and fs.
+
+    A notch is on unless the capture band [fc - fs/2, fc + fs/2] overlaps its
+    stopband (touching an edge counts). Runs on the merged config, so the result
+    overwrites default.yml, user.yml and forced.yml alike. That is why the
+    true/true every node persisted in its first-boot user.yml needs no LEGACY_*
+    migration. Rationale and the live-retune caveat: README.md, "RF notch filters".
+    """
+    try:
+        capture = config['capture']
+        fc = capture['fc']
+        fs = capture['fs']
+    except (KeyError, TypeError):
+        return
+
+    # With no usable fc or fs, leave the merged values: blah2 cannot tune anyway,
+    # and default.yml guarantees both keys exist.
+    if (not all(isinstance(v, (int, float)) and not isinstance(v, bool) for v in (fc, fs))
+            or fc <= 0 or fs < 0):
+        print(f"Cannot derive notch filters from capture.fc={fc!r}, fs={fs!r}; leaving them as merged")
+        return
+
+    device = capture.setdefault('device', {})
+    if not isinstance(device, dict):
+        return
+
+    low, high = fc - fs / 2, fc + fs / 2
+    rf = not (low <= RF_NOTCH_MAX_HZ and high >= RF_NOTCH_MIN_HZ)
+    dab = not (low <= DAB_NOTCH_MAX_HZ and high >= DAB_NOTCH_MIN_HZ)
+    device['rfNotch'] = rf
+    device['dabNotch'] = dab
+    print(f"Derived notch filters for {low / 1e6:g}-{high / 1e6:g} MHz: "
+          f"rfNotch={str(rf).lower()}, dabNotch={str(dab).lower()}")
+
+
 def migrate_doppler_span(user):
     """Drop a Doppler span the user never chose, so default.yml can change it.
 
@@ -463,6 +509,9 @@ def main():
         # Migrate legacy field formats forward (e.g. scalar -> per-tuner gainReduction)
         migrate_gain_reduction(config)
 
+        # Notches follow the final frequency, whatever any overlay said
+        derive_notches(config)
+
         # Write merged config to output
         print(f"Writing merged config to {output_config_path}")
         os.makedirs(os.path.dirname(output_config_path), exist_ok=True)

config-merger/test/test_merge_config.py

@@ -1048,6 +1048,88 @@ def test_upstream_chain_falls_back_to_adsblol_when_unset(self):
         self.assertIn('ADSB_UPSTREAMS=https://api.adsb.lol\n', env)
 
 
+    def merge_notches(self, capture, user_capture=None):
+        """Merge the given capture section and return the resulting device block"""
+        self.write_yaml(os.path.join(self.defaults_dir, 'default.yml'), {'capture': capture})
+        self.write_yaml(os.path.join(self.defaults_dir, 'forced.yml'), {})
+        user_yml = os.path.join(self.config_dir, 'user.yml')
+        if user_capture is not None:
+            self.write_yaml(user_yml, {'capture': user_capture})
+        elif os.path.exists(user_yml):
+            os.remove(user_yml)  # else the first-boot copy of a previous call wins
+        output = self.run_merge()
+        return self.read_yaml(output)['capture']['device']
+
+    def test_notches_on_for_uhf(self):
+        """The shipped 503 MHz leaves both notches on, as they were before derivation"""
+        with open(os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))),
+                               'config', 'default.yml')) as f:
+            capture = yaml.safe_load(f)['capture']
+        device = self.merge_notches(capture)
+        self.assertEqual(capture['fc'], 503000000)
+        self.assertIs(device['rfNotch'], True)
+        self.assertIs(device['dabNotch'], True)
+
+    def test_rf_notch_off_in_fm_band(self):
+        device = self.merge_notches({'fc': 98_000_000, 'fs': 2_000_000,
+                                     'device': {'rfNotch': True, 'dabNotch': True}})
+        self.assertIs(device['rfNotch'], False)
+        self.assertIs(device['dabNotch'], True)
+
+    def test_dab_notch_off_in_dab_band(self):
+        device = self.merge_notches({'fc': 204_640_000, 'fs': 2_000_000,
+                                     'device': {'rfNotch': True, 'dabNotch': True}})
+        self.assertIs(device['rfNotch'], True)
+        self.assertIs(device['dabNotch'], False)
+
+    def test_notch_band_edges_use_fs(self):
+        """A capture band that reaches into a stopband turns that notch off"""
+        # 116 MHz +/- 1 MHz touches the RF stopband's 115 MHz top edge
+        device = self.merge_notches({'fc': 116_000_000, 'fs': 2_000_000, 'device': {}})
+        self.assertIs(device['rfNotch'], False)
+        # the same fc with a narrower fs clears it
+        device = self.merge_notches({'fc': 116_000_000, 'fs': 1_000_000, 'device': {}})
+        self.assertIs(device['rfNotch'], True)
+        # 154 MHz +/- 1 MHz reaches the DAB stopband's 155 MHz bottom edge
+        device = self.merge_notches({'fc': 154_000_000, 'fs': 2_000_000, 'device': {}})
+        self.assertIs(device['dabNotch'], False)
+        self.assertIs(device['rfNotch'], True)
+        device = self.merge_notches({'fc': 154_000_000, 'fs': 1_000_000, 'device': {}})
+        self.assertIs(device['dabNotch'], True)
+
+    def test_notches_overwrite_user_yml(self):
+        """The persisted first-boot true/true and any hand-set value lose to fc"""
+        device = self.merge_notches(
+            {'fc': 503_000_000, 'fs': 2_000_000, 'device': {'rfNotch': True, 'dabNotch': True}},
+            user_capture={'fc': 98_000_000, 'device': {'rfNotch': True, 'dabNotch': False}})
+        self.assertIs(device['rfNotch'], False)
+        self.assertIs(device['dabNotch'], True)
+
+    def test_notches_follow_forced_fc(self):
+        """Derivation runs after forced.yml, so a forced fc decides"""
+        self.write_yaml(os.path.join(self.defaults_dir, 'default.yml'),
+                        {'capture': {'fc': 503_000_000, 'fs': 2_000_000, 'device': {}}})
+        self.write_yaml(os.path.join(self.defaults_dir, 'forced.yml'), {'capture': {'fc': 204_640_000}})
+        device = self.read_yaml(self.run_merge())['capture']['device']
+        self.assertIs(device['dabNotch'], False)
+        self.assertIs(device['rfNotch'], True)
+
+    def test_notches_left_alone_without_fc(self):
+        """No usable fc: the merged values stand and the merge still succeeds"""
+        device = self.merge_notches({'fs': 2_000_000, 'device': {'rfNotch': True, 'dabNotch': False}})
+        self.assertIs(device['rfNotch'], True)
+        self.assertIs(device['dabNotch'], False)
+
+        device = self.merge_notches({'fc': 'abc', 'fs': 2_000_000,
+                                     'device': {'rfNotch': False, 'dabNotch': True}})
+        self.assertIs(device['rfNotch'], False)
+        self.assertIs(device['dabNotch'], True)
+
+    def test_no_capture_section_adds_no_notches(self):
+        self.write_yaml(os.path.join(self.defaults_dir, 'default.yml'), {'network': {'ip': '0.0.0.0'}})
+        self.assertNotIn('capture', self.read_yaml(self.run_merge()))
+
+
 class TestAtomicWrites(unittest.TestCase):
     """The compose .env must be replaced whole, never rewritten in place.
 

config/default.yml

@@ -7,6 +7,9 @@ capture:
     bandwidthNumber: 0
     gainReduction: [40, 40] #20-59 dB each, [reference, surveillance], higher is less gain
     lnaState: 4 #1-9, higher is more gain
+    # Derived by config-merger from fc and fs on every merge, overwriting any
+    # value set here or in user.yml. Kept so config.yml always has both keys
+    # (blah2 aborts without them). See README.md, "RF notch filters".
     dabNotch: true
     rfNotch: true
   replay:

staging/README.md

@@ -123,6 +123,12 @@ This creates `config/config.yml` by merging:
 
 **Optional**: Edit `config/user.yml` to override specific settings.
 
+`merge-config.sh` is a plain merge, not config-merger, so it does not derive the
+RF and DAB notch filters from `capture.fc`. The `true`/`true` from `default.yml`
+passes through, which is right for UHF. If you override `fc` into 77-115 MHz or
+155-235 MHz, set `rfNotch` or `dabNotch` to `false` in `config/user.yml` as well.
+See the main [README](../README.md#rf-notch-filters).
+
 ### Step 4: Start Services
 
 ```bash

staging/merge-config.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 # Manually merge config files for staging
 # Combines: default.yml → user.yml → forced.yml = config.yml
+# Unlike config-merger, this does not derive the RF/DAB notch filters from fc.
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"