Grant executor capabilities in upgrade-database.yml; fall back when the eBPF counter load fails

CHANGELOG.md

@@ -90,6 +90,27 @@ changes; the linked API and deployment documentation contains operational detail
   databases require the explicit upgrade to schema 26. See
   `docs/operations/configuration.md#destination-limits-and-opt-outs`.
 
+### Fixed
+- `deploy/ansible/upgrade-database.yml` gives the candidate executor binary
+  exactly `executor_capabilities` before it activates and starts it (#414).
+  It used to start the executor without CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN
+  and CAP_NET_RAW until a full deployment followed. A host that refuses the
+  capabilities now stops the upgrade before the service or database is
+  touched. The executor role, `rollout-executors.yml` and
+  `upgrade-database.yml` now grant them through one task file
+  (`deploy/ansible/tasks/executor-capabilities.yml`). `rollout-executors.yml`
+  now also grants them when `executor_ambient_caps` is set, as the role and
+  `verify.yml` already expected, and skips `setcap` when the binary already
+  carries the set.
+- With `packet_counter = "auto"`, an executor whose eBPF load fails falls
+  back to the userspace counter instead of exiting with `packet counter
+  cleanup unconfirmed` and being restarted in a loop (#414). A load attaches
+  nothing, so any load failure is a clean rollback. A failed attach whose
+  release cannot be confirmed still stops the executor. When the process
+  lacks CAP_BPF or CAP_NET_ADMIN, the logged error names the missing
+  capability rather than cilium/ebpf's "MEMLOCK may be too low" or "prealloc
+  maps not supported" hint, and the fallback reason is `not_permitted`.
+
 ## [0.3.0-rc.1] - 2026-10-06
 
 ### Security

deploy/README.md

@@ -433,9 +433,11 @@ executor's identity.
 
 ### Executor capabilities and vantage-point metadata
 
-The executor role gives the installed executor binary exactly the file
-capabilities in `executor_capabilities`
-([`group_vars/executors.yml`](ansible/group_vars/executors.yml)):
+The executor role, `rollout-executors.yml` and `upgrade-database.yml` give
+the installed executor binary exactly the file capabilities in
+`executor_capabilities`
+([`group_vars/executors.yml`](ansible/group_vars/executors.yml)), all through
+[`ansible/tasks/executor-capabilities.yml`](ansible/tasks/executor-capabilities.yml):
 
 | Capability | Needed for |
 | --- | --- |
@@ -449,7 +451,9 @@ Without the first three an executor reports `enforcement_mode: fallback`,
 without `cap_net_raw` also `tagging.ipv4: none` and no ICMP. `cap_sys_resource`
 is not granted: from Linux 5.11 BPF memory is charged to the memory cgroup,
 and on an older kernel the unit sets `LimitMEMLOCK=infinity` instead. A host
-that refuses file capabilities keeps deploying with a warning;
+that refuses file capabilities keeps deploying with a warning (a rollout or
+database upgrade, which starts that binary in place of a running executor,
+stops instead);
 [`ansible/verify.yml`](ansible/verify.yml) fails on any executor whose binary
 lacks the set. Set `executor_enable_bpf: false` for such a host.
 

deploy/ansible/roles/executor/tasks/main.yml

@@ -88,55 +88,20 @@
   notify: Restart executor
 
 # Without these the eBPF tagger cannot load and probes leave untagged, so this
-# is what makes packet attribution verifiable; executor_capabilities in
-# group_vars/executors.yml says what each one is for. The binary gets exactly
-# that set: setcap replaces whatever the file carried before. Some hosts
-# (restricted containers, noexec/nosuid or non-xattr filesystems) reject file
-# capabilities outright; that costs those nodes their attribution tags but
-# must not fail the whole deploy, so the failure is reported rather than
-# fatal, and verify.yml reports the missing capabilities. Set
-# executor_enable_bpf=false on such a host to skip the attempt entirely.
+# is what makes packet attribution verifiable. Some hosts (restricted
+# containers, noexec/nosuid or non-xattr filesystems) reject file capabilities
+# outright; that costs those nodes their attribution tags but must not fail the
+# whole deploy, so the failure is reported rather than fatal, and verify.yml
+# reports the missing capabilities. Set executor_enable_bpf=false on such a
+# host to skip the attempt entirely. The same task file grants them in
+# rollout-executors.yml and upgrade-database.yml.
 - name: Set the executor's capabilities on its binary
-  when: executor_enable_bpf | default(true) | bool
+  ansible.builtin.import_tasks: "{{ role_path }}/../../tasks/executor-capabilities.yml"
   vars:
-    # The installed payload itself, not the link next to it: file
-    # capabilities live on the executable.
-    executor_release_binary: "{{ payload_prefix }}/lib/debuglet/{{ debuglet_release.version }}/bin/debuglet-executor"
-  block:
-    - name: Read the executor binary's file capabilities
-      ansible.builtin.command:
-        argv: [getcap, "{{ executor_release_binary }}"]
-      register: executor_getcap
-      changed_when: false
-      failed_when: false
-      check_mode: false
-
-    # getcap prints "PATH cap_a,cap_b=ep" for a file that carries them.
-    - name: Grant exactly the executor's capabilities
-      ansible.builtin.command:
-        argv:
-          - setcap
-          - "{{ executor_capabilities | join(',') }}=ep"
-          - "{{ executor_release_binary }}"
-      register: executor_setcap
-      failed_when: false
-      changed_when: executor_setcap.rc == 0
-      # A running executor keeps the capabilities it started with.
-      notify: Restart executor
-      when: >-
-        ((executor_getcap.stdout | default('') | regex_search('\\s(\\S+)=ep\\s*$', '\\1') or ['']) | first).split(',') | sort
-        != executor_capabilities | sort
-
-    - name: Warn when the capabilities could not be set
-      ansible.builtin.debug:
-        msg: >-
-          Could not set {{ executor_capabilities | join(',') }} on
-          {{ executor_release_binary }}
-          ({{ executor_setcap.stderr | default('') | trim | default('setcap failed', true) }}).
-          This node will run without the eBPF egress tagger, so its probes
-          will carry no attribution tag and cannot be verified, and without
-          cap_net_raw it offers no ICMP.
-      when: executor_setcap.rc | default(0) != 0
+    executor_capabilities_binary: "{{ payload_prefix }}/lib/debuglet/{{ debuglet_release.version }}/bin/debuglet-executor"
+    executor_capabilities_required: false
+  # A running executor keeps the capabilities it started with.
+  notify: Restart executor
 
 - name: Install executor systemd unit
   ansible.builtin.template:

deploy/ansible/rollout-executors.yml

@@ -63,15 +63,14 @@
       ansible.builtin.include_role:
         name: payload
 
-    # Exactly executor_capabilities (group_vars/executors.yml); setcap
-    # replaces whatever the staged file carried.
+    # Exactly executor_capabilities (group_vars/executors.yml), shared with
+    # the executor role and upgrade-database.yml. A refusal stops the rollout
+    # here, before the running executor is drained.
     - name: Grant the executor's required kernel capabilities
-      ansible.builtin.command:
-        argv:
-          - setcap
-          - "{{ executor_capabilities | join(',') }}=ep"
-          - "{{ payload_prefix }}/lib/debuglet/{{ deploy_version }}/bin/debuglet-executor"
-      when: executor_enable_bpf | default(true) | bool and not (executor_ambient_caps | default(false) | bool)
+      ansible.builtin.import_tasks: tasks/executor-capabilities.yml
+      vars:
+        executor_capabilities_binary: "{{ payload_prefix }}/lib/debuglet/{{ deploy_version }}/bin/debuglet-executor"
+        executor_capabilities_required: true
 
     - name: Refuse a release incompatible with the current schema
       ansible.builtin.command:

deploy/ansible/tasks/executor-capabilities.yml

@@ -0,0 +1,70 @@
+---
+# Give one installed executor binary exactly executor_capabilities
+# (group_vars/executors.yml says what each one is for). This is the only place
+# the capabilities are granted: the executor role, rollout-executors.yml and
+# upgrade-database.yml all import it, so an executor started by any of them
+# runs with the same set. Without it the eBPF tagger and packet counter cannot
+# load and probes leave untagged.
+#
+# setcap replaces whatever the file carried before. It runs only when getcap
+# reports a different set, so a repeat run changes nothing, and in check mode
+# getcap still runs while setcap does not. Nothing is granted when
+# executor_enable_bpf is false. The unit's ambient set
+# (executor_ambient_caps) does not replace the file capabilities: it is the
+# same list, and verify.yml requires them on the binary either way.
+#
+# Inputs:
+#   executor_capabilities_binary    the installed executable itself, not the
+#                                   link to it: file capabilities live on the
+#                                   executable
+#   executor_capabilities_required  true: a refused setcap fails the host, for
+#                                   playbooks that are about to start that
+#                                   binary in place of a running executor.
+#                                   false (default): it is reported as a
+#                                   warning, for hosts (restricted containers,
+#                                   noexec/nosuid or non-xattr filesystems)
+#                                   that reject file capabilities outright
+#
+# Registers executor_setcap, which is changed when the capabilities were
+# replaced: a running executor keeps the capabilities it started with, so the
+# caller restarts it.
+
+- name: Read the executor binary's file capabilities
+  ansible.builtin.command:
+    argv: [getcap, "{{ executor_capabilities_binary }}"]
+  register: executor_getcap
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: executor_enable_bpf | default(true) | bool
+
+# getcap prints "PATH cap_a,cap_b=ep" for a file that carries them.
+- name: Grant exactly the executor's capabilities
+  ansible.builtin.command:
+    argv:
+      - setcap
+      - "{{ executor_capabilities | join(',') }}=ep"
+      - "{{ executor_capabilities_binary }}"
+  register: executor_setcap
+  failed_when: >-
+    executor_setcap.rc | default(0) != 0
+    and executor_capabilities_required | default(false) | bool
+  changed_when: executor_setcap.rc | default(1) == 0
+  when:
+    - executor_enable_bpf | default(true) | bool
+    - >-
+      ((executor_getcap.stdout | default('') | regex_search('\\s(\\S+)=ep\\s*$', '\\1') or ['']) | first).split(',') | sort
+      != executor_capabilities | sort
+
+- name: Warn when the capabilities could not be set
+  ansible.builtin.debug:
+    msg: >-
+      Could not set {{ executor_capabilities | join(',') }} on
+      {{ executor_capabilities_binary }}
+      ({{ executor_setcap.stderr | default('') | trim | default('setcap failed', true) }}).
+      This node will run without the eBPF egress tagger, so its probes
+      will carry no attribution tag and cannot be verified, and without
+      cap_net_raw it offers no ICMP.
+  when:
+    - executor_enable_bpf | default(true) | bool
+    - executor_setcap.rc | default(0) != 0

deploy/ansible/upgrade-database.yml

@@ -19,7 +19,10 @@
 #      upgrade_accept_data_loss=true, naming the data concerned;
 #   6. requires twice the size of the database and its -wal and -shm companions
 #      to be free on the database's filesystem (one copy for the backup, one for
-#      the migration's own journal);
+#      the migration's own journal); on an executor it also gives the candidate
+#      binary exactly executor_capabilities (tasks/executor-capabilities.yml,
+#      the same task file the executor role uses), so the executor it starts in
+#      step 11 can load its eBPF programs;
 #   7. stops the service; 8. copies the database and its companions into a new
 #      backup-TIMESTAMP directory next to it; 9. runs the candidate daemon's
 #      -upgrade-database mode as the service user through runuser, so the
@@ -313,6 +316,16 @@
               and the service were not touched.
             quiet: true
 
+        # The candidate is started below in place of the running executor,
+        # so it needs exactly the capabilities the executor role grants;
+        # without them its eBPF packet counter and tagger cannot load. A
+        # refusal stops here, before the service or the database is touched.
+        - name: Grant the candidate executor its kernel capabilities
+          ansible.builtin.import_tasks: tasks/executor-capabilities.yml
+          vars:
+            executor_capabilities_binary: "{{ payload_prefix }}/lib/debuglet/{{ debuglet_release.version }}/bin/debuglet-executor"
+            executor_capabilities_required: true
+
         - name: Name the backup directory
           ansible.builtin.set_fact:
             upgrade_backup_dir: "{{ upgrade_database | dirname }}/backup-{{ now(utc=true, fmt='%Y%m%dT%H%M%SZ') }}"