Skip to content

Commit 1c73817

Browse files
the2pizzathe2pizza
authored
Log parser (#24)
* feat: pixel-agent web analytics pipeline Add a standalone pixel-agent daemon that parses nginx access logs for /pixel requests, aggregates visits by page/country/referrer/UTM, and sends them to the API. API changes: - New WebMetricStorage for in-memory web analytics with snapshot/reload - POST /metrics/web/ingest (service token) for daemon ingestion - GET /admin/api/web/metrics and /timeline (admin token) for dashboards - GC and snapshot tasks for WebMetricStorage Admin panel: - New Web Analytics section with total visits, top pages/countries/referrers - Timeline chart for visits Other: - Updated analytics.js to collect lang and UTM parameters - Example configs for pixel-agent and API web metrics - Prometheus endpoint on pixel-agent (:9102/metrics) * feat(pixel-agent): local storage, embedded admin, nom parser - Rewrite pixel log parser from regex to nom. - Move WebMetricStorage from API into pixel-agent. - Store web metrics locally with snapshots instead of pushing to API. - Add embedded admin UI and JSON/Prometheus endpoints to pixel-agent. - Remove web analytics ingestion from API admin panel. - Update pixel-agent config: snapshot_path, admin_listen, admin_port, retention. * chore(pixel-agent): add systemd unit and deploy script * ci: build and release pixel-agent, add install script and nginx example * feat(pixel-agent): track host in analytics.js and add top hosts chart * feat(pixel-agent): add top sources and referers sections to admin UI * feat(pixel-agent): add backfill binary and script for old gzipped logs
1 parent caffacd commit 1c73817

23 files changed

Lines changed: 2065 additions & 5 deletions

‎.github/workflows/release.yml‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,9 +91,43 @@ jobs:
9191
name: fcore-auth
9292
path: target/release/auth
9393

94+
build-pixel-agent:
95+
needs: [tests]
96+
runs-on: ubuntu-22.04
97+
steps:
98+
- name: Checkout
99+
uses: actions/checkout@v4
100+
- name: Install Rust
101+
uses: dtolnay/rust-toolchain@nightly
102+
- uses: Swatinem/rust-cache@v2
103+
- name: Build Pixel Agent
104+
run: cargo build --release --bin pixel-agent --no-default-features
105+
- name: Upload Artifact
106+
uses: actions/upload-artifact@v4
107+
with:
108+
name: fcore-pixel-agent
109+
path: target/release/pixel-agent
110+
111+
build-pixel-agent-backfill:
112+
needs: [tests]
113+
runs-on: ubuntu-22.04
114+
steps:
115+
- name: Checkout
116+
uses: actions/checkout@v4
117+
- name: Install Rust
118+
uses: dtolnay/rust-toolchain@nightly
119+
- uses: Swatinem/rust-cache@v2
120+
- name: Build Pixel Agent Backfill
121+
run: cargo build --release --bin pixel-agent-backfill --no-default-features
122+
- name: Upload Artifact
123+
uses: actions/upload-artifact@v4
124+
with:
125+
name: fcore-pixel-agent-backfill
126+
path: target/release/pixel-agent-backfill
127+
94128
release:
95129
name: Create Release
96-
needs: [build-api, build-auth, build-node]
130+
needs: [build-api, build-auth, build-node, build-pixel-agent, build-pixel-agent-backfill]
97131
runs-on: ubuntu-latest
98132
steps:
99133
- name: Checkout
@@ -113,6 +147,8 @@ jobs:
113147
cp artifacts/fcore-node-armv7-unknown-linux-gnueabihf/node release_dist/node-armv7
114148
cp artifacts/fcore-api/api release_dist/api-x86_64
115149
cp artifacts/fcore-auth/auth release_dist/auth-x86_64
150+
cp artifacts/fcore-pixel-agent/pixel-agent release_dist/pixel-agent-x86_64
151+
cp artifacts/fcore-pixel-agent-backfill/pixel-agent-backfill release_dist/pixel-agent-backfill-x86_64
116152
chmod +x release_dist/*
117153
118154
- name: Upload to GitHub Release
@@ -125,5 +161,6 @@ jobs:
125161
config-node-example.toml
126162
config-api-example.toml
127163
config-auth-example.toml
164+
config-pixel-agent-example.toml
128165
env:
129166
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/rust.yml‎

Lines changed: 75 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,9 +135,83 @@ jobs:
135135
name: fcore-auth
136136
path: target/x86_64-unknown-linux-musl/release/auth
137137

138+
build-pixel-agent:
139+
needs: [tests]
140+
runs-on: ubuntu-22.04
141+
142+
steps:
143+
- uses: actions/checkout@v4
144+
145+
- name: Install Rust
146+
uses: dtolnay/rust-toolchain@nightly
147+
with:
148+
targets: x86_64-unknown-linux-musl
149+
150+
- uses: Swatinem/rust-cache@v2
151+
with:
152+
key: pixel-agent
153+
154+
- name: Install cross
155+
run: cargo install cross
156+
157+
- name: Build Pixel Agent
158+
run: |
159+
cross build --release \
160+
--bin pixel-agent \
161+
--no-default-features \
162+
--target x86_64-unknown-linux-musl
163+
164+
- name: Verify binary
165+
run: |
166+
file target/x86_64-unknown-linux-musl/release/pixel-agent
167+
ldd target/x86_64-unknown-linux-musl/release/pixel-agent || true
168+
169+
- name: Upload Artifact
170+
uses: actions/upload-artifact@v4
171+
with:
172+
name: fcore-pixel-agent
173+
path: target/x86_64-unknown-linux-musl/release/pixel-agent
174+
175+
build-pixel-agent-backfill:
176+
needs: [tests]
177+
runs-on: ubuntu-22.04
178+
179+
steps:
180+
- uses: actions/checkout@v4
181+
182+
- name: Install Rust
183+
uses: dtolnay/rust-toolchain@nightly
184+
with:
185+
targets: x86_64-unknown-linux-musl
186+
187+
- uses: Swatinem/rust-cache@v2
188+
with:
189+
key: pixel-agent-backfill
190+
191+
- name: Install cross
192+
run: cargo install cross
193+
194+
- name: Build Pixel Agent Backfill
195+
run: |
196+
cross build --release \
197+
--bin pixel-agent-backfill \
198+
--no-default-features \
199+
--target x86_64-unknown-linux-musl
200+
201+
- name: Verify binary
202+
run: |
203+
file target/x86_64-unknown-linux-musl/release/pixel-agent-backfill
204+
ldd target/x86_64-unknown-linux-musl/release/pixel-agent-backfill || true
205+
206+
- name: Upload Artifact
207+
uses: actions/upload-artifact@v4
208+
with:
209+
name: fcore-pixel-agent-backfill
210+
path: target/x86_64-unknown-linux-musl/release/pixel-agent-backfill
211+
138212
collect-binaries:
139213
runs-on: ubuntu-latest
140-
needs: [build-node, build-api, build-auth]
214+
needs: [build-node, build-api, build-auth, build-pixel-agent, build-pixel-agent-backfill]
141215

142216
steps:
143217
- name: Download all artifacts

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
dev/config-agent.toml
55
dev/config-api.toml
66
dev/config-auth.toml
7+
dev/pixel-logs
78
target
89
snapshots/*.bin
910
dev/data/*

‎Cargo.lock‎

Lines changed: 23 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ netlink-packet-amnezia-wireguard = { version = "0.1.2", optional = true }
6161
netlink-packet-core = { version = "0.8", optional = true }
6262
netlink-packet-generic = { version = "0.4", optional = true }
6363
netlink-sys = { version = "0.8", optional = true }
64+
maxminddb = "0.24"
65+
nom = "7"
6466

6567

6668
[build-dependencies]
@@ -93,3 +95,11 @@ path = "src/bin/api/main.rs"
9395
name = "auth"
9496
path = "src/bin/auth/main.rs"
9597

98+
[[bin]]
99+
name = "pixel-agent"
100+
path = "src/bin/pixel_agent/main.rs"
101+
102+
[[bin]]
103+
name = "pixel-agent-backfill"
104+
path = "src/bin/pixel_agent/backfill.rs"
105+

‎config-api-example.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ admin_token = "change-me-in-production"
3232
reciever = "tcp://0.0.0.0:3002"
3333
max_points = 10000
3434
retention_seconds = 604800 # 7 days
35+
snapshot_path = "snapshots/metrics.bin"
3536

3637
[metrics.log]
3738
enabled = true

‎config-pixel-agent-example.toml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
log_level = "info"
2+
log_path = "/var/log/nginx/pixel.log"
3+
offset_path = "/var/lib/pixel-agent/offset.dat"
4+
snapshot_path = "/var/lib/pixel-agent/metrics.snapshot"
5+
geoip_db = "/usr/share/GeoIP/GeoLite2-Country.mmdb"
6+
admin_listen = "0.0.0.0"
7+
admin_port = 9102
8+
poll_interval_sec = 30
9+
flush_interval_sec = 300
10+
snapshot_interval_sec = 300
11+
bucket_minutes = 5
12+
retention_hours = 168
13+
max_points = 10000
14+
retention_seconds = 604800

‎dev/analytics.js‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
(function () {
2+
var img = new Image();
3+
var params = [];
4+
5+
params.push('page=' + encodeURIComponent(location.pathname + location.search));
6+
params.push('host=' + encodeURIComponent(location.host));
7+
8+
if (document.referrer) {
9+
params.push('ref=' + encodeURIComponent(document.referrer));
10+
}
11+
12+
var lang = navigator.language || navigator.userLanguage || '';
13+
if (lang) {
14+
params.push('lang=' + encodeURIComponent(lang));
15+
}
16+
17+
function getQueryParam(name) {
18+
var match = new RegExp('[?&]' + name + '=([^&]*)').exec(location.search);
19+
return match ? decodeURIComponent(match[1].replace(/\+/g, ' ')) : '';
20+
}
21+
22+
[
23+
'utm_source',
24+
'utm_medium',
25+
'utm_campaign',
26+
'utm_content',
27+
'utm_term'
28+
].forEach(function (key) {
29+
var value = getQueryParam(key);
30+
if (value) {
31+
params.push(key + '=' + encodeURIComponent(value));
32+
}
33+
});
34+
35+
img.src = 'https://media.frkn.org/pixel?' + params.join('&');
36+
})();

‎dev/backfill-pixel-logs.sh‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
# Backfills pixel-agent storage from rotated nginx pixel logs.
5+
# Usage: sudo ./backfill-pixel-logs.sh /etc/pixel-agent/config.toml
6+
7+
CONFIG_PATH="${1:-/etc/pixel-agent/config.toml}"
8+
LOG_DIR="/var/log/nginx"
9+
BACKFILL_LOG="/tmp/pixel-backfill.log"
10+
11+
if [ ! -f "$CONFIG_PATH" ]; then
12+
echo "Config not found: $CONFIG_PATH"
13+
exit 1
14+
fi
15+
16+
echo "Preparing backfill log at $BACKFILL_LOG..."
17+
: > "$BACKFILL_LOG"
18+
19+
# Append older gzipped logs in reverse chronological order (oldest first).
20+
# Adjust glob if your logrotate naming differs.
21+
for gz in $(ls -1 "$LOG_DIR"/pixel.log.*.gz 2>/dev/null | sort -V); do
22+
echo "Unpacking $gz..."
23+
zcat "$gz" >> "$BACKFILL_LOG"
24+
done
25+
26+
# Append uncompressed rotated logs if any.
27+
for rotated in $(ls -1 "$LOG_DIR"/pixel.log.[0-9] 2>/dev/null | sort -V); do
28+
echo "Appending $rotated..."
29+
cat "$rotated" >> "$BACKFILL_LOG"
30+
done
31+
32+
# Append current log.
33+
if [ -f "$LOG_DIR/pixel.log" ]; then
34+
echo "Appending current $LOG_DIR/pixel.log..."
35+
cat "$LOG_DIR/pixel.log" >> "$BACKFILL_LOG"
36+
fi
37+
38+
echo "Running backfill..."
39+
pixel-agent-backfill "$CONFIG_PATH" "$BACKFILL_LOG"
40+
41+
echo "Cleaning up..."
42+
rm -f "$BACKFILL_LOG"
43+
44+
echo "Backfill complete. Restart pixel-agent to pick up the updated snapshot."

‎dev/deploy-pixel-agent.sh‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
# Build and deploy the pixel-agent binary to a remote server.
5+
# Usage: ./deploy-pixel-agent.sh user@host
6+
7+
HOST="${1:?usage: $0 user@host}"
8+
REMOTE_DIR="/usr/local/bin"
9+
CONFIG_DIR="/etc/pixel-agent"
10+
DATA_DIR="/var/lib/pixel-agent"
11+
12+
echo "Building release binary..."
13+
cargo build --release --bin pixel-agent
14+
15+
echo "Uploading binary..."
16+
scp target/release/pixel-agent "${HOST}:${REMOTE_DIR}/pixel-agent.new"
17+
ssh "${HOST}" "mv ${REMOTE_DIR}/pixel-agent.new ${REMOTE_DIR}/pixel-agent && chmod +x ${REMOTE_DIR}/pixel-agent"
18+
19+
echo "Ensuring directories and user..."
20+
ssh "${HOST}" "id -u pixel-agent &>/dev/null || useradd --system --no-create-home pixel-agent"
21+
ssh "${HOST}" "mkdir -p ${CONFIG_DIR} ${DATA_DIR} && chown pixel-agent:pixel-agent ${DATA_DIR}"
22+
23+
echo "Uploading config if missing..."
24+
ssh "${HOST}" "test -f ${CONFIG_DIR}/config.toml || echo 'Config not present, copy config-pixel-agent-example.toml manually'"
25+
26+
echo "Uploading systemd unit..."
27+
scp dev/pixel-agent.service "${HOST}:/etc/systemd/system/pixel-agent.service"
28+
29+
echo "Reloading and restarting..."
30+
ssh "${HOST}" "systemctl daemon-reload && systemctl enable --now pixel-agent && systemctl status pixel-agent --no-pager"
31+
32+
echo "Done."

0 commit comments

Comments
 (0)