tuniq

tuniq is a Unix command-line utility for streaming frequency analysis.

Overview

• Replaces sort | uniq -c | sort -rn | head with one tool.
• Works on unsorted stdin or multiple files.
• Uses worker sharding for multicore throughput.
• Sorts only unique values for deterministic output.
• Writes results to stdout and diagnostics/stats/progress to stderr.

Installation

From source

go install github.com/flaviomartins/tuniq/cmd/tuniq@latest

Building from source requires Go 1.25 or newer.

From release artifacts

Download binaries from GitHub Releases and place tuniq in your PATH. Release binaries are built with the latest stable Go toolchain.

Building

make build

or:

go build -o tuniq ./cmd/tuniq

Usage

tuniq [flags] [file ...]

If no files are provided, tuniq reads from stdin. Multiple files are processed as one logical stream.

Flags:

• -n N top N entries
• -l N, --max-lines N stop after reading N lines (0 means no limit)
• -a show all entries
• -r reverse ordering (ascending count)
• -c show counts (default true)
• --no-count hide counts
• -u N, --update-every N live updates every N lines (plain output only)
• --no-status hide the live status bar (spinner, rate, sparkline)
• --status show the live status bar (overrides status=false in config)
• --csv CSV output
• --json JSON output
• --workers N worker shards (default: GOMAXPROCS)
• --progress, --progress-every, --progress-every-seconds progress cadence
• --stats, --stats-rss processing stats
• --memory-limit-bytes hard stop on estimated counter memory
• --version, --help

Examples

Basic usage

tuniq -n 10 data.txt

Multiple files as one stream

tuniq data.0.txt data.1.txt data.2.txt

Live top-N from a stream

curl -q -sN https://stream.wikimedia.org/v2/stream/recentchange \
  | jq --unbuffered -r '.title // empty' \
  | tuniq -n 20 -u 1

From a stream with ssecat

ssecat https://stream.wikimedia.org/v2/stream/recentchange \
  | jq --unbuffered -r '.title // empty' \
  | tuniq -n 20 -u 1

Value-only output

cat queries.txt | tuniq --no-count

Machine-readable output

cat queries.txt | tuniq --csv -n 100
cat queries.txt | tuniq --json -n 100

Configuration

tuniq loads defaults from these paths in order (later entries override earlier):

1. ~/.tuniqrc (legacy homedir)
2. ~/.config/tuniq/.tuniq (or OS equivalent from os.UserConfigDir)
3. ./.tuniqrc (project override)

Config format is key=value with # comments. CLI flags always override config values.

Supported keys:

top_n=20
max_lines=0
show_all=false
reverse=false
show_count=true
update_every=0
output=plain
workers=8
progress=true
progress_every=500000
progress_every_seconds=0
stats=false
stats_rss=false
status=true
memory_limit_bytes=0

XDG directories

Config: ~/.config/tuniq/.tuniq
State: ~/.local/state/tuniq (or OS equivalent)

Package layout

• cmd/tuniq: CLI entrypoint and signal wiring.
• pkg/processor: stream processing orchestration and runtime pipeline.
• pkg/platform: OS-specific RSS helpers.
• pkg/config, pkg/output, pkg/version: configuration, output formatting, and build metadata.

Output

Default plain output:

15234 apple
11201 orange
9321 banana

Ordering is deterministic:

1. count descending (or ascending with -r)
2. value alphabetical as tiebreak

Live mode status bar

When live updates are enabled (-u), a status bar appears below the top-N table:

15234 apple
11201 orange
 9321 banana
 8102 grape
 7654 mango
⠹ streaming  1,250,000 lines  ▁▂▄▅▇▇█▆  45.2k/s

The top-N table redraws on the configured live cadences (-u for lines and --progress-every-seconds for time). Between those redraws, tuniq still refreshes the status bar about once per second so active streams do not look stalled and idle open streams, including before the first line arrives, can flip to waiting:

Element	Meaning
⠹ (spinning)	Data is actively arriving — spinner advances with each update
⠹ (frozen)	Stream is open but silent — no EOF received yet
streaming	Lines are flowing in (green)
waiting	No new lines since last render (yellow)
✓ complete	EOF received — final counts are exact (dim)
1,250,000 lines	Total lines processed so far
▁▂▄▅▇▇█▆	8-sample rolling throughput sparkline
45.2k/s	Current ingest rate

Use --no-status (or status=false in config) to hide the bar entirely.

Performance and memory

• Counting is shard-local, then merged.
• Sorting happens over unique keys, not total lines.
• Memory growth is proportional to cardinality.
• --memory-limit-bytes aborts when estimate exceeds limit (spill-to-disk is not implemented).

Development

make fmt
make vet
go test ./...
make build

Contributing

Contributions are welcome. Please run formatting, vet, and tests before opening pull requests.

License

MIT (see LICENSE).