Observe v2.4.0: Ingest Agents, Alert Silence Windows and Faster Search

v2.4.0 concentrates on three directions: easier onboarding, more controllable alerting and faster search. If you are in a hurry, jump to the end for upgrade steps and compatibility notes; the rest walks through each change in turn.

New: a lightweight ingest agent

This release introduces a single-binary ingest agent (about 20MB) that runs on application hosts to collect log files and system metrics, replacing hand-rolled Logstash or Filebeat setups. It supports JSON parsing, multi-line stack-trace merging and regex filtering, and its configuration is driven from the platform — changing a collection rule no longer means SSH-ing into every machine.

curl -fsSL https://dl.jjhub.cn/agent/install.sh | bash
jjhub-agent --token <token> --endpoint https://ob.example.com

The agent buffers locally by default: if the platform is briefly unavailable, logs are written to disk and queued, then backfilled automatically once connectivity returns. That matters in production — a network blip should never be a reason to lose logs.

New: alert silence windows and on-call rotation

• Silence windows: scoped by service, host or time range, one-click muting during releases or load tests, auto-restored on expiry, with optional change-management integration to open and close them automatically
• On-call rotation: weekly or monthly schedules with automatic handoff, so alerts stop going to the whole team
• Alert grouping: one wave of flapping collapses into a single aggregated event, reducing notification noise

Together these three capabilities address the same old problem: too many alerts means nobody reads them. For configuration and usage, see the companion article on taming alert storms.

Improved: search and export

• Log search: large time-range queries are about 40% faster, and with hot/cold index tiering, cold data no longer slows down hot queries
• Tracing: smoother hydration for long traces, so a trace with tens of thousands of spans still opens instantly
• Export: query results can be exported as CSV or JSON, with the per-export limit raised from 10,000 to 100,000 rows
• New saved queries: frequently used search statements can be saved as templates and shared across the team

Benchmark data

The performance work was measured on an internal benchmark (16 cores / 64GB RAM, 100 million log records):

• Large time-range queries (30 days): P99 latency dropped from 3.2s to 1.9s, about 40%
• Opening a trace with tens of thousands of spans: from 2.8s down to 0.9s
• Ingest agent memory: about 40MB steady state, 120MB at load-test peak

Numbers vary with environment and data distribution, so treat these as indicative and rerun them on your own dataset before drawing conclusions.

Fixes

• Fixed a compile failure caused by a naming conflict between the SMS module and a utility function
• Fixed alert rules failing to trigger in certain scenarios
• Fixed the file watcher occasionally missing the tail of a log file on Kylin
• Fixed paginated search returning duplicate records on the Dameng database
• Fixed occasional audit-log writes being lost under high concurrency

Upgrade steps and compatibility

1. Back up the current data directory and configuration
2. Replace the old binary with the new one and restart (configuration is backward-compatible, no changes required)
3. Confirm the version shows v2.4.0 on the version page, then run the ingestion and alert self-checks

Compatibility: v2.4.0 is backward-compatible for the API, database schema and ingestion tokens, so existing agents do not need a coordinated upgrade. Upgrade production during off-peak hours — roll out to one instance first, watch it for an hour, then go wide. If anything goes wrong, restore the previous binary and the backup; there are no destructive data-layer changes.

Preview of the next release

The current plan for v2.5:

• Ingest agent support for Windows and more domestic operating systems
• Alert escalation: P0 alerts automatically escalate to a supervisor when nobody acknowledges them
• Regex and fuzzy matching in search
• One-click deployment scripts for sovereign environments

These are planned directions, not committed dates; the formal release notes take precedence.