JJH OBSERVE v2.5.0: Log Pipelines and Alert Noise Reduction

New Features

This release upgrades logs from "stored and searchable" to "processable and routable":

• Log Pipeline: parse, filter, extract fields, and mask logs at the collection source before ingestion. Split nginx access logs into status, upstream_time, request_uri, and query them directly with SQL
• Alert noise-reduction engine: auto-aggregates same-source alerts, with silence windows, alert suppression (higher-severity alerts auto-mute lower ones), and dedup/debounce
• Cross-service metric correlation: overlay downstream QPS and error rate directly in the trace view to see where the bottleneck sits

Pipelines use declarative YAML with hot reload — no collector restart. An example that splits nginx access logs into fields:

pipelines:
  - name: nginx-access
    match: source == "nginx"
    steps:
      - parse:
          format: regex
          pattern: '^(?P<ip>\S+) .* "(?P<method>\S+) (?P<request_uri>\S+)" (?P<status>\d+) .* (?P<upstream_time>\S+)$'
      - mask:
          fields: [ip]

The config applies to new logs immediately after saving; existing logs are not back-filled.

The noise-reduction engine works in two layers. The first deduplicates identical alerts from the same source within a configurable window (default five minutes), so a flapping threshold sends one notification instead of forty. The second applies suppression rules: when a P0 or P1 alert is active, lower-severity alerts from the same service are held until it clears. Both layers are configurable per tenant, and the dedup window, silence windows, and suppression relationships are all set in YAML, not hard-coded.

Improvements

• Query performance: SQL aggregations average 40% faster via vectorized execution and partition pruning
• Index management: automatic cold/hot tiering — data older than 30 days moves to object storage, queries stay transparent
• Trace sampling: dynamic sampling rate based on error rate; anomaly windows sample more aggressively
• Console: topology filters by namespace/environment; dashboards support custom metric cards

Fixes

• Fixed arm64 timezone init failure that shifted log timestamps (the Kylin deployment pitfall)
• Fixed a rare Collector memory leak under high concurrency that grew slowly over time
• Fixed DM database metadata connections not retrying after a restart
• Fixed occasional GROUP BY timeouts on high-cardinality fields in log search

Breaking Changes and Deprecations

• Alert rules created before v2.3.0 using the legacy threshold syntax are deprecated and will be removed in v2.6.0. The migration tool rewrites them to the new format; run migrate-alerts before upgrading to v2.6.0.
• The old REST API endpoints under /api/v1/logs are deprecated in favor of /api/v2/logs. They still work in v2.5.0 but return a deprecation header; plan to move by the next release.
• The log pipeline processor syntax changed slightly: the filter step now uses match instead of the old where keyword. Existing pipelines are auto-converted on upgrade.

What's Next

The v2.6.0 milestone is focused on three things. First, anomaly detection on metrics: a baseline learned per service and per hour, so alerts fire on deviation rather than fixed thresholds. Second, a logs-to-metrics pipeline so you can turn high-cardinality log fields into counters and gauge series directly from the collector, cutting storage for logs you only ever query as numbers. Third, better multi-tenant isolation for managed-service and platform-team deployments — per-tenant quotas and data retention policies.

If you rely on the legacy alert syntax or the v1 log API, now is the time to migrate; both are scheduled for removal. As always, the full, continuously-updated changelog lives on the docs site, and the upgrade script is the supported path for every version.

Upgrade Notes

From v2.4.x:

./upgrade.sh --version 2.5.0

The script backs up config and metadata automatically; data directories stay in place. Recommended order: upgrade the query layer, then collectors, then the alert module — all rolling with no write interruption. Validate pipeline config syntax in staging before pushing to production.

Before the production cutover, run the upgrade against a staging clone and replay a day of production traffic through the pipeline to confirm field extraction and masking behave as expected. The pipeline syntax is validated on save, but validation catches syntax, not semantics — a regex that parses your staging logs but not production's edge cases will only show up under real traffic.

Known issue: upgrading directly from v2.4.0 or earlier requires a one-time manual migration of custom alert rules (the script provides a migrate-alerts subcommand). Run it on a single node only — repeating it duplicates the rules. The full changelog is on the docs site.