Observe v2.3.0: Log Clustering, Alert Deduplication and Tiered Storage

New: automatic log clustering

During promotions and incidents, log volume spikes and error logs mix dozens of distinct failures together. Going through them one by one isn't realistic. v2.3.0's log clustering groups similar errors automatically, ranks them by frequency, and tells you which categories deserve attention right now.

Clustering matches on log templates (variables extracted into wildcards) plus lexical similarity. These three lines collapse into one cluster:

ERROR [order] create order 1000231 failed: timeout
ERROR [order] create order 1000234 failed: timeout
ERROR [order] create order 1000239 failed: timeout

After template extraction this becomes ERROR [order] create order * failed: timeout. Click a cluster to expand the raw logs — no guessing keywords before searching. Rare one-off errors land in an "other" bucket so they don't drown out the genuinely high-frequency problems.

New: alert deduplication

Alert storms are the top pain point in operations. A single database outage can cascade into dozens of alerts, burying the root cause in a wall of noise.

v2.3.0 adds two layers of noise reduction:

• Correlation grouping: merge alerts for the same service and metric within a time window into one, with the alert count and peak value attached
• Root-cause inference: rule-match common chains (e.g. DB connection-pool exhaustion → upstream interface timeout → order-service errors) and mark downstream alerts as "suspected caused by X", yielding one primary alert with linked secondary alerts

Combined with the existing dynamic baseline, signal-to-noise improves noticeably. Deduplicated alerts can be delivered via email, webhook or WeCom, and each grouped alert carries a "view all related alerts" link so no detail is lost.

New: tiered storage

As log volume grows, storage dominates cost. Yet more than 90% of searches hit the last 7 days; month-old logs are rarely touched.

Tiered storage recovers that cost:

• Hot tier: last 7 days, SSD, fully indexed, millisecond search
• Warm tier: 7–30 days, spinning disk, partially indexed
• Cold tier: 30+ days, compressed archive, query on demand after thaw

Tiers are configurable per service and per namespace:

storage:
  tiers:
    - name: hot
      retention: 7d
      storage_class: ssd
      full_index: true
    - name: warm
      retention: 30d
      storage_class: hdd
    - name: cold
      retention: 0          # 0 means keep forever
      storage_class: archive
      compressed: true

Historical logs don't have to be deleted, and cost drops sharply. Cold data stays queryable — the first query just waits for a thaw — which suits compliance retention and low-frequency lookbacks. After moving to cold, single-log query latency rises from milliseconds to seconds, so don't shrink the default 7-day hot window: it's what keeps day-to-day troubleshooting snappy.

Improvements and fixes

• Trace hydration performance up ~35%; large spans expand more smoothly
• Lower pagination latency for log search over million-row result sets
• Fixed alert rules firing at offset times across time zones
• Fixed collector heartbeat failures occasionally causing false offline status
• Fixed occasional inaccuracy in multi-tenant quota accounting at month-end settlement

Getting started

All three features can be turned on from the console without touching your collection side: log clustering is a one-click toggle in the top-right of Log Search; alert deduplication is enabled via the "auto-group" option under a rule's advanced settings; and tiered storage is configured per namespace under Storage Management. Enable tiered storage first for your highest-volume service — that's where the savings are largest. Log clustering and alert deduplication are safe to enable everywhere: they only affect display and notifications, never the underlying data.

If clustering results look off, adjust the cluster granularity: finer granularity produces more clusters, while coarser granularity merges distinct errors into one group. When in doubt, leave the defaults for a couple of days and observe. If you're still on fixed-threshold alerts from v2.2, flip your noisiest metrics to dynamic baselines first and watch the false-alarm rate drop.

Upgrade notes

Private-deployment customers should contact the delivery team for ARM/x86 installers; SaaS users are updated automatically. Back up your config and index mappings before upgrading — rollback steps are in the delivery documentation.

Compatibility: this is a minor release, so the API and storage format are backward-compatible, and v2.2.x alert rules and dashboards carry over unchanged. If you're still on v2.2's fixed-threshold alerts, this upgrade is a good moment to switch to dynamic baselines — false-alarm rates typically drop noticeably.