Migrating off ELK to a Sovereign Observability Platform: A Cutover Plan

Why migrate, and why it's hard

Government and enterprise teams move their log stack off ELK for two reasons: sovereignty compliance, which requires core systems to run on domestic hardware and software, and ELK's licensing and operational cost, which grows uglier as data volume climbs — more nodes, messier Grok rules, and someone dedicated to babysitting cluster health.

But a log system is "the production system behind the production system" — a bad migration is an incident. The hard parts come down to three things: historical logs must not be lost, existing queries and dashboards must keep working, and there must be no data gap during the move. Here's a four-stage breakdown, each with a verifiable deliverable.

Stage 1: Take inventory

Map your existing assets before touching anything, or the migration becomes a blind cutover:

• List every Logstash pipeline, recording its input sources, Grok rules and outputs. This is the most overlooked step, and Grok rules are the leading cause of field mismatches during migration
• List the Kibana dashboards and alerts actually in use — not all of them (plenty of teams have dozens, and use fewer than five)
• Measure average daily log volume and peak QPS to size the new platform, so you don't discover mid-migration that the spec is too small
• Confirm retention and compliance requirements — finance often mandates 3+ years

The output is a checklist you'll verify against at every later stage.

Stage 2: Dual-write in parallel

Don't cut traffic over in one shot. First have the application send logs to both ELK and the new platform for one to two weeks. This validates ingestion, parsing and search on the new side while accumulating a batch of "new-format" data to align backfill against.

During dual-write, compare log counts, fields and timestamps between the two sides. A count mismatch usually means the collection side is dropping data; a field mismatch usually means parsing rules weren't fully migrated. Run a daily reconciliation at a fixed time and log the differences — don't wait until cutover day to find them.

Stage 3: Backfill history

Historical logs are the easiest place to trip up. Don't dump terabytes of indices from Elasticsearch as-is, for two reasons: index mappings and field types won't line up, and it inflates storage cost.

A more practical approach is "backfill on demand": import only the high-frequency last 30–90 days, keep older logs as cold storage in the old ELK, and leave the query entry point available for a while. Use elasticsearch-dump or scrolling export, and throttle concurrency so you don't crush the source cluster.

# Rolling export of the last 30 days of order-service logs
multielasticdump --input=http://old-es:9200 \
  --output=/backup/logs \
  --match='order-service-*' \
  --scrollTime=10m \
  --concurrency=2

Backfill also needs field mapping: Elasticsearch's @timestamp, nested fields, and the custom fields Grok extracts all have to map onto the new platform's field model. Run a pilot import of one index first, confirm the fields line up, then batch the rest.

Stage 4: Cutover and cleanup

Cut over during a low-traffic window, in this order: switch the query entry point to the new platform first, then stop writes to the ELK side, then decommission Logstash. This order guarantees at least one system is always receiving logs.

Three cleanup tasks:

• Migrate dashboards: Kibana's query syntax differs from the new platform's SQL-like syntax, so translate only the few dashboards you actually use
• Migrate alerts: re-create ElastAlert or Watcher rules in the new platform — note that dynamic baselines can replace many hand-tuned fixed thresholds
• Keep a rollback window: leave ELK in read-only mode for at least 30 days so you can switch back if something breaks

The three most common pitfalls

Most failures we've seen weren't technical. They came down to three causes:

1. A sloppy inventory: Grok rules, dashboards and alerts weren't fully listed, so after cutover fields parse wrong and key alerts never migrated
2. A dual-write window too short: cutting over after two or three days leaves parsing differences in the new platform undiscovered
3. Greedy backfill: trying to import years of data at once crushes the source cluster, explodes storage cost and slows the cutover

Avoid those three and the migration is basically stable. The core principle in one sentence: better to go a little slow than to lose a single log or break a single alert.