Migrating Off ELK to a Domestic Stack: A Field Guide

Why move off ELK

Plenty of government units still run a self-hosted ELK stack for logs. The problem isn't the technology — it's compliance. Elasticsearch changed its license at 7.11 and the enterprise features stopped being free; a fork (OpenSearch) exists but still means owning the whole stack. Meanwhile, domestic-computing (xinchuang) acceptance requires a stack that runs on domestic hardware and software. An ELK on X86 servers simply won't pass review.

Moving to a domestic stack means three things: domestic servers, a domestic OS, and a domestic or open-source database. Below is our field record of replacing the log platform for one government agency.

Assess before you move

Don't touch anything until you know what you have:

1. Volume: how many GB of logs per day, and how long they're retained — this drives storage and retention policy.
2. Ingestion surface: which systems, how many instances, and which agent (Filebeat / Logstash / homegrown).
3. Query habits: the query patterns your team actually relies on, so they map cleanly onto the new platform.

The assessment determines the migration path: small volume with an acceptable outage means a one-shot migration; large volume that can't stop means dual-write. We've seen teams skip this step and discover mid-migration that a critical dashboard depended on a field nobody documented — don't be that team.

Three migration paths

• One-shot: export history (JSON/CSV) into the new platform during a maintenance window. Fits small volumes with little need for historical search.
• Dual-write (recommended): old and new platforms both receive logs for a while; once you confirm the new platform is complete and query results match, turn the old one off.
• On-demand backfill: keep history in cold storage, send new data to the new platform, and export history only when someone needs it. Cheapest, but historical search gets awkward.

We recommend dual-write. It's simple: send each log line twice — once to the old ELK, once to Observe — and compare for a week.

Deploying on the domestic stack

Target environment: Kunpeng 920 servers + Kylin V10 (ARM) + OceanBase (MySQL mode).

# 1. Download the ARM binary
wget https://ob.example.com/releases/observe-arm64.tar.gz
tar -xzf observe-arm64.tar.gz

# 2. Configure the database connection (OceanBase, MySQL protocol)
cat > env <<EOF
DB_HOST=10.0.1.10
DB_PORT=2881
DB_USER=observe
DB_PASS=********
DB_NAME=observe
JWT_SECRET=$(openssl rand -hex 32)
EOF

# 3. Start
./observe --config ./env

Because Observe speaks the MySQL protocol, pointing it at OceanBase means swapping host and port — zero code changes. The same applies to other MySQL-compatible domestic databases such as TiDB or PolarDB.

Data security during the move

Dual-write means two copies of sensitive data are in flight, so treat the transition window as higher risk, not lower. Enable field-level scrubbing for phone numbers and ID numbers on the new platform from day one, so nothing sensitive is ever stored in the clear. Restrict access to the new platform to the migration team until cutover, then open it with the same permission model as the old one.

Also decide what happens to the old Elasticsearch cluster. Decommissioning isn't just shutting it down — the disks still hold data. Wipe or physically destroy them per your data-handling policy, and keep a signed record, because auditors will ask what happened to the old copy.

Cutover and acceptance

• Data consistency: pick 20 known error log lines at random and confirm both platforms return them, with matching results.
• Query performance: the same aggregate query on the new platform should be no slower than 2x the old — usually it's faster.
• Compliance: data never leaves the internal network, logs are permission-isolated, sensitive fields are scrubbable, and operations are audited.
• Rollback: keep the old ELK alive for a week, and only decommission it after it's been stable.

After the migration

• Port the old Grok parsing rules into the new platform's structured parsing rules with the same field names, so the team's habits don't change.
• Migrate alert rules one by one and test that each actually fires — don't wait for a production incident to validate your alerts.
• Review retention and cold-storage settings now, not later: many teams find the new platform's compression lets them keep far more history on the same disk.