Moving a self-built ELK stack to domestic ARM + Kylin? A smooth migration playbook: dual-write cutover, hot/cold archiving of history, alert-rule translation — and the pitfalls we hit.
Teams running a self-built ELK stack fear two things: Elasticsearch scaling and index management quietly consuming their engineering time, and domestic-computing compliance forcing log data onto domestic hardware and software. The hard part of migration is not moving the data — it is keeping the business running, losing no alerts, and keeping history searchable throughout. Treat it as a process change, not a tool swap.
Dual-write transition. Start by writing every log to both ELK and Observe from the application side, then watch for one to two weeks to confirm volume, fields and timestamps match. During this phase ELK stays primary while Observe runs as the shadow. Most problems — wrong fields, dropped multi-line logs, timezone drift — surface here, while the blast radius is still zero. Technically the cleanest way to dual-write is to put an OpenTelemetry Collector in front of both backends and configure two exporters, one to the old Elasticsearch and one to Observe, so the application only ever talks to one endpoint.
Verification. Use a script to sample and compare log counts and key fields between the two platforms, focusing on the three areas that diverge most: multi-line logs, JSON structure and timezones. A daily diff of counts per service catches silent ingestion gaps long before cutover.
Cutover. Once verified, point the collection layer (Filebeat or the OpenTelemetry Agent) at the new platform and stop writing to ELK. Keep ELK read-only for a while as a fallback. If anything looks wrong after the switch, the old data is still there and the rollback is a one-line config change.
You do not need to move every historical log. Split by the retention period required for compliance (commonly 90 or 180 days): migrate hot data to the new platform and export cold data to object storage, re-importing only when needed. Rebuilding ELK indices in full is slow and expensive, and most queries hit the last few weeks anyway — hot/cold tiering is the right answer. The compliance argument matters here too: archived cold data on object storage can be kept for years at a fraction of the cost of hot Elasticsearch nodes, and it satisfies the “data stays on the internal network” requirement just as well as a live index does.
Kibana Watcher and ElastAlert rules cannot be reused directly; each must be rewritten into the new platform's YAML. Sort by trigger frequency and migrate only the rules that have fired in the last 30 days — discard the dead ones:
alert_rules:
- name: error-log-burst
expr: count(logs, level='ERROR', window='5m') > 100
severity: P1
notify: [email-oncall]Treat the translation as a chance to slim down: many teams discover after migrating that more than half of their old rules never fired. Those rules are not protection — they are the noise that trained everyone to ignore alerts in the first place.
Domestic-computing environments usually come with an acceptance review, so line up these items before cutover rather than after:
log.file.path, so saved searches and dashboard queries must have their field names updated after migration.Migration is not swapping tools — it is moving your alerting and search habits along with the data. Dual-write transition plus hot/cold tiering plus rule slimming turns a big project into a low-risk cutover, and the discipline you build along the way pays off on the new platform from day one.