Metrics tell you a service is slow; flame graphs tell you which line is slow. This post covers OBSERVE continuous profiling: reading flame graphs, low-overhead sampling config, and three troubleshooting scenarios for CPU, memory, and lock contention.
CPU hits 90%, and your endpoint's p99 jumps from 50 ms to 300 ms. Metrics tell you something is wrong; traces tell you it's the order-service; but to answer "which function is burning CPU, which object is eating memory," you need profiling. Metrics and traces are great at narrowing the blast radius. Profiling is the step that names the culprit.
The old approach is to attach pprof or async-profiler when something breaks — except by the time you connect, the traffic spike has passed and you've missed the scene. OBSERVE's continuous profiling turns that one-shot capture into an always-on recorder: by default it samples once every 60 seconds for about 10 seconds each time, keeping CPU overhead under 3% while continuously collecting stack traces from every process. When something goes wrong, you can rewind through the last 15 minutes — or longer — of stack snapshots. No reproduction required, and no "it didn't reproduce on my machine" during the postmortem.
The horizontal axis is sample share, the vertical axis is the call stack, and the wider a bar, the more it costs. Read it in this order:
A common mistake is looking only at total width. What you actually want to optimize is self time — the time a function spends in its own code, excluding the children it calls. Some functions are narrow themselves but are invoked from tens of thousands of call sites; those add up too, and you find them with the "callers" view rather than the "callees" view. Two more pitfalls: recursion flattens a call chain into a confusing wide bar, and JIT inlining can hide a function entirely — so if a frame you expect isn't there, check whether the JIT inlined it rather than assuming it's cheap.
Install a collection agent on the host and label the process — no code changes needed. A typical config:
profiling:
enabled: true
interval: 60s
duration: 10s
targets:
- service: order-service
process: java
lang: jvm
cpu: true
alloc: trueFor JVM services enable both CPU and allocation sampling; for Go use pprof's CPU and heap profiles; for Node.js watch event-loop blocking and GC. Keep sampling overhead under 3% and you can leave it on in production. You can also configure it to raise the sampling frequency in low-traffic windows and dial it back during peaks automatically, so the overhead tracks actual load. If a host is already running hot, disable allocation sampling there and keep only CPU — allocation tracing is usually the more expensive of the two.
Tie profiling to alerting: when a CPU or memory alert fires, automatically capture a 5-minute flame-graph snapshot around the incident so you skip manual reproduction and walk into the postmortem with concrete evidence. That one change alone often cuts the time from "alert" to "understood" by more than half.
A final note worth internalizing: profiling data is cheap to collect and expensive to ignore. Teams that turn it on during a quiet week usually find a CPU sink or a slow-growing leak that would have surfaced during the next peak. The value also compounds — flame graphs from a healthy window become the reference you diff against when things break, which is far easier than reconstructing what "normal" looked like from memory. Start with CPU and heap on your three busiest services and expand from there; you'll likely have answers before the next incident, not after.