Once an observability platform goes multi-tenant, permissions stop being optional. This article breaks down three-layer isolation, RBAC resource granularity, permission-check performance, and how to kill shared accounts and permission creep.
Once an observability platform goes multi-tenant, permissions stop being optional. Logs routinely carry phone numbers, order IDs, and API response bodies — if an operator in tenant A can read tenant B's logs, that is a data breach. Early on we ran an internal penetration test where we deliberately built a query endpoint that "forgot" the tenant filter, and a single SQL statement pulled out another tenant's entire order log. After that, isolation and permissions became a core part of the platform. This article breaks down how Jujing OBSERVE structures tenant isolation and RBAC.
Isolation cannot be solved with a single layer. We split data access into three:
The first layer is the tenant. Every record is tagged with tenant_id at write time, and search, alerting, and trace queries all enforce a mandatory tenant_id filter. The key is double enforcement: the application layer appends the tenant condition when building SQL, and the storage layer applies a row-level policy as a backstop — so a forgotten WHERE clause cannot leak data. Both layers have to fail for a breach, which drops the risk from "eventually someone will forget" to "almost impossible."
The second layer is the environment. A tenant typically runs production, staging, and test environments with the same log shape but different sensitivity. Environment acts as a second-level isolation; production and test do not talk to each other by default. Reproducing a production issue in test requires a separately requested, restricted export permission.
The third layer is the data stream. Certain fields — phone numbers, ID numbers, tokens — need masking and are shown in plaintext only to authorized roles. Masking happens at the presentation layer while storage keeps the original value, decoupling masking from authorization and keeping change costs low. Masking uses field-level regex rules: a phone number matching 1[3-9]\d{9} becomes 138****5678, configured centrally so rules change without touching code.
Once the three layers stack up, "who can see what" becomes a fixed set of rules instead of a single fragile query.
The permission model is classic RBAC, but the hard part is defining "resource" precisely. We set resource granularity to "data stream + environment + action":
role: read-only-ops
permissions:
- resource: "logs"
actions: ["read", "search"]
environments: ["prod", "staging"]
- resource: "alerts"
actions: ["read", "ack"]
environments: ["prod"]Roles fall into five groups by responsibility: admin (user:manage and data:export), operator (read/write logs and alerts, operate collectors), developer (read-only on their own services' logs and traces), auditor (audit:read only, never sees plaintext of masked fields), and visitor (dashboards only). Policies support tag-based conditions, such as "only the finance group can view logs from applications tagged finance."
Permission checks run in a shared middleware. Each data endpoint declares resource and action, and the middleware enforces them centrally instead of hand-written checks scattered across handlers. Check results are cached by "user + role" so every request does not hit the permission table; role changes actively invalidate the cache so permission edits take effect immediately.
Multi-tenant incidents rarely come from a complicated model — they come from a new feature that forgot to plug into permissions. A new export endpoint or query page that allows access by default leaves a hole. Our rule is deny by default: a new endpoint returns 403 unless it explicitly declares permissions.
The second trap is shared accounts. When several people share one admin account, you cannot tell who did what after an incident. We enforce individual accounts plus operation auditing; the admin account is used only once at initialization, and every subsequent login, config change, and data export is recorded. Audit logs are stored separately, append-only, and read-only for auditors — nothing can delete or modify them.
The third trap is permission creep: an operator asks for access to log A today, log B tomorrow, and six months later their scope has snowballed. We set an expiration on temporary grants so they auto-revoke, and we produce a periodic permission report for the owner to review. That turns authorization into a "loan" rather than a "permanent grant."
Multi-tenant permission design is about turning "who can see what" from a verbal agreement into machine enforcement. The earlier you do it, the less technical debt you owe later.