Operating Ceph: Capacity, Recovery Headroom, and the Questions Raw Terabytes Don’t Answer

Raw terabytes answer a procurement question: how much storage media did we buy? They do not answer the operational question: how much data can this Ceph cluster safely accept?

That distinction matters because a distributed storage system spends capacity on more than user data. It stores replicas or erasure-code chunks, distributes objects across failure domains, reserves room for recovery, and accumulates imbalance as devices and workloads change. A cluster can have free space in aggregate and still be unable to place or recover data where it needs to.

In a previous role, I operated a Ceph cluster for the first time and had to learn the system while administering it without another Ceph operator. The environment details remain private. The useful lesson does not: capacity has to be evaluated as a recovery condition, not a single utilization percentage.

This article explains the model I use to ask better questions. It is not a sizing recipe for a particular deployment.

Start with the system Ceph is actually operating

Ceph’s base storage layer is RADOS, the Reliable Autonomic Distributed Object Store. Applications reach it through interfaces such as RBD block devices, CephFS, or the object gateway. Beneath those interfaces, several components divide responsibility:

• OSDs, or object storage daemons, store data and participate in replication, recovery, backfill, and rebalancing.
• Monitors maintain cluster maps and use consensus to establish authoritative cluster state.
• Managers expose additional monitoring and management functions.
• Metadata servers manage file-system metadata when CephFS is in use.
• CRUSH, Ceph’s placement algorithm, determines where object replicas or chunks should live according to the cluster map and placement rules.

The Ceph Reef architecture documentation describes these roles and the path from objects to placement groups and OSDs. The operational consequence is straightforward: no single capacity number describes the whole path.

Before interpreting free space, I want to know:

1. Which client interface and pool will receive the data?
2. Is that pool replicated or erasure-coded?
3. What failure domain does its CRUSH rule use?
4. Are placement groups clean and active?
5. Is utilization reasonably balanced across the OSDs that can satisfy the rule?
6. What failure must the cluster be able to recover from without crossing a stop threshold?

If those questions are unanswered, a raw-capacity total creates confidence without evidence.

Convert raw capacity through the protection policy

A replicated pool stores multiple copies of each object. With a replica count of three, one unit of logical data consumes roughly three units of raw storage before other effects. An erasure-coded pool divides data into k data chunks and m coding chunks; its theoretical space-amplification factor is (k + m) / k. A k=4, m=2 profile therefore consumes about 1.5 units of raw storage for one unit of logical data.

Those ratios are starting points, not promises. Ceph’s erasure-coding documentation notes both the capacity advantage and the performance tradeoffs, especially during recovery and backfill. It also explains that k, m, and the CRUSH failure domain jointly define storage overhead and durability.

Even a correct protection ratio does not produce a safe usable-capacity figure. At minimum, the estimate must also account for:

• expected failure and maintenance scenarios;
• the OSDs and failure domains eligible for the pool;
• uneven device utilization;
• metadata and small-object amplification;
• near-full, backfill-full, and full thresholds;
• growth during the time required to repair or replace hardware; and
• workload behavior during degraded operation.

This is why I avoid presenting “raw divided by three” as the answer for a size-three pool. It estimates data-protection overhead. It does not establish recovery headroom.

Failure domains turn free space into a placement problem

CRUSH can separate copies or chunks across hosts, racks, or another modeled boundary. That policy is valuable only when the topology and capacity behind it are real.

Consider a replicated pool whose rule requires three distinct hosts. Free capacity stranded on one host cannot substitute for missing space on another. The same issue appears with erasure coding: a profile needs enough eligible failure domains to place its chunks, and recovery needs somewhere valid to move or reconstruct them.

Placement groups make this visible. A placement group, or PG, is a logical grouping Ceph uses to distribute objects across OSDs. When the cluster map changes, Ceph remaps PGs and moves or reconstructs their contents. The placement-group documentation explains the resulting states, including active, clean, degraded, recovering, and backfilling.

The important distinction is between logical availability and complete redundancy. A degraded PG may still serve I/O, but some objects do not have the intended number of copies. “Client traffic works” is not equivalent to “the protection policy is satisfied.”

Recovery headroom is capacity with a failure in progress

Recovery headroom is the space and operating margin required to restore the intended data-protection state after a modeled failure. It depends on more than the amount of data on the failed device.

A useful review models the failure explicitly:

• Which OSD, host, or other failure domain is unavailable?
• How much data must be reconstructed or moved?
• Which remaining OSDs are eligible destinations?
• Will any destination cross a backfill-full or full threshold?
• How long could recovery take under current client load?
• How much new data may arrive during that interval?
• What happens if another component becomes unavailable before recovery completes?

The model should include planned maintenance. Taking a host down intentionally still removes a failure domain. If the cluster cannot tolerate that state while continuing normal growth, the maintenance plan is consuming the same margin reserved for an unplanned failure.

Ceph reports distinct health checks when an OSD approaches capacity limits. In Reef, OSD_NEARFULL is an early warning, OSD_BACKFILLFULL indicates that an OSD is too full to accept backfill, and OSD_FULL means writes are blocked at the full threshold. PG_RECOVERY_FULL and PG_BACKFILL_FULL identify recovery or backfill blocked by destination capacity. These are not interchangeable alerts; the health-check reference describes different operational states.

A cluster that has free raw terabytes but cannot backfill is not healthy enough to absorb ordinary growth.

Use multiple views as evidence

The dashboard is useful, but the decision should survive a comparison with cluster state. For a capacity review, I would collect at least:

• overall cluster health and detailed health messages;
• logical and raw usage by pool;
• OSD utilization, including variance between devices;
• the CRUSH tree and the pool’s placement rule;
• PG state counts and any stuck, degraded, misplaced, or undersized PGs;
• configured near-full, backfill-full, and full ratios;
• recovery or backfill activity and throughput;
• recent growth rate for the relevant pool; and
• hardware or media faults that reduce the capacity actually available.

Common Ceph interfaces for these views include ceph status, ceph health detail, ceph df detail, ceph osd df tree, ceph pg stat, and ceph osd dump. The exact collection method should be documented and access-controlled for the environment. A production runbook should also state which output is retained as change evidence and who interprets ambiguous health states.

I do not treat a single HEALTH_OK observation as a capacity approval. It is a point-in-time summary. The decision also needs trend data, pool policy, topology, and a modeled recovery case.

Capacity thresholds are operational policy

Ceph’s thresholds provide safeguards, but their existence does not decide how much headroom an organization should maintain. The operating limit should sit below the technical stop points by enough margin to detect growth, schedule work, and recover from the agreed failure.

That policy needs named actions. For example:

• Normal: growth remains inside the approved recovery model.
• Review: projected growth or OSD imbalance consumes the planning margin.
• Restrict: new allocations require an owner and a recovery-impact review.
• Stop: growth pauses because backfill, recovery, or the modeled failure would cross a safety threshold.

The percentages are environment-specific. I would not invent universal values because media size, failure-domain count, workload, recovery speed, procurement lead time, and risk tolerance all change the answer. What matters is that each threshold has an owner, an evidence source, and a response.

Watch the failure modes hidden by aggregate totals

Several conditions can make a reassuring total misleading:

• Skewed OSD utilization: the fullest eligible OSD reaches a threshold before the average does.
• Incorrect topology: the CRUSH map does not represent the physical failure boundary the policy assumes.
• Degraded redundancy: logical data remains available while recovery capacity is already being consumed.
• Blocked backfill: valid destinations are too full even though unrelated devices have space.
• Fast workload growth: the cluster crosses a limit before hardware can be added and data can rebalance.
• Slow or competing recovery: client I/O and recovery contend for network, CPU, and storage performance.
• Unowned alerts: health warnings exist, but no operator is accountable for acting on them.

The last failure is organizational, not technical. It is still a storage failure mode.

Leave a recoverable system for the next operator

Operational ownership is incomplete if the next person receives only a dashboard and a list of nodes. The handoff should explain:

• pool purpose, protection policy, and client dependencies;
• CRUSH rules and the physical boundaries they represent;
• approved capacity and recovery assumptions;
• alert thresholds and escalation owners;
• expected PG states during maintenance and failure;
• safe procedures for adding, removing, or replacing OSDs;
• recovery controls and the reason for any tuning;
• evidence required before declaring recovery complete; and
• known exceptions, temporary states, and their expiration dates.

Administrative access should follow least-privilege principles. Capacity reporting usually requires observation, while topology changes, pool changes, and OSD lifecycle operations can alter data placement or availability. Those are different trust levels and should not share an informal access path.

Raw capacity is inventory, not readiness

Ceph turns independent storage devices into a system that can place, protect, and recover data across failures. That capability is why the architecture is useful. It is also why the raw total is insufficient.

The defensible capacity answer connects five things: the pool’s protection policy, the CRUSH failure domain, current placement and health, the modeled recovery event, and the organization’s response thresholds. If one is missing, the number describes installed media more accurately than operational capacity.