Backing up Kubernetes workloads is easy when the amount of data is small. The problem starts when applications use large PersistentVolumes.

Imagine a database using a 1 TB volume. If only 5 GB of data changed since yesterday, copying the entire 1 TB volume again is wasteful. It takes more time, uses more network bandwidth, and requires more storage.

This is where Changed Block Tracking (CBT) becomes useful.

Changed Block Tracking allows a storage system to identify which blocks have changed between two snapshots. A backup application can then focus on those blocks instead of scanning and transferring the entire volume.

Kubernetes introduced support for this capability through CSI snapshot metadata APIs. The feature was initially introduced as Alpha and moved to Beta with the external-snapshot-metadata v1.0.0 release in March 2026. The current Beta implementation supports block volumes, not file volumes.

What Is Changed Block Tracking?

Changed Block Tracking is a way of answering a simple question:

Which blocks of this volume changed since the previous snapshot?

Traditional backup systems often have to read a large amount of data to determine what should be backed up.

With CBT, the storage system can provide metadata describing allocated blocks and changes between snapshots.

For example, assume a 500 GB volume contains:

  • 350 GB of allocated data

  • 150 GB of unused space

  • 8 GB changed since the previous snapshot

A traditional full backup may process a large portion of the volume.

With CBT, the backup application can ask the storage system for the blocks that changed between two snapshots and transfer only the relevant data.

The important point is that CBT does not replace snapshots. It works with snapshots to make incremental backup operations more efficient.

Why Kubernetes Backups Need CBT

PersistentVolumes are commonly used by databases, file-processing applications, analytics systems, and other stateful workloads.

Consider a PostgreSQL database running on a 1 TB PersistentVolume.

The database may change only a small amount of data every day, but the volume itself remains 1 TB.

Without block-level change information, the backup application may need to process a much larger amount of data than the application actually changed.

That can affect:

  • Backup duration

  • Network traffic

  • Storage consumption

  • CPU usage

  • I/O pressure

  • Backup windows

CBT gives the backup application more precise information about what changed.

The Kubernetes project describes the feature as a way for CSI storage drivers to identify changed blocks between snapshots so backup applications can concentrate on changed data rather than processing the entire volume.

How CBT Fits Into Kubernetes Storage

CBT is not implemented entirely inside the Kubernetes API server.

It works through the Container Storage Interface (CSI) and several components work together.

A simplified flow looks like this:

Application
    |
    v
PersistentVolumeClaim
    |
    v
PersistentVolume
    |
    v
CSI Driver
    |
    v
Storage System
    |
    +---- Volume Snapshot
    |
    +---- Changed Block Metadata
    |
    v
Backup Application

The storage provider is responsible for knowing which blocks are allocated and which blocks changed.

The CSI layer exposes the required functionality.

The Kubernetes-facing metadata service makes that information available to backup applications.

The Three Important Components

The CBT implementation has three important pieces.

CSI Snapshot Metadata Service

The CSI driver can provide a snapshot metadata service.

This service exposes information about snapshot blocks through gRPC APIs.

Two important operations are:

GetMetadataAllocated
GetMetadataDelta

GetMetadataAllocated provides information about blocks allocated in a snapshot.

GetMetadataDelta provides information about blocks changed between two snapshots of the same volume.

This distinction is important for backup software because an initial backup and an incremental backup do not have the same requirements.

SnapshotMetadataService Custom Resource

Kubernetes needs a way to discover that a CSI driver provides snapshot metadata.

The SnapshotMetadataService Custom Resource is used for this purpose.

A simplified example looks like this:

apiVersion: cbt.storage.k8s.io/v1beta1
kind: SnapshotMetadataService
metadata:
  name: example.csi.driver
spec:
  address: snapshot-metadata.example.svc:8443

The actual resource contains additional information such as the service endpoint, CA certificate, and authentication audience.

The important idea is that the resource advertises the availability of the snapshot metadata service to clients.

External Snapshot Metadata Sidecar

The external-snapshot-metadata sidecar sits between Kubernetes clients and the CSI driver's snapshot metadata service.

A simplified architecture looks like this:

Backup Application
       |
       v
Kubernetes SnapshotMetadataService
       |
       v
External Snapshot Metadata Sidecar
       |
       v
CSI Snapshot Metadata Service
       |
       v
Storage System

The sidecar handles the Kubernetes-facing part of the communication and connects to the CSI driver's implementation.

The Kubernetes CSI project maintains this sidecar as part of the CBT implementation.

A Simple Backup Example

Let's consider a practical example.

Suppose an application uses a 100 GB block volume.

At 10:00 AM, the backup system creates Snapshot A.

Volume
|
+-- Snapshot A

The application continues running and changes some data.

At 10:00 AM the next day, the backup system creates Snapshot B.

Volume
|
+-- Snapshot A
|
+-- Snapshot B

Now the backup application needs to know:

What changed between Snapshot A and Snapshot B?

Instead of reading the complete volume, it can request delta metadata.

Conceptually:

GetMetadataDelta(
    previousSnapshot = Snapshot A,
    currentSnapshot  = Snapshot B
)

The response represents the blocks that changed.

The backup application can then read and transfer those blocks.

Snapshot A
     |
     | compare
     v
Snapshot B
     |
     v
Changed Blocks
     |
     v
Backup Storage

This is the basic idea behind incremental block-level backups.

Initial Backup vs Incremental Backup

CBT becomes easier to understand when you separate the first backup from later backups.

Initial Backup

For the first backup, there is no previous snapshot to compare against.

The backup application may request allocated block information.

Conceptually:

GetMetadataAllocated(Snapshot A)

The result tells the backup system which blocks are allocated.

The application can use that information to build the initial backup.

Incremental Backup

For a later backup, there is a previous snapshot.

For example:

Snapshot A -> Snapshot B

The backup application can request:

GetMetadataDelta(Snapshot A, Snapshot B)

The returned metadata identifies the changes between the two snapshots.

The backup application can then process those blocks instead of treating the entire volume as new data.

Why Block Tracking Is Different From File-Level Backup

A common assumption is that incremental backup means checking which files changed.

CBT works at a lower level.

Consider a database file:

database.db

The file may be several hundred gigabytes.

A small database update could change only a few blocks inside that file.

From a file-level perspective:

database.db = modified

From a block-level perspective:

Block 1000 = unchanged
Block 1001 = unchanged
Block 1002 = changed
Block 1003 = unchanged
Block 1004 = changed

CBT is interested in the second view.

This can be particularly useful for applications such as databases where a small logical change can occur inside a large file.

CBT Does Not Mean Every Backup Becomes Small

This is an important point.

CBT does not guarantee that every incremental backup will contain only a tiny amount of data.

If an application changes most of the volume, the delta can still be large.

For example:

Volume size:       1 TB
Changed data:      5 GB

An incremental backup may be relatively small.

But if the workload changes:

Volume size:       1 TB
Changed data:      700 GB

the backup still has a large amount of data to process.

CBT provides information about the changed blocks. It does not reduce the amount of data that actually changed.

Block Volume Support Matters

The current CBT API is designed for block volumes.

It does not provide the same changed-block tracking capability for file volumes or network file shares.

That means you should not assume that every PersistentVolume automatically benefits from CBT.

Before planning a backup architecture around this feature, check:

  1. Whether your volume is a block volume.

  2. Whether the CSI driver supports the required snapshot functionality.

  3. Whether the CSI driver supports snapshot metadata.

  4. Whether the backup application understands the CBT API.

  5. Whether the required Kubernetes and CSI versions are supported.

The current Beta documentation lists Kubernetes 1.33 as the minimum Kubernetes version and CSI specification 1.10 or newer for this implementation.

What a Backup Application Actually Does

A backup application generally needs to perform more work than simply calling the metadata API.

A simplified workflow looks like this:

1. Create or identify a snapshot
2. Wait until the snapshot is ready
3. Identify the previous backup snapshot
4. Request changed-block metadata
5. Receive block metadata as a stream
6. Read the corresponding data
7. Transfer changed data
8. Store backup metadata
9. Record the snapshot relationship

The streaming part is important.

Large volumes can produce a large amount of metadata, so the API is designed to return metadata through streaming responses rather than requiring everything to be loaded into memory at once.

Why Streaming Matters

Imagine a large volume with millions of tracked blocks.

Returning every block in one huge response could create unnecessary memory pressure.

Instead, a backup application can process the metadata incrementally.

Conceptually:

CSI Driver
   |
   | block metadata
   v
+---------+
| Chunk 1 |
+---------+
     |
     v
+---------+
| Chunk 2 |
+---------+
     |
     v
+---------+
| Chunk 3 |
+---------+
     |
     v
Backup Application

The backup application processes each portion and continues until the stream is complete.

This approach is much more suitable for large datasets.

CBT and Kubernetes VolumeSnapshots

CBT works alongside the existing Kubernetes volume snapshot model.

A VolumeSnapshot represents a point-in-time snapshot of a PersistentVolume through the CSI storage ecosystem.

For example:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: database-snapshot
spec:
  volumeSnapshotClassName: example-snapshot-class
  source:
    persistentVolumeClaimName: database-pvc

The snapshot itself is not the backup application.

It provides the point-in-time reference that can be used by the backup workflow.

CBT adds another piece of information:

Snapshot A
     +
Snapshot B
     |
     v
Changed Blocks

This makes snapshots more useful as reference points for incremental backup systems.

Common Mistakes

Assuming Every CSI Driver Supports CBT

Installing Kubernetes 1.37 does not automatically make every storage driver capable of changed block tracking.

CBT requires support from the CSI/storage side.

Always verify the capabilities of the storage driver you are using.

Treating CBT as a Replacement for Backups

CBT is not a backup system.

It provides changed-block metadata that a backup application can use.

You still need:

  • Backup storage

  • Retention policies

  • Recovery procedures

  • Encryption

  • Access control

  • Monitoring

  • Backup validation

Ignoring Snapshot Relationships

Incremental backup depends on knowing which snapshots are being compared.

For example:

Backup 1 -> Snapshot A
Backup 2 -> Snapshot B
Backup 3 -> Snapshot C

The backup application must maintain the relationship between these points.

Losing the required metadata can make incremental recovery more complicated.

Assuming Changed Blocks Equal Changed Files

They are different concepts.

CBT works at the storage block level.

A database might modify a small section of a large database file. CBT can identify the affected blocks even though the entire file appears modified from a file-system perspective.

Forgetting Recovery Testing

A successful backup job does not automatically mean that the backup is recoverable.

A production backup process should periodically test restoration.

For example:

Production Volume
       |
       v
Snapshot A
       |
       v
Initial Backup
       |
       v
Snapshot B
       |
       v
Incremental Backup
       |
       v
Restore Test
       |
       v
Application Validation

The final step is important.

The backup is useful only if the application can actually recover from it.

Best Practices for Using CBT

Verify Storage Support First

Before designing the backup architecture, confirm that the CSI driver supports:

  • Volume snapshots

  • Snapshot metadata

  • Changed block tracking

  • Required CSI specification version

Do not assume that a Kubernetes cluster upgrade automatically provides storage-driver support.

Keep Snapshot Chains Manageable

Incremental backup systems often depend on relationships between snapshots.

Define a clear retention policy and make sure your backup application knows which snapshots are still required.

Monitor Backup Size

Track the amount of data transferred during each backup.

For example:

Backup #1    420 GB
Backup #2      7 GB
Backup #3     12 GB
Backup #4      9 GB
Backup #5    310 GB

A sudden increase can indicate a workload change or backup-chain behavior that deserves investigation.

Test Restore Regularly

Do not measure backup success only by whether the backup job finishes successfully.

Test:

Backup
  |
  v
Restore
  |
  v
Mount Volume
  |
  v
Start Application
  |
  v
Validate Data

This gives you much more confidence in the recovery process.

Protect Backup Metadata

The changed-block information is useful only when the backup system maintains the information needed to interpret the incremental chain.

Protect backup metadata with the same care as the backup data itself.

Advantages of Changed Block Tracking

Area

Traditional Full Processing

CBT-Based Incremental Processing

Data processed

Large volume

Changed blocks

Network usage

Higher

Potentially lower

Backup window

Longer for large volumes

Potentially shorter

Storage usage

More repeated data

Less repeated data

Database workloads

Can be expensive

Better suited to small block changes

Dependency

Backup system scans/processes volume

Requires CSI/storage support

The actual improvement depends on how much data changes and how efficiently the storage driver and backup application implement the feature.

Limitations to Keep in Mind

CBT is useful, but it is not a universal solution.

The main limitations are:

  • It requires CSI/storage-driver support.

  • Current support focuses on block volumes.

  • Backup software must understand the metadata API.

  • Changed metadata can still be large for heavily modified volumes.

  • Snapshot and backup lifecycle management remains necessary.

  • Restore testing is still required.

  • Storage-provider implementation quality matters.

The feature is currently in Beta, so organizations should evaluate compatibility and operational behavior before making it a core dependency for critical backup workflows.

A Practical Way to Evaluate CBT

If you are considering CBT for a production environment, start with a non-critical workload.

Use a test volume and measure:

1. Full backup size
2. Incremental backup size
3. Backup duration
4. Network traffic
5. Storage I/O
6. Restore duration
7. Restore success rate

Then change different amounts of data and repeat the test.

For example:

Test 1: Change 1% of the volume
Test 2: Change 10% of the volume
Test 3: Change 50% of the volume
Test 4: Change 90% of the volume

This gives you a much better understanding of how CBT behaves with your particular storage system and workload.

Summary

Changed Block Tracking adds a useful capability to the Kubernetes storage ecosystem for incremental backups.

Instead of forcing a backup application to process an entire volume every time, CBT allows a compatible CSI storage system to report allocated blocks and the blocks that changed between snapshots.

The basic workflow is straightforward:

PersistentVolume
      |
      v
Snapshot A
      |
      | application changes data
      v
Snapshot B
      |
      v
GetMetadataDelta
      |
      v
Changed Blocks
      |
      v
Incremental Backup

The important thing to remember is that CBT is not the backup itself. It is a mechanism that gives backup software better information about storage changes.

For large Kubernetes workloads, especially stateful applications, that information can help reduce unnecessary data processing and make incremental backup workflows more efficient.

Before using it in production, verify CSI driver support, Kubernetes compatibility, backup-tool support, snapshot management, and—most importantly—restore behavior.