Four levels of verification

Why a backup is not enough

While building Zekura we found that a repository that looks perfectly healthy on the source machine can be unreadable on a fresh machine: the local cache masks it. That is why Zekura verifies like a restore, not like the source. Every verification happens on a separate machine, the verifier, never on your server.

The four levels

Level	What	How often
L1 Integrity	the verifier connects with an empty cache, reads back every object of the snapshot and checks all checksums	after every run
L2 Sample	a sample of files is restored and compared by sha256 with the original on the source	daily
L3 Application	a database is loaded into a disposable instance and counted against the run's metrics; Nextcloud is started and scanned; for files you can define your own health check	weekly
L4 Boot	a Proxmox VM is restored as a new VM and booted; the time until the guest agent answers is measured	monthly

On a job's page you see the three dots L1, L2, L3 and for VMs also L4. In the score, a deeper verification weighs more: L1 counts 0.5, L2 0.7, L3 0.9 and L4 1.0.

What you see on a run's page

At the top the duration, the amount of new data, the number of files and the chain record. Below that, per path or component, the number of files, logical and new size and the number of errors. Then the snapshot with manifest, root object and the agent's signature. And finally the verifications, with per level the verdict, the verifier, the start time, the evidence (objects read, duration, engine version) and the chain record.

If a run or a verification fails, that is signed and recorded too. The ledger hides nothing.

The ransomware brakes

• Anomaly detection. If the share of new data in a run suddenly jumps compared with previous runs, that is what encryption by ransomware looks like. Zekura reports it, writes it to the chain and puts a hold on the job: nothing is pruned until you lift the hold, so the snapshots from before the encryption are guaranteed to remain.
• The agent cannot delete. Its key may only append. Pruning is done by a separate worker, and under object lock even those deletions are only delete markers until the retention period is over.
• Delete-marker monitor. Every day Zekura counts the delete markers under every destination and compares that with what the chain accounts for. Unexplained markers raise a critical alert; the data itself is not gone yet.
• 72-hour delay for every change that weakens your protection.

If you want to see how that was tested: lab/attack-sim.sh in the repository plays an attacker with root on a customer host and the agent key, and then recovers the data via the root object from the chain.