Origins: levels and health checks
How origin levels are judged healthy, who probes, how many failures switch, what happens when all are down, failback modes, and node retries.
How it works
A site's origins can be split into levels: level 1 is the primary, the rest are backups. With health checks on, the control plane probes every enabled origin regularly; when every origin of the current level is unhealthy, it switches to the next level. A level switch is a site republication, after which nodes start choosing origins from the new current level.
Nodes also record failures on every origin request: an origin that fails the set number of times within the set time is skipped for a while, and the request is retried on another origin within the retry count and time budget. This is the node's own judgement and does not change the current level.
Decision rules
| Item | Rule | Default |
|---|---|---|
| Probe cycle | Each round the control plane probes the sites probed longest ago | Every 30 s, up to 500 sites per round |
| One probe | GET to the origin's health check path; timeout per probe | 5 s |
| Success | A response with status below 500 | — |
| Failure | 5xx, connection failure, certificate verification failure, timeout | — |
| Unhealthy | Consecutive failures | 3 |
| Healthy again | Consecutive successes | 2 |
| Level health | Healthy if any origin of the level is; unhealthy only if all are; otherwise unknown | — |
| Level switch | When the current level is unhealthy, switch to the next healthy level; if none, to the next level not confirmed down | — |
| All unhealthy | Stay on the current level and raise an alert | — |
| Automatic failback | When a higher level is healthy again, switch back at once | — |
| Manual failback | When a higher level is healthy again, only notify that you can switch back | — |
Health checks need at least two enabled origins on the site.
Who probes
Votes are taken in this order; an origin decided by an earlier source is not probed by later ones:
- Your own online nodes: up to 3 nodes per round; a verdict needs at least 2 votes and a majority;
- dnspup checkpoints (with the dnspup probing plugin): one vote per checkpoint, added to the node votes;
- The control plane probing directly.
So the origin firewall must allow your nodes, the control plane and, with dnspup probing, dnspup's checkpoints. The control plane's probe uses the User-Agent cdnca-control-api/origin-probe.
Insight and logs
- Every level switch and every "can switch back" notice goes to site messages and the notification center ("Origin level switched automatically", "Origin recovered (manual failback available)"); the same notice goes out when every level is unhealthy.
- The origin status card on the site edit page shows the current level and each origin's health (healthy, unhealthy, unknown).
- Health checks judge whether an origin is usable; origin 5xx in the access logs are observations of visitor requests. They differ: an origin failing a few requests is not necessarily judged unhealthy.
Common questions
The origin is down but nothing switched
- It takes 3 consecutive failures, one round every 30 s, so about 90 seconds at best;
- while another origin of the same level is healthy, the level stays healthy;
- when every level is unhealthy, the current level is kept;
- a 4xx (such as 404) counts as success. Make the health check path return 5xx when the origin is broken.
Requests still reach the primary after the switch
A switch is a publication; until a node has the new configuration it uses the old level. With manual failback the primary is not switched back automatically after it recovers; switch back on the site yourself.
Maintaining an origin
Disable the origin in the origin settings, or switch to the backup level by hand, before the maintenance, and enable it afterwards; do not rely on health checks to switch.
Best practice
- Have at least two origins, on different levels;
- make the health check path test only the origin itself, not external services, and return 5xx when it is broken;
- use manual failback if the primary tends to flap.