Releases and Health

Health checks, how a release is rolled out, failures, rollback, status, and logs.

Health checks

  • A service with http_port has a health check. A service without it has none and counts as ready as soon as its process is running.
  • With health_check_path the check is an HTTP GET to that path on http_port, sent to each instance every 5 seconds with a 1-second timeout. A status from 200 to 399 passes. The path may include a query string. Without a path the check only opens a TCP connection to the port.
  • The check does not pass through the public endpoint. Its Host header is the internal address of the instance and it has no X-Forwarded-For header, so a framework that rejects unknown hosts must exempt the health path.
  • An instance receives traffic only while it passes the check. After 3 consecutive failures, about 15 seconds, it is taken out of routing; it returns as soon as one check passes. A failing check never restarts an instance.
  • There is no separate liveness check: a process that hangs without exiting is not restarted. Exit on unrecoverable errors, or restart the service through the API.
  • There is no startup delay setting. Checks begin when the container starts and fail until the application answers; a release has 10 minutes in total to become ready.

How a release is rolled out

Every deploy, restart, and rollback creates a release with a fixed image digest, configuration, environment, and secrets.

  • Service without volumes: a new instance starts next to the old one. After it has passed its health check for 10 seconds, traffic moves to it and the old instance is shut down. Several instances are replaced one at a time. A healthy release causes no downtime, and for a short time both versions run side by side.
  • Service with volumes: the old instance is shut down first, and the new one starts only after it has stopped. Two versions never run at the same time and never write to the same volume. The service is unavailable from the shutdown until the new instance has passed its health check for 10 seconds.
  • Service with volumes and several instances: a deploy replaces them one after another. A restart, a rollback, and a change of volume mounts or sizes stop all instances before the new ones start.
  • Adding the first volume to a service, or removing its last one, also stops the old instances before the new ones start.
  • Because the previous release can be started again automatically, keep the data on a volume readable by the previous version until the new release is confirmed healthy.

Failed releases and crashes

  • While a release is rolled out it fails at once if one of its instances exits, is killed for exceeding its memory, cannot download the image, or cannot be started. It also fails when it is not ready within 10 minutes. There is no automatic retry: fix the cause and deploy again.
  • If an earlier release was active, it keeps serving (service without volumes) or is started again automatically (service with volumes). The automatic restore appears as a deployment with action recover, and the service status is degraded until it or a later deployment succeeds.
  • If there was no earlier release, the failed instances are stopped and the service status becomes failed or stopped.
  • Once a release is active, an instance whose process exits is restarted automatically. When one instance has restarted 10 times, all instances of the service are stopped and its status becomes stopped. Deploy again to start it; new instances count from zero.
  • A failing health check alone never stops or restarts a running service.

Rollback, restart, stop

  • Rollback deploys an earlier release again with its image digest, configuration, environment, and secrets. Current endpoints are kept, and volume data is not rolled back.
  • Rollback is available for the releases marked can_rollback among the 20 most recent deployments: those that were active at some time.
  • Restart starts fresh instances of the current release. Saved configuration changes that were not deployed are not applied.
  • Stop shuts down all instances and cancels a release in progress. Volumes are kept. Start the service again with deploy or restart.
  • Deleting a service removes its instances, endpoints and linked CDN pull zones and keeps its volumes. Deleting an application removes everything in it, including volume data and CDN pull zones linked to its endpoints. Storage Zones are kept.

Status and logs

  • Service status: draft (never deployed), deploying, running, degraded (the latest deployment failed and the previous release serves or is being restored), failed (the latest deployment failed and nothing serves), stopping, stopped, stop_failed, deleting, delete_failed.
  • Deployment status: queued, deploying, and health_checking are in progress; active serves traffic; superseded was active and has been replaced; failed and cancelled are final.
  • Every deployment has a timeline (progress) and, when it fails, a reason (failure), both in the language of the request.
  • Logs are the standard output and standard error of the container. Write logs there; log files inside the container are not collected.
  • The API returns the last 1000 lines, at most 256 KiB, of one instance, for its current run or for the run before its last restart.
  • Output exists only while its instance exists. A release, restart, stop, or rollback replaces instances, and their earlier output is gone. For history, send logs to your own collector over HTTPS.
  • Values of the service's secrets are replaced with [REDACTED] in returned logs.
  • The service response includes the current month's CPU time, memory use, outgoing traffic, and cost. Live CPU, memory, and bandwidth graphs are available in the dashboard only.

Exact request and response fields, limits, and error responses are in the Management API reference.