Skip to content

State, change, or data loss when containers restart

Why state loss occurs

Containers are designed to be ephemeral instances of services. This means that, by design, they're intended to be periodically stopped and restarted. Given the design of how state is managed in Docker, services that are not configured to handle state can be subject to data loss when containers restart.

The solution to persisting container state is by storing data in volumes mounted in the service configuration.

This article discusses recommendations when (1) handling state in containers and (2) deploying updates to application services.

Use volumes to preserve state

By creating Docker volumes and mounting them to services, data written to the mounted volume directory is retained on disk and included in the MedStack Control's backup system.

Data written to Docker volumes will persist beyond the lifecycle of the containers that mount them. When new containers are deployed to a node whose disk contains the volume state, those containers will also be able mount the volume and thus the data stored in that volume.

Deploy updates with new images

When updating applications, the best practice is to lean on CI/CD pipelines to perform the following sequence of events. If not automated in pipelines, this process could also be done manually.

  1. Merge changes/updates to code to the master branch of the application.
  2. Build a new container image of the application.
  3. Push the new container image to the container image registry.
  4. Update the service in MedStack Control to use the new container image.

Services running in MedStack Control do not automatically deploy new container images pushed to the registry. Deploying new container images to MedStack Control requires the deliberate action of a user to either (A) restart a service that is configured to use the latest container image, or (B) update the image tag or digest in the service configuration to roll out the updated application.

We recommend the latter, that is, using image digests to control what's deployed in production.

Exceptions

There are some exceptions when modifying applications where code may want to be managed in a volume. For example, when managing a Wordpress application, or applications that have a similar structure.

In Docker Hub's Wordpress container image, components and themes are stored in subfolders within the /var/www/html directory. If mounted to a Docker volume, changes to data inside this folder would be preserved. So, if the Wordpress container is destroyed, new Wordpress containers would still be able to mount the volume and access the data within.

One notable limitation by this method of updating services is that volume data is not synchronized between nodes. This means that if a stateful service runs containers on two or more nodes, there is a possibility that the data mounted to each container is different from other containers of the service.