SD Card Longevity and Docker/Kubernetes

Docker as a technology has enabled radical changes in the world of applications; how apps are built, shipped, and supported. Containers are now the de-facto way modern applications are adopted in almost every industry and every use case.

This change enabled "edge computing" to gain rapid momentum and is the underlying deployment mechanism for Industry 4 / IOT application workloads.

Docker was initially designed to run on servers rather than lightweight devices. While it works perfectly fine on all kinds of hardware, there are unintended negative consequences when SD cards are used as the primary storage mechanism on hosts.

SD cards (well, memory cells in SD cards) have a limited number of write cycles before they become read-only; for most cards, this is 10,000 times (per cell), but there are enterprise-class ($$) cards that offer more, and pro class cards that include software features to better distribute writes across cells (TRIM / Wear Leveling). 10,000 seems like a lot, and in the photography world (where SD cards originated), it is a lot, but in the IOPS-heavy world of Linux disk IO, 10,000 can run out quickly.

When using SD Cards with containers (Docker/Podman), you need to consider all of the elements that would cause a write to occur:

• stdin/stdout/stderr - logs for containers are written to a journal file that exists on the Host FS; this means a file is being constantly written to (and deleted/recreated whenever the container is recreated)
• Image files - when pulling an image, the TAR file for each layer is downloaded to the docker host, then extracted before being placed into the image repository. This causes temporary writes for the downloaded files and then writes for the extracts.
• Container run-time FS - the docker overlay FS sets up a redirected file system for each container based on the container image (always read-only) and a writeable redirected location for any run time changes. This runtime location is destroyed and recreated whenever the container is instantiated.
• Writes from actual data being persisted - any data stored in a persistent container volume on the host has its read/write IOPS mix. Data stored temporarily (such as an MQTT queue) would significantly impact the IO write counter.
• Orchestrators, such as Kubernetes, generate disk IO; you should also be aware. Sure, lightweight Kubernetes runtimes exist, such as MicroK8s, K0s, and K3s, that use different DB engines, but still, there is a write overhead of maintaining orchestrator state in a DB. etcd as the Kubernetes database is IOPS heavy and would have a profoundly negative impact on SD card lifetime.

So, what can you do to extend the lifespan of your SD Card by removing unneeded writes?

A couple of things.

1) disable noatime - atime (last accessed time) writes a timestamp into the FS every time a file is accessed. This is unnecessary for most systems, so it can be disabled. To disable, edit /etc/fstab, and add noatime to the options as per the pic below.

2) Adjust the threshold at which the Linux Kernel swaps RAM into Disk - By default Linux Swaps to disk when memory gets below 60% free. You can increase this threshold by adjusting the setting vm.swappiness to 10%  (mem must get below 10% free before swapping - to be aggressive and keep everything in memory, set this to 1). To make this change, edit /etc/sysctl.conf and add vm.swappiness=10, then save and reboot.

3) Redirect docker temporary image downloads to tmpfs - by default, image files downloaded to a docker host are pre-staged into the /var/lib/docker/tmp directory before being extracted and loaded into the image repo. This is an inefficient use of the SD card, so a dedicated TMPFS mount should be added and mapped to this volume. Edit fstab and add a mount as per the below (note if you need more than 1GB temp storage space, increase this, but note it uses RAM):

4) Consider using TMPFS for your container volumes if they do not need to persist data beyond reboots (i.e., the container volume acts as a temporary cache).   You can pre-create the named volume in Portainer and then use this named volume when deploying your container. Note that the size you set here uses physical RAM.

5) Use a Docker logging driver that redirects the logs of your containers to a central logging system, removing the disk IO load of the Docker built-in JSON logging driver. You can also look to use the JournalD driver, which writes container logs to the host logfile, and then configure the host with a Log2RAM driver, keeping logs purely in RAM.

6) Final suggestion is to purchase an SD card that is a LOT larger than you need. Why? Because each memory CELL can only be rewritten a certain number of times, so to increase longevity get more cells... and that means larger capacity. Writes are spread across all cells on the SD card, so more cells, less writes per individual cell. Be aware that with MLC cards, it's not a 1:1 cell to MB mapping, but it's a good rule of thumb that a 64GB SD card has many more cells than a 16GB one.

In Summary...

Two of the mitigations recommended above offset SD card longevity with RAM utilization, so if you are RAM constrained, these won't apply. If you are using a resource constrained device AND have to use SD cards, just be aware that the cards will likely expire much sooner than you might think.

I hope this helps you extend the life of your SD Cards when used in the field, at the edge, or at home on RPi's.

Neil

To read more about Portainer's solution for IOT/IIOT, click here