Upgrading containerd from 1.7 to 2.x? Read this first!

Posted on by

Since Kubernetes 1.20 deprecated Dockershim (and removed it entirely in Kubernetes 1.24), containerd has become the de facto container runtime for Kubernetes environments. HCL Connections Component Pack runs on Kubernetes, making containerd a critical dependency for Connections deployments.

Although HCL generally follows the Kubernetes release lifecycle in its Component Pack (CR) releases, containerd 1.7 remained the highest supported version for a long time. This changed with HCL Connections 8 CR14, which introduced support for Kubernetes 1.35 and, consequently, containerd 2.2.x. At the same time, support for containerd 1.7.x was dropped for Kubernetes 1.35 deployments.

CR14 continues to support Kubernetes 1.34 and containerd 1.7.29, allowing administrators to postpone the containerd upgrade if desired. However, this will likely only delay the inevitable until a future release. If you are planning to upgrade your Connections Component Pack environment to a release that requires containerd 2.x, there is an important change you should be aware of.

The change

Containerd 2.x no longer sets the maximum number of open files (LimitNOFILE) to infinity by default. Instead, it inherits the systemd defaults:

Soft limit: 1,024
Hard limit: 524,288

This article provides a good overview of the change and what it entails.

While these values may be sufficient for many workloads, heavily used Kubernetes environments can encounter issues when applications require a large number of simultaneously open file descriptors.

The problem we encountered

On the next workday after our upgrade, we started seeing the following error in our MongoDB pods:

EMFILE: Too many open files

As a result, MongoDB became unstable and crashed repeatedly.

Whether you encounter this issue depends largely on the size and workload characteristics of your environment. In our case, the environment serves approximately 45.000 users and experiences heavy usage of Activities Plus / Huddo Boards, resulting in significant load on the MongoDB backend.

Our solution

We chose to restore the previous behavior by setting the open file limit back to infinity for the containerd service.

On Red Hat Enterprise Linux, this can be done by creating a systemd override:

cat > /etc/systemd/system/containerd.service.d/limits.conf <<EOF
[Service]
LimitNOFILE=infinity
EOF

systemctl daemon-reload
systemctl restart containerd

After restarting containerd, any affected workloads should also be restarted so they can inherit the new limits. For example:

kubectl rollout restart sts mongo7 -n connections

Verifying the open file limit

You can verify the active limit on a Kubernetes node running containerd:

systemctl show containerd -p LimitNOFILE

Expected output after applying the override:

LimitNOFILE=infinity

You can also verify the limit of the running containerd process:

cat /proc/$(pidof containerd)/limits | grep "Max open files"

Example output:

Max open files            1073741816           1073741816           files

To check the limits seen inside a pod:

kubectl exec -it <pod> -- cat /proc/self/limits | grep "Max open files"

Or, for MongoDB pods specifically:

kubectl exec -it mongo7-0 -n connections -- cat /proc/self/limits | grep "Max open files"

If the limit was inherited correctly, both the soft and hard limits should show as a ridiculously large number, like 1073741816.

Final Thoughts

If you are upgrading from containerd 1.7 to 2.x as part of a Kubernetes and HCL Connections Component Pack upgrade, verify your open file limits before going live. Environments with high user counts or database-intensive workloads may be particularly vulnerable to this change.

Special thanks to my colleague Jan Flipsen, who quickly identified the issue and found the solution this morning after our upgrade to CR14 last night.