Podman 4 – important changes

Where CentOS 8 Stream comes with Podman 3.x, CentOS 9 Stream is shipped with Podman 4.x (currently 4.2.0). I found that information in documentation is often still based on Podman 3 and older, and there are some important changes in Podman 4 which make Podman behave differently from before.

Networking

The biggest change in Podman 4 is the networking stack. Where previous versions of Podman used CNI for networking, Podman 4 uses netavark, accompanied by aardvark-dns for container name resolution. Netavark has a lot of advantages compared to CNI. The biggest advantages, as listed by the developers are:

• Better IPv6 support
• Improved support for containers in multiple networks
• Improved performance

Different behaviour Podman-compose

Podman-compose, the counterpart of Docker compose, was also changed to incorporate new features in Podman. In previous versions of podman-compose, one could use the switch -t 1podfw in the podman-compose command for applications that consisted of multiple containers. This would create a single pod with multiple containers. Within a pod, all services can connect to each other on localhost, even if these services are within different containers. This switch is gone in recent versions and the podman-compose command no longer creates pods. Instead, it adds a single network for containers that belong to one application and counts on Aardvark for providing dns services so containers can connect to the other containers on this network, based on their container name. This means that not only do you need to change your yaml-files to incorporate these changes, you’re also dependent on a properly working dns service of Aardvark.

To explain what I mean with that, imagine a Nextcloud installation, where you combine the Nextcloud container with a MariaDB backend. With the previous podman-compose, where both containers were created in one pod, the MYSQL_HOST parameter in the app configuration would be: MYSQL_HOST=localhost Now in the new podman-compose, this needs to be replaced by: MYSQL_HOST=nextcloud_db, assuming nextcloud_db is the name of your MariaDb container (see at the bottom of this article for an example compose file for Nextcloud)

Caveat for Aardvark DNS service

The Aardvark dns service uses port 53. But what if this port is used by a DNS server on the host or a DNS container? The short answer: it won’t work. The containers will not be able to address each other based on their container names.

I created a bug report for this issue in the Podman github and got the answer that they were aware of this problem and had created a fix in Podman 4.2.0. At the time this version was not yet available in the main Linux releases. In the mean time, my release (CentOS 9 Stream) got updated, so I could implement it in my environment.

To do so, edit the /usr/share/containers/containers.conf file and find the dns_bind_port parameter. Uncomment it and assign a port. I chose port 54 (it’s officially meant for some old Xerox network service which is no longer in use). After saving, you need to restart the podman service (or the machine). After that, the dns service works as expected and containers on the same network can be addressed by their container name.

version: '3.6'
volumes:
  nextcloud:
  db:

services:
  db:
    image: mariadb
    container_name: nextcloud_db
    command: --transaction-isolation=READ-COMMITTED --innodb_read_only_compressed=OFF --binlog-format=ROW
    restart: always
    volumes:
      - /virtual/nextclouddb:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=<root_password>
      - MYSQL_PASSWORD=<user_password>
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud

  app:
    image: nextcloud:25
    container_name: nextcloud_app
    hostname: <my_hostname>
    ports:
      - 9080:80
    depends_on:
      - db
    volumes:
      - /<path-to>/nextcloud:/var/www/html
      - /<path-to>/media/nextcloud:/var/www/html/data
    environment:
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=<user_password>
      - MYSQL_HOST=nextcloud_db
    networks:
      - nextcloud
    restart: always

networks:
  nextcloud: {}