On the day, i.e Saturday 12 July, I had a talk scheduled on "Container Runtimes."

It's been a while that I wanted to do this talk and dive into the lower realms of container management. We often talk about and demo container orchestration using high level tools, while not giving much attention to the underlying technologies. Therefore, my talk was like a primer on Linux namespaces and cgroups, followed by an introduction to container engines, mentioning the low-level and high-level ones.

Before my talk, I had a very interesting discussion with Renghen. As he was going through my slide deck, he noticed that I mentioned cgroups and he pointed out that all the modern features that we enjoy in container orchestration have been enabled thanks to cgroups v2 which was a complete overhaul of version 1, not just an update. Cgroup v2 provides more granular control over resource allocation, brought rootless features, simplifies resource management under one single unified hierarchy, among other improvements. I took note of that and included it when I spoke about control groups.

I shared a list of current actively developed container runtimes, among which I highlighted youki — a container runtime written in Rust. I co-maintain the youki package on the openSUSE Build Service and I wanted to share how I am using youki as an experiment to dive deeper and learn more about the low-level workings of container management.

Youki creates and starts a container but you will need a higher level container engine to actually manage your container resources. For that, I mentioned Podman. In fact, I asked how many in the room knew about Podman and to my surprise many raised their hands. It turns out that Podman is being heavily favoured over Docker by the development team at Accenture — something I was very glad to hear as a proponent of Podman since its inception. I did an introduction to Podman at the Middlesex University in 2019, the same year the stable version of Podman was released. Since then, Podman has always been my demo tool when talking about containers.

I happily shared a few SUSE geeko plushies with the attendees that were able to answer questions that I asked about container runtimes at the end of my presentation.

Note that Mauritius is far from "everywhere" else and connectivity issues are nothing new.

ish@coffee-bar:~> podman pull docker.io/library/nextcloud
Trying to pull docker.io/library/nextcloud...
Getting image source signatures
Copying blob e0276193a084 done  
Copying blob eb2d00c10344 done  
Copying blob 3a60f364b0c5 done  
Copying blob e0d3d1244592 done  
Copying blob f54006e0dc29 done  
Copying blob 8559a31e96f4 done  
Copying blob b22875b95a2a done  
Copying blob f65316e96b10 done  
Copying blob 3e309988c00b done  
Copying blob 0c78caf16ec3 [======================================] 13.1MiB / 13.2MiB
Copying blob 4fc30aae7ee5 done  
Copying blob 37b016cacdc6 done  
Copying blob fd56bf3cc539 done  
Copying blob e3cd35f544b5 done  
Copying blob 467fea8f6f80 done  
Copying blob 0dc7444f9282 done  
Copying blob 547ae6684264 done  
Copying blob 3139b6de5be7 done  
Copying blob 00e77223b529 done  
Copying blob 2b37e3b3a856 done  
  read tcp 192.168.100.6:33536->104.18.124.25:443: read: connection reset by peer
Error: error pulling image "docker.io/library/nextcloud": unable to pull docker.io/library/nextcloud: unable to pull image: Error writing blob: error storing blob to file "/var/tmp/storage720474498/10": read tcp 192.168.100.6:33536->104.18.124.25:443: read: connection reset by peer

Podman does not retry to copy the image in case of failures.

A few days ago there was a suggestion to implement a similar feature in Podman that is present in Buildah, which provides the "image copy retry" functionality.

For the curious, the implementation of retryCopyImage in Buildah can be seen here.

Container images that are pulled by Buildah are stored in the local repository which can also by accessed by Podman, so that's an advantage. I tried pulling the Nextcloud container image using Buildah and it completed successfully.

ish@coffee-bar:~> buildah pull nextcloud
Getting image source signatures
Copying blob e0d3d1244592 done  
Copying blob 8559a31e96f4 done  
Copying blob eb2d00c10344 done  
Copying blob 3a60f364b0c5 done  
Copying blob f54006e0dc29 done  
Copying blob e0276193a084 done  
Copying blob f65316e96b10 done  
Copying blob b22875b95a2a done  
Copying blob 3e309988c00b done  
Copying blob 0c78caf16ec3 done  
Copying blob 4fc30aae7ee5 done  
Copying blob 37b016cacdc6 done  
Copying blob e3cd35f544b5 done  
Copying blob fd56bf3cc539 done  
Copying blob 467fea8f6f80 done  
Copying blob 0dc7444f9282 done  
Copying blob 547ae6684264 done  
Copying blob 2b37e3b3a856 done  
Copying blob 00e77223b529 done  
Copying blob 3139b6de5be7 done  
Copying config 327476ebe3 done  
Writing manifest to image destination
Storing signatures
327476ebe3280c7b570d8463edd136956eab120959976b643cb7dbfaa73f98c1

Now, the downloaded container image is also accessible by Podman.

ish@coffee-bar:~> podman images
REPOSITORY                     TAG      IMAGE ID       CREATED        SIZE
docker.io/libreoffice/online   latest   0586fecfa3c1   28 hours ago   2.84 GB
docker.io/library/nextcloud    latest   327476ebe328   3 days ago     774 MB

To conclude, while we are waiting that a retryCopyImage function is available in Podman, we can use Buildah to pull container images that are troublesome due to network issues.

While I am not a fan of alias docker=podman I believe Estu added that part in his slide below to make the Podman transition a bit smoother for users already familiar with Docker commands.

Note that the latest release of openSUSE Leap, i.e version 15.2, comes with support for Podman through the libcontainers-common package.

Previously, if one needed to experiment with Podman and/or deploy in production then openSUSE Tumbleweed, Kubic or MicroOS were the supported choices. Now, one may deploy Podman containers on their existing Leap infrastructure (after upgrading to the latest version 15.2).

All sessions videos from the openSUSE Virtual Summit 2020 are available on the openSUSE YouTube channel.

One couldn't just reboot the system every time a simple program was needed - e.g using Vi to edit a YAML config or using database client utils for a rapid inspection.

But now we have a toolbox script available in MicroOS & Kubic that allows us to launch a container and use debugging utilities that can interact with the host. The script is written in Bash and it is based on CoreOS/toolbox.

Thorsten Kukuk, SUSE Engineer & Architect (SLES / MicroOS), announced the availability of the toolbox container on the openSUSE Kubic mailing list in October.

The toolbox script resides in the system $PATH and thus can be called directly. On its first execution, the script will pull the toolbox container from the openSUSE Container Registry.

microos:~ # toolbox
Trying to pull registry.opensuse.org/opensuse/toolbox...
Getting image source signatures
Copying blob 92a022f7c0d8 [>-------------------------------------] 2.8MiB / 122.9MiB
Copying blob f0c8107beec9 [==>-----------------------------------] 3.0MiB / 36.4MiB

The host root filesystem can be accessed at /media/root/ and since the toolbox container is spawned as the root user, the files will be modifiable.

toolbox:/ # ls /media/root/
bin   dev  home  lib64	opt   root  sbin     srv  tmp  var
boot  etc  lib	 mnt	proc  run   selinux  sys  usr

Non-root toolbox containers will have read-only access to the host files, unless the --root flag is provided.

ish@microos:~> toolbox --root

Next, use zypper to install the required utilities and interact with the host and other containers.

Exiting toolbox will stop and the container and next time the script is called, the container is started again. Therefore, previously installed utilities remain available.

Each toolbox container is named by appending the username calling it, e.g toolbox-ish.

microos:~ # podman ps --format "{{.ID}} {{.Status}} {{.Names}}"
7f61fba95487 Up 10 minutes ago toolbox-ish
00ada606ec4f Up 33 minutes ago toolbox-root

As a quick response, I explained him very breifly about "container networking" and pointed him to the Kubernetes documentation on IP allocation. I do realize though, that most of the times, documenation can be lengthy and {boring}, and that you would just want a simple article or blog post that clears your doubts.

Tell me about Podman networks

Rootless containers (i.e containers started using Podman as a regular user) do not obtain an IP address. Podman uses slirp4netns to allow Internet connectivity inside the container.

Communication with a rootless container is achieved by mapping the container ports to the host, e.g using -p 8080:80 to map a webserver port 80 to the host on port port 8080.

$ podman run -dt --name webserver -p 8080:80 nginx
$ curl http://localhost:8080

Therefore, two rootless containers can communicate over their published ports on the host. Let's experiment this by starting an openSUSE Leap container and installing the telnet package.

$ podman run -dt --name leap leap
$ podman exec -it leap bash

4a0f95e011b9:/ # zypper in telnet

We run ip a s on the host to find its IP address. Say the IP address is 192.168.100.8. Now, from within Leap container let's telnet port 8080 over the host IP.

4a0f95e011b9:/ # telnet 192.168.100.10 8080
Trying 192.168.100.10...
Connected to 192.168.100.10.
Escape character is '^]'.

The connection went through successfully, meaning from the Leap container we've been able to access the Nginx container through it's mapped port on the host.

This same experiment can be repeated using two different pods, say you have a pod that contains your web services and another pod that contains your databases.

$ podman pod create --name webservice -p 8080:80
$ podman run -dt --name webserver --pod webservice nginx

$ podman pod create --name db -p 3306:3306
$ podman run -dt --name mariadb --pod db -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" mariadb

The Nginx container will be able to reach the MariaDB database over 192.168.100.10:3306 as the same port is mapped on the host.

Ideally, these two containers could have been created in the same pod and therefore share the same network space. Then, the Nginx container would reach the database over localhost:3306 easily.

I used the above Nginx/MariaDB example to explain rootless inter-Pod communication, which was the question that was asked to me initially.

What about rootfull containers?

Rootfull containers are those that are created using Podman with root privileges, either by the root user itself or using sudo privilege.

Containers created using Podman with root privileges obtain an IP address. Podman then uses the Container Network Interfec (CNI) instead of slirp4netns for networking provisioning.

Details about the network subnet is found in the CNI config file.

$ cat /etc/cni/net.d/87-podman-bridge.conflist

{
    "cniVersion": "0.3.0",
    "name": "podman",
    "plugins": [
      {
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "subnet": "10.88.0.0/16",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ]
        }
      },
      {
        "type": "portmap",
        "capabilities": {
          "portMappings": true
        }
      }
    ]
}

So, let's start a container with root privileges and see.

$ sudo podman run -dt --name db postgres

$ sudo podman inspect -f "{{.NetworkSettings.IPAddress}}" db
10.88.0.30

The podman inspect ... command returns the container's IP address and the same is ranged within the subnet specified in the CNI config. The PostgreSQL database can be accessed over 10.88.0.30:5432 from the host or from within any other container started using root privileges.

$ telnet 10.88.0.30 5432
Trying 10.88.0.30...
Connected to 10.88.0.30.
Escape character is '^]'.