What is Container Operating System: Immutable, Auto-Updating, Security Minded Fedora CoreOS Intro

Let's talk about requirements for running containers in production.

First of all, you need a server. This could be a hardware or a virtual server.

Then, this server needs to run Linux. There are also Windows Containers, but let's be honest, the majority of container workloads these days are running on Linux.

To run the container on Linux you need a container manager. That could be Podman, Containerd, Docker and many others.

On top of that, the server that runs containers should be hardened. You need to have the latest security patches applied and you need something like SELinux to be enabled in enforcing mode.

Is there anything else? Or is that the full list of requirements?

If we assume that to run containers in production we only need a Linux kernel, a host security mechanism and a container manager, then we don't really need the complete Linux distribution.

What we need instead is a different kind of operating system that is focused entirely on running containerized workloads.

We will call such systems "Container operating systems" and they will be the topic of this article.

The concept of a Container Operating System is not new.

One of the first popular systems of this kind, called CoreOS, appeared back in 2013.

Later CoreOS company was acquired by RedHat. One of the results of these aquisition was the release of two different operating systems.

First appeared the RedHat CoreOS exists only as a special operating system for OpenShift clusters, without any use cases outside of the OpenShift.

But there is also Fedora CoreOS, that has mostly the same features, but is more suitable for general use cases.

Amazon Web Services also released their own Container Operating System, called Bottlerocket.

The most obvious consumer of BottleRocket operating systems are users of AWS managed container platforms like EKS and ECS.

But Amazon made bottlerocket an opensource product, with idea being to make it useful even outside of the AWS environments.

There were few other container operating systems out there, but all of them are either deprecated, like RancherOS, or not yet mature enough.

To be fair, even Fedora CoreOS and Bottlerocket were released only in 2020, so it's also hard to talk about their maturity and stability.

Still, there is a clear trend in a container world to move towards specialised, lightweight operating systems that focus only on running containerised workloads.

So it makes sense to start investing your attention into this idea.

In this article, I am going to show you Fedora CoreOS.

We are going to spin up a test virtual machine and configure it to run a container.

I am going to use the LanguageTool container image and Podman. Check out videos on our channel on building, scanning and promoting images and using Podman to run them.

DevOps consulting: DevOps is a cultural and technological journey. We'll be thrilled to be your guides on any part of this journey. About consulting

Fedora CoreOS Demo

I will show everything on Amazon Web Services.

Don't worry if you are using a different way to spin up your infrastructure - I am not going to use any AWS specific features in this article.

I've already prepared a simple Terraform template to provision an EC2 instance:

If you never used Terraform before, checkout out the free Terraform Lightning Course available on our channel.

Fedora CoreOS is already available on all major cloud providers, so I can simply pull the latest image with the help of the data resource:

data "aws_ami" "fco" {
  most recent = true
  filter {
    name = "name"
    values = ["fedora-coreos-*"]
  }
  owners = ["125523088429"]
}

Then I am using this image when creating a new instance:

ami = data.aws_ami.fco.id

Let me apply the template:

mkdev-fedora-coreos terraform apply

I am going to copy the public IP of the new instance:

And ssh there with the core user:

mkdev-fedora-coreos ssh core@3.120.133.31

Let me switch to the root user:

[core@ip-172-31-42-113 ~]$ sudo su

The first thing I want to do is to install some packages. The problem is that there is no dnf and no yum available. So how do modify the system?

OSTREE

Fedora CoreOS uses something called OSTree.

In simple words, OSTree is a git-like approach to manage your operating system.

Instead of changing your system package per package and file per file, you apply every change as a new filesystem layer.

Every change to the system also requires a reboot.

OStree itself is a platform-agnostic toolkit, that operating system vendors can integrate with their package managers.

That's why there is an rpm-ostree, that is about similar to standard rpm, but it works with the ostree.

ostree and rpm-ostree are components that make CoreOS immutable and makes features like fully automated patching very easy to implement.

Let me try to install tcpdump and see what happens.

rpm-ostree install tcpdump

The output of this command looks very different from your normal dnf install usage.

rpm-ostree creates a whole new filesystem layer and commits it to the filesystem tree.

which tcpdump

Our package is still not available for usage, and rpm-ostree offers us to perform a reboot:

The reason for this is that ostree creates a new bootable filesystem, called a Deployment and the only way to switch to it is to reboot the server.

We can see the current and future deployment if we run rpm-ostree status command.

At the bottom is our current deployment and at the top there is a new one, with tcpdump package layered on top of the previous deployment:

Let me do this and login back to the machine.

systemctl reboot

Now the package is finally available.

If you think that this is the most inconvenient way to modify the operating system, you are right.

The trick here is that Fedora CoreOS was not meant to be used like a normal operating system. You are discouraged from doing any modifications to the system. Instead, for container workloads, you should have immutable infrastructure.

If you need to change something, you create a new server from scratch.

IGNITION

The only moment when you can change the system is during the first boot. This change is performed with the help of Ignition.

If you ever used cloud-init before, then think of Ignition as a new cloud-init.

If you did not use cloud-init, then, in simple words, it's the most minimal configuration management system for the cloud native world.

Ignition files are not meant to be machine-readable.

For humans, there is a special human-readable format that can be converted to ignition configuration files with the help of Fedora CoreOS Configuration Transpiler, fcct. Let's try it out.

I already prepared the configuration file in advance, let me copy it here.

cp ../config.yaml .
bat config.yaml

The first thing we do in this file is create a file ecr-login, inside the storage section. ecr-login is just a tiny snippet to authentication the system against the AWS Elastic Container Registry.

Then, inside systemd section, there is a systemd service definition.

This systemd service logs in to the registry, pulls the latest image version and starts the container with podman.

Fedora CoreOS has both podman and docker pre-installed, as the two most obvious choices for running your containers.

That's the complete system configuration that we need.

APPLY IGNITION

Now we need to path this fcct file to fcct tool to create an ignition config.

Instead of doing it manually, I am going to create another script that will do it for me automatically and pass the result as the user data to the EC2 instance.

#!/bin/bash
ign=$(fcct ./config.yaml)
jq -n --arg ign "$ign" '{"ign":$ign}'

I first invoke fcct tool to create an ignition json and then pass it to jq to generate Terraform-compliant json output.

Now I am adding a new external data source inside the Terraform template to run this script.

data "external" "fcct" {
  program = ["${path.module}/generate-ignition.sh"]
}

and, finally, changing the ec2 instance to use the output of the script.

  user_data = data.external.fcct.result.ign

Let me apply the template and see what happens.

I am logging in to the new server...

mkdev-fedora-coreos ssh core@52.57.233.207

Let's check the status of the container with systemctl status langtool:

Seems like it still pulls the image.

Let me tail the log of the unit:

[root@ip-172-31-32-18 core]# journalctl -u langtool -f

Note that even AWS CLI is running inside the separate container. Fedora CoreOS encourages you to run everything in containers, instead of installing local packages.

Finally, langtool container is running.

And finally let me leave the server and try to access it from the outside world.

We can see a valid output from the langtool API!

Conclusion

As you can see, Fedora CoreOS is a bit different from how you would normally manage your container host.

It forces immutability and it has only essentials to run your containers.

On top of that, it has an automatic updates feature. It can patch itself on regular basis, that would lead to a reboot of the system.

Both immutability and automatic updates are especially useful for embedded systems and bare metal machines. They are also handy when you have Kubernetes clusters that you need to maintain and keep configuration sprawl down to zero.

Container Operating Systems are an important addition to the container world. They can significantly simplify the configuration and management of your Kubernetes clusters, as well as open new posibilities for running regular workloads in containers.

Give them a try and if you used them already, tell about your experiences in the comments!

Here's the same article in video form, so you can listen to it on the go: