OKD is the open source equivalent of RedHat’s Openshift product. Openshift is an opiniated patched Kubernetes flavour, marketing itself as more developer oriented and more secure. I always forget what the acronym OKD stands for, so Google it yourself it you like to know.

Changing jobs about a year ago, I inherited some long running clusters and the internal work flow using and operating the platforms. I had about 4 years of experience operating and developing on ‘vanilla’ Kubernetes before.

Developer oriented

They try to make it as easy as possible for developers to start using the platform. I don’t have experience with some of the simplifying stuff but I believe you can start from examples (templates) if you are in the business of developing services in common language frameworks.

And there is some integration in the command line tool oc and the web interface to easily start a new project using a template. But we are not afraid (anymore) of a bunch of YAML, so lets skip that stuff.

Console web interface

Most people like to click things in some fancy user interface. The Openshift web panel is far superior to the Kubernetes dashboard. Which is no surprise of course, if RedHat wants to create value for their customers.

A nice feature is the extensibility and how nicely it integrates with RedHat operators. If you install Redhat Virtualization, for example, a few minutes later you can create VMs, view VNC consoles and live migrate your VMs. All just a mouse click away, pretty neat.

Container build pipeline for free

More interestingly, OKD provides a container registry that you can easily install (yet another YAML declaration). Also some extra API resources are introduced such as BuildConfig, Image and ImageStream. These allow you to manage builds and container images just like any other Kubernetes resource. Think about the nice fine grained RBAC authorization system you get for free for example.

BuildConfig is a bit a special and powerful one: you can trigger builds in Openshift using the command line:

oc start-build <buildconfig_name>

So pretty neat, you don’t need some devops guy to write you some build pipeline, Openshift provides it (albeit basic) for you.

Openshift templates

Managing Kubernetes resources can be a burden (YAML hell), I don’t need to tell you. There are many solutions, each with their own focus and solutions. The popular ‘generalized’ open source ones I know best are Helm and Kustomize.

Openshift offers its own templating engine and an easy way way to use it using their commandline tool oc. Compared to Helm the templating is very simple and offers only the basic feature: define variables and possible default values.

Very simple and effective if you want to specify different environments with customized definitions.

Integration Authentication providers

A lot of companies have some kind of LDAP service running (AD, FreeIPA, …) to enable authentication of services throughout their organization.

In OKD or Openshift you can integrate with your LDAP by simply applying some YAML: https://docs.okd.io/4.10/authentication/identity_providers/configuring-ldap-identity-provider.html

This is also possible in vanilla k8s but is far harder to accomplish.

My view about the developer stuff

I am a big fan of the web interface. Me, I like to use the console tools of course but most of my colleagues only touch Openshift using the web interface.

Unfortunately, if I look at some of the developer offerings today, it seems to become obsolete. Openshift has to follow trends such as GitOps and for many organizations YAML is maybe not that scary anymore?

In the new CI/CD offering they push, Tekton, the BuildConfig concept for example, is not used. Instead a Tekton Task does about the same as the Openshift ‘black box’: Use Buildah to create you a container image based on your Dockerfile. And BuildConfig has quite some ‘obscure’ options that I would never use but make it look quite complicated.

My opinion: everything evolves so fast, no surprise there. Openshift does its best to support even their initial solutions that might be less relevant today (community gone), that you as a newcomer can waste a lot of time following outdated documentation for example.

But I am not against support of course, and they still offer efficient workflows, for examples managing templates and using BuildConfig concept to build your CI/CD pipeline using Jenkins for example.

Security

This is an impressive undertaking from Redhat. They managed to introduce quite some security enhancements out of the box. Maybe it is possible to do all this with vanilla k8s and/or well trained developers, but nothing can beat the ease of a solution automatically installed, out of the box.

UIDs, GIDs, que?

Maybe you know this, maybe you don’t. But even if the users do know, I bet many Kubernetes cluster in the wild are full of root processes, or are using another ‘risky’ user account. With risky I mean: you probably don’t like a hacker that manages to manipulate your outdated Python/PHP/Ruby/Go, make your choice, in such way he has shell access in your container and gets root (or equivalent) access to your systems. There might be some container restrictions left, but having a hacker with root access in your system will lead to trouble such as mysterious wasting of CPU cycles, maliciously encrypted data, sooner or later. It is so easy to apply some YAML copy pasted from the internet or use a Helm chart that nobody cares anymore how the processes exactly run on the physical system.

To developers: you really should use the Docker USER directive somewhere at the bottom of your Dockerfile to ensure your new service does run as a restricted user. But how do you know the chosen UID does not collide with some system user that can sudo without password?

Of course your Kubernetes savvy colleague might enforce you to do this using a concept such as Admission Controller: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ and maybe the sysadmin carefully manages the system users and sudo rights on the underlying machines. But that is pretty advanced stuff if you ask me, it definitely does not come out of the box. If you don’t agree and know a easy to implement solution, I am all ears :-)

Openshift does it differently, effectively taking this burden away from the developer and platform engineer: The basic ideas are:

  • Container IDs are different from system IDs
  • Openshift allocates a specific range of allowed system IDs per project (~ namespace).
  • PODs automatically choose the first ID from this list
  • SELinux provides extra security preventing f.e. data from one project to be accessed by another project unless the so called MCS parameters are specified. For which you need privileged SCC permissions.

It is a lot more complicated than this key points. If you are interested and need some literature to fill your afternoon, here you go and have fun: https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids

If a container wants the ‘dangerous’ default Kubernetes behaviour and/or the root user, the ServiceAccount needs special permissions Using Security Context Constraints or SCC.

The SCC concept

SCC is comparable to PodSecurityPolicies in the Kubernetes world. By default containers get the restricted SCC that forbids many things, such as allocating host ports, specifying system UID/GID values etc.

Only an administrator can set these permissions so developers can’t to dangerous stuff by accident.

Air gapped setups

Air gapped means your infrastructure cannot access the internet directly. Container images are downloaded from an internal registry or proxy and so on.

Again kudos to RedHat, the documentation and support to run your k8s disconnected from the internet is great!

My experience with vanilla k8s is that this is difficult to achieve, it will definitely involve a lot more work. How will you deploy your fancy k8s cluster if docker, quay, … is down?

We like security

In my opinion the UID/GID solution is brilliant and I don’t know of a similar solution for vanilla Kubernetes. Do you?

The SCC concept is less novel and also exists in vanilla Kubernetes: https://kubernetes.io/docs/concepts/security/pod-security-policy/ PSP is deprecated now upstream, so I am curious to see if Openshift is also gonna move to something else or stick to SCC in the future.

And air gapped infra is just common sense if you have any consideration for security.

Automatic updates

This is probably the nicest feature: updates are just one click or console command away. This updates include everything from the ground up: OS, kernel, OKD, operators, …

Openshift will take care of the rolling reboots of your masters and workers. If your services are deployed in a ‘cloud native way’, nobody will notice.

Good luck with your k8s upgrades. How did the transition from Docker to Containerd CRI go? Guess you went straight to public cloud? Can’t blame you.

To be fair, the upgrade process of Openshift is not as perfect and easy as in above sales pitch but it makes my life easier. The nuance is maybe for another blog post. And the upgrade only works smooth because the architecture stays identical. A switch of container runtime, is of course always a disruptive process.

Operators, operators and operators

It is clear, Operators are a hype nowadays. Every respectable software component will have its own Kubernetes Operator to try to be relevant in the huge market space k8s offers.

Nothing against hypes, by the way. Openshift evolved from version 3 to 4 over the years and the big change is also: Operators everywhere.

You can literally manage everything in Openshift using some YAML that is processed by some operator running its reconciliation loop to apply your desired state to the systems. That also includes system services, files, kernel settings, mounts, …

Achieving ‘infra as code’ becomes a simple job. The Red Hat OpenShift GitOps operator, which is basically ArgoCD does the rest together with your favourite Git provider and work flows to accomplisch GitOps.

The downside

Load on the masters, etcd and the underlying infrastructure storage. In my plain vanilla k8s days, I always used to argue that VM infra for the masters is the most cost effective solution. And maybe even 1 master is sufficient if your cloud infrastructure is reliable and offers a HA solution. Kubernetes can perfectly tolerate a short downtime of its control plane as long is it does not happen often.

But forget about it if you want to host OKD. The platform comes with a multitude of so called ClusterOperators that you have to run. Consequently the memory footprint of an empty platform is huge and the storage requirements are high because of the many Kubernetes API calls that makes those operators tick.

So you really need a very fast cloud platform or bare metal masters and at least 3 instances for even small clusters. If you have budget limitations, you can consider running a minimal cluster of 3 nodes playing both master and worker roles. You just have to live with the fact that user services are running a bit close to the etcd instances containing your full cluster state including secrets.