Extending Kustomize: Basic Templating

Kustomize is my favorite tool to manage the bunch of YAML we need to manage operating our clusters and applications. Since it is integrated in kubectl and oc, it is possible to apply a config to your cluster using one command. Or if you like Gitops, ArgoCD can sync your kustomize layer(s) stored in git automatically.

A project usually consists of a base layer and a few subsidiary layers that are customized versions of the base. Hence its name. There are many so called transformations supported, to customize things such as image tag, namespace, … So building a deploy process based on Kustomize is also easy (change image tag).

Each layer contains a config file Kustomization.yaml and kustomize is the binary you can use to manipulate these, build configs, apply, … As mentioned before, also kubectl and oc have support: apply, diff, …

The problem

We manage OKD infra using Kustomize and one of the many ClusterOperators allows you to manage MachineConfig resources. This allows you to specify for example systemd service units, enable and start them.

Or any system files for that matter. Using the operator assures us system config is identical on all nodes of the same type without using any other config management tool such as Ansible.

Let’s build a simple example MachineConfig:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: myconfig
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,Y29uZmlnOiB2YWx1ZQ==
        filesystem: root
        mode: 420
        path: /etc/myconfig.yaml

If you would push this to Openshift, the Machine operator will update all your worker nodes with a new config file /etc/myconfig.yaml. As you can see the file contents are base64 encoded. Feel free to ‘decrypt’.

Now let’s write a few words about GitOps. One of the promises is: easy accountability of actions using git history, it should be obvious what has changed using git diff. I don’t know about you but a change in a base64 blob is not obvious to me. I have to decode the ‘from’ and ’to’ and diff those to make sense of what has changed. Big fail if you ask me.

I want to add that my problem also had a duplication: we want the same config for masters and workers. But to keep it simple, we only consider the base64 problem.

Solution: Go templating

I have quite some experience with Helm and the Go templating that is used there. So I immediately had the idea to use a simple template to accomplish this.

Some might call this an anti-pattern for Kustomize since it sells as a simpler to use tool as Helm. And here, we make it a bit more complicated again. Might be the reason why I could not find an existing plugin. Luckily it is actually pretty simple to do yourself.

The Kustomize project

This is how our project looks like. A folder that you can name to your liking and probably store in git. Let’s study the file contents.

Kustomization.yaml

The kustomize config file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
- generateMachineConfigs.yaml

We use a generator extension defined in generateMachineConfigs.yaml that will yield a valid YAML resource, in our case of type MachineConfig.

generateMachineConfigs.yaml

apiVersion: kustomize.plugins.dev/v1
kind: GoTemplate
metadata:
  name: goTemplate
template: machineconfig-config.yaml.template
config: generateMC-config.yaml

This defines the generator plugin and its inputs: the template and the config file. Kustomize will look for your plugin using apiVersion and kind, as explained later.

machineconfig-config.yaml.template

Here you finally see a Go template. We use the b64enc internal function to encode the configuration read in from the config file generateMC-config.yaml.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: myconfig
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,{{ $.config | b64enc }}
        filesystem: root
        mode: 420
        path: /etc/myconfig.yaml

The template config, revealing our very exciting config we want on all our worker machines:

config: |
    config: value

Tying it all together: the plugin, location and a kustomize wrapper

Plugin

The plugin is a very simple bash script:

#!/bin/bash

template=$(cat "$1" | yq e .template -)
config=$(cat "$1" | yq e .config -)

gucci -f $config $template

The plugin processes the template using gucci: https://github.com/noqcks/gucci yq is another dependency as you can see. Kustomize includes the generator definition file as first argument of the plugin.

Wrapper and script location

GoTemplate:

function k {
  XDG_CONFIG_HOME="$PWD/PLUGINS/" \
  kustomize build --enable-alpha-plugins --stack-trace $1
}

This wrapper can be used to enrich kustomize with your plugin. If you source this function in your shell first, you can use simply the command k to build a config using kustomize and your plugin.

The script should have execution permissions and have as path relative to the working dir while sourcing the wrapper:

PLUGINS/kustomize/plugin/kustomize.plugins.dev/v1/gotemplate/GoTemplate

The problem#

Solution: Go templating#

The Kustomize project#

Kustomization.yaml#

generateMachineConfigs.yaml#

machineconfig-config.yaml.template#

#

Tying it all together: the plugin, location and a kustomize wrapper#

Plugin#

Wrapper and script location#