Mapping Chart Versions to Cluster Versions using Helmfile

Helmfile is a wrapper around the helm command that helps you manage multiple helm charts with a single command. It encourages DRY and allows you to pass different values based on environments that you define. If you’ve never heard of helmfile, I recommend having a look at this excellent open-source tool.

I’d like to share a trick I use to map versions using another variable as a lookup index. This article assumes you are already familiar with the basics of helmfile.

The Problem

Some software is closely tied to the version of Kubernetes. Usually such software interacts with the Kubernetes API and depends on specific behavior in Kubernetes that may change over time. Some examples are Cluster Autoscaler, Kube Proxy, CoreDNS, and AWS vpc cni.

This presents a problem if you are using helmfile to manage multiple clusters running different versions of Kubernetes. You cannot hardcode the chart or image versions in helmfile.yaml, since those versions would be used for all clusters.

You could use helmfile’s Go templates features to render the correct versions for each environment using a seperate values.yaml file. The problem here is that over time you have various versions scattered throughout many different values files. This becomes confusing and difficult to keep track of.

The Solution

One solution I found is to define a yaml map containing variables that change according to version of Kubernetes. The keys of the map are the version of Kubernetes.

Setup

First, let’s look at an example environments definition in helmfile.yaml:

# helmfile.yaml

environments:
  staging-cluster-1:
    values:
      - values/global/versions.yaml
      - values/staging/values.yaml
      - clusterVersion: "1.24"
  staging-cluster-2:
    values:
      - values/global/versions.yaml
      - values/staging/values.yaml
      - clusterVersion: "1.23"
  production-cluster-1:
    values:
      - values/global/versions.yaml
      - values/production/values.yaml
      - clusterVersion: "1.24"
  production-cluster-2:
    values:
      - values/global/versions.yaml
      - values/production/values.yaml
      - clusterVersion: "1.23"

---

releases:
  ...

We are defining an environment for each Kubernetes cluster. Notice for each environment, in addition to an environment-specific values file, we are passing the same values/global/versions.yaml values file. Also note that the variable clusterVersion is defined for each environment, and is different across environments.

Let’s have a look at the global versions value file:

# values/global/versions.yaml


clusterVersionIndex:
  "1.23":
    clusterAutoscaler:
      chartVersion: 9.24.0
      image:
        tag: v1.23.0
    awsCoreDNS:
      image:
        tag: v1.8.7-eksbuild.3
      chartVersion: 0.187.2
    awsVpcCni:
      chartVersion: 1.2.0
    awsKubeProxy:
      chartVersion: 1.23.0
  "1.24":
    clusterAutoscaler:
      chartVersion: 9.27.0
      image:
        tag: v1.24.3
    awsVpcCni:
      chartVersion: 1.13.4
    awsKubeProxy:
      chartVersion: 1.24.10
    awsCoreDNS:
      chartVersion: 0.193.0

We define a yaml map called clusterVersionIndex, with the index of each sub-map being a Kubernetes version. The version number is quoted, to force a string value instead of a number, which would causes problems with yaml parsing.

Finally, let’s go back to the helmfile.yaml and have a look how we can use the clusterVersionIndex map together with the clusterVersion variable defined in our environments.

# helmfile.yaml 

environments:
  staging-cluster-1:
    values:
      - values/global/versions.yaml
      - values/staging/values.yaml
      - clusterVersion: "1.24"
...


---

# {{ $globalVersions := index .Values.clusterVersionIndex .Values.clusterVersion }}
# {{ $versions := mustMerge .Values.k8sVersions $globalVersions }}

releases:
- name: aws-kube-proxy
  namespace: kube-system
  version: {{ $versions | get "awsKubeProxy.chartVersion" }}
  chart: aws-kube-proxy
  values:
    - releases/{{`{{ .Release.Name }}`}}/values.yaml.gotmpl

- name: aws-coredns
  namespace: kube-system
  version: {{ $versions | get "awsCoreDNS.chartVersion" }}
  chart: aws-coredns
  values:
    - releases/{{`{{ .Release.Name }}`}}/values.yaml.gotmpl

- name: cluster-autoscaler
  namespace: infra
  chart: autoscaler/cluster-autoscaler
  version: {{ $versions | get "clusterAutoscaler.chartVersion" }}
  values:
    - releases/{{`{{ .Release.Name }}`}}/no-image-values.yaml.gotmpl
  set:
    - name: image.tag
      value: {{ $versions | get "clusterAutoscaler.image.tag" }}

Note the two lines that appear to be comments. This is actually code processed by helmfile and used to perform the actual lookup using the index function of the go template library. The comment # hash is added to make the code more readable in IDEs (e.g., vs-code) and to avoid helmfile passing any values rendered by the code to the next rendering phase (helmfile performs multiple rendering passes).

The $globalVersions variable contains the complete dictionary value for the corresponding clusterVersion. For example, using the values defined previously, for clusterVersion: "1.24", the value of $globalVersions would be:

    clusterAutoscaler:
      chartVersion: 9.27.0
      image:
        tag: v1.24.3
    awsVpcCni:
      chartVersion: 1.13.4
    awsKubeProxy:
      chartVersion: 1.24.10
    awsCoreDNS:
      chartVersion: 0.193.0

We could have used the $globalVersions variable directly at this point. However, we wanted to provide another important functionality: allow specific Kubernetes clusters to override this version.

This is where the function mustMerge comes into play. The line below performs a merge with another yaml map named k8sVersions:

# {{ $versions := mustMerge .Values.k8sVersions $globalVersions }}

So in any environment-specific values file, such as values/staging/values.yaml , we could define values for any item in the $globalVersions map in the k8sVersions map to override that value. For example:

# values/staging/values.yaml

k8sVersions:
  clusterAutoscaler:
    chartVersion: 9.26.0

In the final variable $versions, which we will use in our release definitions, the cluster autoscaler version will be “9.26.0”, not “9.27.0”, because we performed an override for that map key.

If we do not need to override any versions, we just leave an empty map definition:

# values/staging/values.yaml

k8sVersions: {}

Finally, let’s use the values of the map defined for $versions in one of our chart release definitions:

- name: cluster-autoscaler
  namespace: infra
  chart: autoscaler/cluster-autoscaler
  version: {{ $versions | get "clusterAutoscaler.chartVersion" }}
  values:
    - releases/{{`{{ .Release.Name }}`}}/values.yaml.gotmpl
  set:
    - name: image.tag
      value: {{ $versions | get "clusterAutoscaler.image.tag" }}

Here we are using helmfile’s get function with a pipe to lookup a key from the $versions.

Summary

At first glance, this trick might seem a little confusing. But I found it solves a problem tracking and changing multiple hardcoded versions passed in different values files. This makes it easy to define all the versions required for a specific Kubernetes cluster version in one place – which makes upgrades a little less error-prone and less painful.

The Problem

The Solution

Setup

Summary

Leave a Comment Cancel Reply