Chef Cookbook Continuous Integration With Gitlab and Kubernetes

chef, ci/cd, containers, devops

EDIT: Chef changed their chefdk docker image so that git didn’t work by default. I created a personal docker image (mmencel/chefdk:latest) with a fix to make it work.

I recently put together a small Kubernetes cluster in our datacenter and was looking for an easy win to demonstrate the viability of using Kubernetes and containers to host some of our services. We use Gitlab (Community Edition) internally to host all our projects and have some static Gitlab CI Runners running on full VMs in VMware. The runners are almost always idle so a full VM is mostly wasted on the service. Recently, Gitlab merged a PR in the gitlab-ci-multi-runner project that enables the use of a Kubernetes executor. The executor automatically spins up and destroys as many runner containers as it needs (up to your configured max) to meet the CI build demand from the Gitlab server.

Since I’m still pretty new to Kubernetes and the docs for the Gitlab/Kubernetes executor aren’t fully fleshed out yet, I struggled a bit to get it working. This may help someone else out who’s in the same situation.

I’ll assume that at this point you have a running Kubernetes cluster.

Create a Runner in Gitlab

As of the time I’m writing this, the Kubernetes executer does not auto-register itself as a runner in Gitlab. I found the easiest way to generate the token I needed in the Kubernetes ConfigMap was to download and run the gitlab/gitlab-runner:alpine docker image on my workstation, connect to the shell of that container, and register a new runner. Start the container….ignore the warnings about the missing toml file…

docker run gitlab/gitlab-runner:alpine

In a second shell, attach to the container.

docker exec -it gitlab/gitlab-runner:alpine /bin/bash

Register a runner from within the container by running this command and following the prompts.

gitlab-ci-multi-runner register

You should now see your new runner in the Gitlab UI. Grab the “runner token” for your new runner as you will need it for the Kubernetes Deployment ConfigMap below.

If this is your only runner then you probably don’t need to worry about tagging it. In my case, I gave the runner a tag of ‘kubernetes’.

Setup the Kubernetes Executor

I took the simpler approach of setting up the executor inside my Kubernetes cluster rather than in a VM instance outside the cluster. That way I didn’t need to worry about setting up the Kubernetes tokens for the executor.

My executor is a Kubernetes deployment that looks like what you see below. The ConfigMap section sets up the config.toml the executor container will use to connect to the runner you just registered in the Gitlab UI. It also defines how many concurrent runner containers can be spun up at any given time. The Deployment section tells Kubernetes how to deploy my executor.

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner
  namespace: gitlab
data:
  config.toml: |
    concurrent = 4

    [[runners]]
      name = "Kubernetes Runner"
      url = "{GITLAB_CI_URL}"
      token = "{RUNNER_TOKEN}"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "gitlab"
        allow_privileged = true

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gitlab-runner
  namespace: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab-runner
  template:
    metadata:
      labels:
        name: gitlab-runner
    spec:
      containers:
      - args:
        - run
        image: gitlab/gitlab-runner:alpine
        imagePullPolicy: Always
        name: gitlab-runner
        volumeMounts:
        - mountPath: /etc/gitlab-runner
          name: config
        - mountPath: /etc/ssl/certs
          name: cacerts
          readOnly: true
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: gitlab-runner
        - name: cacerts
          hostPath:
            path: /usr/share/ca-certificates/mozilla

Once you’ve finished editing your deployment, deploy it with kubectl. It should deploy a new container and start the gitlab executor with the config.toml file you specified in the ConfigMap. In the Gitlab UI you should see the runner Last Contact field get updated as the executor attaches with the runner’s token.

CI Some Cookbooks

I’m just going to show how to do some quick linting tests using Rubocop and Foodcritic with Gitlab CI. Running ChefSpec, KitchenCI and other more advanced testing tools may be a future blog post, though I do give a short example for ChefSpec below.

Tests in Gitlab CI are controlled by the .gitlab-ci.yml file which is deployed in the root of your project in Gitlab. At its most basic you just need to declare what image you want to test with and then some test blocks. Check out the .gitlab-ci.yml docs to read more about what you can do.

I’m going to use the latest ChefDK docker image as that has the tools I need already baked in. I also add the test blocks in for my two lint tools.

image: mmencel/chefdk:latest

rubocop:
  script:
    - /opt/chefdk/bin/cookstyle
foodcritic:
  script:
    - /opt/chefdk/bin/foodcritic . 

Save that as .gitlab-ci.yml in the root of your cookbook project and commit it. As soon as that is pushed to Gitlab, the CI jobs should kick off almost immediately.

Bam! You are a Chef cookbook CI master!

Things To Consider

If you start running any ChefSpec or KitchenCI tests you’re going to need your runners to be able to authenticate to your Chef server. I setup a Chef key for my Gitlab runners to use and then added it as a secure variable to my projects. You can add secure variables to your Gitlab CI runs via the Project Settings – Variables section of each project. Drop a Chef key there and then you can use that in your CI runners as an environment variable. My ChefSpec block looks like this in my .gitlab-ci.yml file.

rspec:
  script:
    - export CHEF_SERVER_KEY=$CHEF_SERVER_KEY
    - mkdir -p ~/.berkshelf/
    - cp .ci/config.json ~/.berkshelf/config.json
    - mkdir -p ~/.chef/
    - cp .ci/config.rb ~/.chef/config.rb
    - /opt/chefdk/embedded/bin/rspec

You could do something very similar for KitchenCI and other tools that need to authenticate to the Chef server.

Wrap Up

I hope this helps someone. It took me a bit of trial and error before I got all the steps to work the way I wanted. Getting CI pipelines setup for my cookbooks has been a goal for a while. Kubernetes has made it easy and super fast to test my cookbook code. I still do linting in my editor, but having it in the CI pipeline is the first step to Continuous Deployment. My ultimate goal is a CI/CD pipeline that runs all the linting, ChefSpec tests, and KitchenCI tests. Getting all those tests to work and come back green in Gitlab CI may allow my organization to begin auto-deploying to staging/production.

I’m always looking for feedback, so if I missed something important or need to clarify a step let me know. I’m usually around the Chef Community Slack channel or you can send me an email or message me on Twitter.

2 thoughts on “Chef Cookbook Continuous Integration With Gitlab and Kubernetes

  1. Hi,

    I’d followed the procedure given in “Setup the Kubernetes Executor” section of this article, but am not able to see the kubernetes runner inside gitlab portal. I am getting below message inside the kuberentes pod.

    2017-05-10T16:13:29.975407552Z ERROR: Checking for jobs… forbidden runner=v1egHj5b
    2017-05-10T16:13:29.975446552Z ERROR: Runner https:///ci/ is not healthy and will be disabled!

    Any help on this would be appreciated.
    Thank you in advance.

    1. Hi Rahul,

      On your Gitlab server in Runners section of the Admin page, do you see the registered runner token that you generated in the “Create a Runner in Gitlab” section above?

      Are you using tags on the runner in Gitlab? I have mine with a tag named “kubernetes”.

      Matt

Leave a Reply

Your email address will not be published. Required fields are marked *