Bootstrapping CoreOS on VMware ESXi with iPXE

October 13, 2016 containers, devops

We run our own data centers at WIU and most of our workloads run on VMs in ESXi. I’m relatively new to containers and just recently started experimenting with Rancher for container orchestration. Rancher is nice, but adding nodes to the Kubernetes cluster is still a manual process. If I want to eventually put production workloads in containers I need a way to autoscale my Kubernetes cluster based on CPU or memory utilization.

The first step in that process was figuring out how to automatically provision container nodes in VMware. I experimented with several OS options including CentOS Atomic and VMware’s Photon OS but finally settled on CoreOS. I am more familiar with RHEL/CentOS, but the community around CoreOS is much more robust and it felt like I would have a better chance of success with it. This post explains how I bootstrap CoreOS to RAM using iPXE, install CoreOS onto disk, apply default configs, and finally join it to the Kubernetes cluster.

It took me a couple days of pulling different pieces of info together before I got to this solution. There are other options as well. I looked into CoreOS bare metal, but that needs an additional bootcfg service to be running somewhere and I was looking for something a little simpler.

The repository files and scripts are managed with Chef and all the configs are in source control on our Gitlab server.

Prepare a Default VMware Template

Create a new empty VM.
To use iPXE, custom ROMs have to be added to the VM. Save the iPXE ROMs to a shared datastore in VMware and edit the VM’s vmx file according to the iPXE Instructions.
Use govc to set enableUUID=1 on the VM.
Convert the empty VM to a template.

Prepare a CoreOS Repository Location

For the iPXE and CoreOS config files and the CoreOS images, an HTTP repository is needed. In these examples, I’ve used http://SOMEHOST/coreos.

The root /coreos contains the config files.

A subdirectory that looks like /coreos/VERSION contains the CoreOS images. The stable CoreOS version in the example uses /coreos/1122.2.0/

iPXE Configuration

Setup of iPXE with DHCP is beyond the scope of this post. Once done however DHCP can be configured so that when a VM is deployed from the template and is powered on, it immediately network boots and reads an iPXE config file that looks something like this.

(./coreos/coreos.ipxe)

#!ipxe
# Boot a persistent CoreOS to RAM

set base-url http://SOMEHOST/coreos

kernel ${base-url}/1122.2.0/coreos_production_pxe.vmlinuz coreos.config.url=http://SOMEHOST/coreos/ignition.json cloud-config-url=http://SOMEHOST/coreos/cloud-config.sh coreos.first_boot=1
initrd ${base-url}/1122.2.0/coreos_production_pxe_image.cpio.gz
boot

First Boot

During the first boot of CoreOS, several things happen.

CoreOS boots the PXE image.
Because coreos.first.boot=1 is set in the .ipxe file, the CoreOS Ignition script is run during bootup. This preps and formats the base disk on the VM (/dev/sda).

(./coreos/ignition.json)

{
  "ignition": {
    "version": "2.0.0",
    "config": {}
  },
  "storage": {
    "disks": [
      {
        "device": "/dev/sda",
        "wipeTable": true,
        "partitions": [
          {
            "label": "ROOT",
            "number": 0,
            "size": 0,
            "start": 0
          }
        ]
      }
    ],
    "filesystems": [
      {
        "name": "root",
        "mount": {
          "device": "/dev/sda1",
          "format": "ext4",
          "create": {
            "force": true,
            "options": [
              "-LROOT"
            ]
          }
        }
      }
    ]
  },
  "files": {},
  "systemd": {},
  "networkd": {},
  "passwd": {}
}

CoreOS finishes booting into memory and then runs the cloud-config.sh file which was also set in the .ipxe file.

(./coreos/cloud-control.sh)

#!/bin/bash

curl -kO http://SOMEHOST/coreos/cloud-config.yaml
sudo coreos-install -d /dev/sda -c cloud-config.yaml -b http://SOMEHOST/coreos
sudo reboot

This script…

pulls the cloud-config.yaml file down from the local repository
installs CoreOS to the newly formatted disk on the VM
reboots the VM

Here is a sample cloud-config.yaml…

(./coreos/cloud-config.yaml)

#cloud-config

write_files:
  - path: /etc/ntp.conf
    content: |
      server SOME.TIME.SERVER

      # - Allow only time queries, at a limited rate.
      # - Allow all local queries (IPv4, IPv6)
      restrict default nomodify nopeer noquery limited kod
      restrict 127.0.0.1
      restrict [::1]
  - path: "/etc/rancher-agent"
    permissions: "0700"
    owner: "root"
    content: |
        #!/bin/bash
        # This script bootstraps the rancher agent for the new server on boot
        INTERNAL_IP=$(ip add show ens160 | awk '/inet/ {print $2}' | cut -d/ -f1 | head -1)
        SERVER="https://rancherserver:8080"
        TOKEN="TOKEN:KEY"
        PROJID="PROJ_ID"
        AGENT_VER="v1.0.2"
        RANCHER_URL=$(curl -sku $TOKEN $SERVER/v1/registrationtokens?projectId=$PROJID | head -1 | grep -nhoe 'registrationUrl[^},]*}' | egrep -hoe 'https?:.*[^"}]')
        docker run \
            -e CATTLE_AGENT_IP=$INTERNAL_IP \
            -e CATTLE_HOST_LABELS='internal=true' \
            -d --privileged --name rancher-bootstrap \
            -v /var/run/docker.sock:/var/run/docker.sock \
            rancher/agent:$AGENT_VER $RANCHER_URL

coreos:
  units:
    - name: open-vm-tools.service
      command: start
      content: |
        [Unit]
        Description=Service for virtual machines hosted on VMware
        Documentation=http://open-vm-tools.sourceforge.net/about.php
        ConditionVirtualization=vmware
        [Service]
        Restart=always
        ExecStart=/usr/bin/docker run --rm --net=host -v /run/systemd:/run/systemd --name open-vm-tools godmodelabs/open-vm-tools
        ExecStop=-/usr/bin/docker stop open-vm-tools
    - name: rancher-agent.service
      command: start
      content: |
        [Unit]
        Description=Rancher Agent
        After=docker.service
        Requires=docker.service
        After=network-online.target
        Requires=network-online.target

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/etc/rancher-agent
    - name: runcmd.service
      command: start
      content: |
        [Unit]
        Description=Set hostname

        [Service]
        Type=oneshot
        ExecStart=/bin/sh -c "IP=$(ip add show ens160 | awk '/inet/ {print $2}' | cut -d/ -f1 |cut -d. -f4 | head -1) ; sudo hostnamectl set-hostname coreos$IP"
    - name: settimezone.service
      command: start
      content: |
        [Unit]
        Description=Set the time zone

        [Service]
        ExecStart=/usr/bin/timedatectl set-timezone America/Chicago
        RemainAfterExit=yes
        Type=oneshot
    - name: systemd-timesyncd.service
      command: stop
      mask: true
    - name: ntpd.service
      command: start
      enable: true

  # Update profile
  update:
    group: stable
    reboot-strategy: off

ssh_authorized_keys:
  - ssh-rsa SSHKEY

Second and Final Boot

Now that CoreOS is installed to disk…

the VM skips the network boot step and boots straight to disk
once fully booted it finishes setting up any default services and containers that were defined in the cloud-config.yaml file (see above). In this example cloud-config.yaml it…
- automatically joins the node to a Kubernetes cluster in Rancher (via the rancher-agent container that is launched)
- configures the timezone
- switches time sync to NTPD and configures an NTP server
- adds open-vm-tools so that VMware can manage/monitor the CoreOS VM
- sets up the hostname as coreos{IP_LAST_OCTET}
- adds SSH keys

More To Come

Hopefully, as I get the automation pieces in place I can come back to this topic and write about it again. If you have suggestions or ways I can improve this process I’d love to hear it.

Devops in Higher Ed