Bootstrapping CoreOS on VMware ESXi with iPXE
We run our own data centers at WIU and most of our workloads run on VMs in ESXi. I’m relatively new to containers and just recently started experimenting with Rancher for container orchestration. Rancher is nice, but adding nodes to the Kubernetes cluster is still a manual process. If I want to eventually put production workloads in containers I need a way to autoscale my Kubernetes cluster based on CPU or memory utilization.
The first step in that process was figuring out how to automatically provision container nodes in VMware. I experimented with several OS options including CentOS Atomic and VMware’s Photon OS but finally settled on CoreOS. I am more familiar with RHEL/CentOS, but the community around CoreOS is much more robust and it felt like I would have a better chance of success with it. This post explains how I bootstrap CoreOS to RAM using iPXE, install CoreOS onto disk, apply default configs, and finally join it to the Kubernetes cluster.
It took me a couple days of pulling different pieces of info together before I got to this solution. There are other options as well. I looked into CoreOS bare metal, but that needs an additional bootcfg service to be running somewhere and I was looking for something a little simpler.
The repository files and scripts are managed with Chef and all the configs are in source control on our Gitlab server.
Prepare a Default VMware Template
- Create a new empty VM.
- To use iPXE, custom ROMs have to be added to the VM. Save the iPXE ROMs to a shared datastore in VMware and edit the VM’s vmx file according to the iPXE Instructions.
- Use govc to set enableUUID=1 on the VM.
- Convert the empty VM to a template.
Prepare a CoreOS Repository Location
For the iPXE and CoreOS config files and the CoreOS images, an HTTP repository is needed. In these examples, I’ve used http://SOMEHOST/coreos.
The root /coreos contains the config files.
A subdirectory that looks like /coreos/VERSION contains the CoreOS images. The stable CoreOS version in the example uses /coreos/1122.2.0/
iPXE Configuration
Setup of iPXE with DHCP is beyond the scope of this post. Once done however DHCP can be configured so that when a VM is deployed from the template and is powered on, it immediately network boots and reads an iPXE config file that looks something like this.
(./coreos/coreos.ipxe)
#!ipxe # Boot a persistent CoreOS to RAM set base-url http://SOMEHOST/coreos kernel ${base-url}/1122.2.0/coreos_production_pxe.vmlinuz coreos.config.url=http://SOMEHOST/coreos/ignition.json cloud-config-url=http://SOMEHOST/coreos/cloud-config.sh coreos.first_boot=1 initrd ${base-url}/1122.2.0/coreos_production_pxe_image.cpio.gz boot
First Boot
During the first boot of CoreOS, several things happen.
- CoreOS boots the PXE image.
- Because coreos.first.boot=1 is set in the .ipxe file, the CoreOS Ignition script is run during bootup. This preps and formats the base disk on the VM (/dev/sda).
(./coreos/ignition.json)
{ "ignition": { "version": "2.0.0", "config": {} }, "storage": { "disks": [ { "device": "/dev/sda", "wipeTable": true, "partitions": [ { "label": "ROOT", "number": 0, "size": 0, "start": 0 } ] } ], "filesystems": [ { "name": "root", "mount": { "device": "/dev/sda1", "format": "ext4", "create": { "force": true, "options": [ "-LROOT" ] } } } ] }, "files": {}, "systemd": {}, "networkd": {}, "passwd": {} }
- CoreOS finishes booting into memory and then runs the cloud-config.sh file which was also set in the .ipxe file.
(./coreos/cloud-control.sh)
#!/bin/bash curl -kO http://SOMEHOST/coreos/cloud-config.yaml sudo coreos-install -d /dev/sda -c cloud-config.yaml -b http://SOMEHOST/coreos sudo reboot
This script…
- pulls the cloud-config.yaml file down from the local repository
- installs CoreOS to the newly formatted disk on the VM
- reboots the VM
Here is a sample cloud-config.yaml…
(./coreos/cloud-config.yaml)
#cloud-config write_files: - path: /etc/ntp.conf content: | server SOME.TIME.SERVER # - Allow only time queries, at a limited rate. # - Allow all local queries (IPv4, IPv6) restrict default nomodify nopeer noquery limited kod restrict 127.0.0.1 restrict [::1] - path: "/etc/rancher-agent" permissions: "0700" owner: "root" content: | #!/bin/bash # This script bootstraps the rancher agent for the new server on boot INTERNAL_IP=$(ip add show ens160 | awk '/inet/ {print $2}' | cut -d/ -f1 | head -1) SERVER="https://rancherserver:8080" TOKEN="TOKEN:KEY" PROJID="PROJ_ID" AGENT_VER="v1.0.2" RANCHER_URL=$(curl -sku $TOKEN $SERVER/v1/registrationtokens?projectId=$PROJID | head -1 | grep -nhoe 'registrationUrl[^},]*}' | egrep -hoe 'https?:.*[^"}]') docker run \ -e CATTLE_AGENT_IP=$INTERNAL_IP \ -e CATTLE_HOST_LABELS='internal=true' \ -d --privileged --name rancher-bootstrap \ -v /var/run/docker.sock:/var/run/docker.sock \ rancher/agent:$AGENT_VER $RANCHER_URL coreos: units: - name: open-vm-tools.service command: start content: | [Unit] Description=Service for virtual machines hosted on VMware Documentation=http://open-vm-tools.sourceforge.net/about.php ConditionVirtualization=vmware [Service] Restart=always ExecStart=/usr/bin/docker run --rm --net=host -v /run/systemd:/run/systemd --name open-vm-tools godmodelabs/open-vm-tools ExecStop=-/usr/bin/docker stop open-vm-tools - name: rancher-agent.service command: start content: | [Unit] Description=Rancher Agent After=docker.service Requires=docker.service After=network-online.target Requires=network-online.target [Service] Type=oneshot RemainAfterExit=yes ExecStart=/etc/rancher-agent - name: runcmd.service command: start content: | [Unit] Description=Set hostname [Service] Type=oneshot ExecStart=/bin/sh -c "IP=$(ip add show ens160 | awk '/inet/ {print $2}' | cut -d/ -f1 |cut -d. -f4 | head -1) ; sudo hostnamectl set-hostname coreos$IP" - name: settimezone.service command: start content: | [Unit] Description=Set the time zone [Service] ExecStart=/usr/bin/timedatectl set-timezone America/Chicago RemainAfterExit=yes Type=oneshot - name: systemd-timesyncd.service command: stop mask: true - name: ntpd.service command: start enable: true # Update profile update: group: stable reboot-strategy: off ssh_authorized_keys: - ssh-rsa SSHKEY
Second and Final Boot
Now that CoreOS is installed to disk…
- the VM skips the network boot step and boots straight to disk
- once fully booted it finishes setting up any default services and containers that were defined in the cloud-config.yaml file (see above). In this example cloud-config.yaml it…
- automatically joins the node to a Kubernetes cluster in Rancher (via the rancher-agent container that is launched)
- configures the timezone
- switches time sync to NTPD and configures an NTP server
- adds open-vm-tools so that VMware can manage/monitor the CoreOS VM
- sets up the hostname as coreos{IP_LAST_OCTET}
- adds SSH keys
More To Come
Hopefully, as I get the automation pieces in place I can come back to this topic and write about it again. If you have suggestions or ways I can improve this process I’d love to hear it.
very informative thanks for share with us