Backing up OpenShift

Introduction

OpenShift provides a lot of container resilience. If a Pod dies and was created through one of the higher level resources (ReplicaSet, Deployment, StatefulSet, etc), OpenShift will recreate it.

So a Pod should be considered ephemeral.

Likewise, an OpenShift cluster should be considered disposable. If there is a problem with the cluster, a new one should be created and the old one destroyed.

However, there are some circumstances where you want to preserve the cluster configuration and restore it in the case of a problem.

In this blog, I will describe how to use Heptio’s Velero to back up and restore an OpenShift cluster.

But first let’s add some data into our OpenShift cluster.

Populating OpenShift

In order to exercise the backup/restore procedure, we need to load some data into OpenShift (more specifically, etcd).

As it doesn’t matter what kind of resource we load, let’s just create a bunch of ConfigMaps for simplicity. In fact, I will use the following script the keep a circular list of 20 ConfigMaps. (See the source code here.)

COUNTER=0
RANGE=100
SNAKE_SIZE=20
while true
do
let DELETE=($RANGE+$COUNTER-$SNAKE_SIZE)%100
echo “Deleting key snake-$DELETE”
kubectl delete configmap snake-$DELETE
CONFIG_MAP=snake-$COUNTER
KEY=k-$COUNTER
VALUE=$(date)
echo “Adding key: $KEY, value: $VALUE”
kubectl create configmap $CONFIG_MAP — from-literal=”$KEY=$VALUE”
let COUNTER=(COUNTER+1)%RANGE
count=$(kubectl get cm | grep snake- | wc -l)
echo “Number of config maps: $count”
sleep 1
done

If you run this script for a few minutes, you will see an output like this:

Adding key: k-6, value: Sun Jul 7 13:00:26 EDT 2019
configmap/snake-6 created
Number of config maps: 20
Deleting key snake-87
configmap “snake-87” deleted
Adding key: k-7, value: Sun Jul 7 13:00:28 EDT 2019
configmap/snake-7 created
Number of config maps: 20
Deleting key snake-88
configmap “snake-88” deleted
Adding key: k-8, value: Sun Jul 7 13:00:30 EDT 2019
configmap/snake-8 created
Number of config maps: 20

Introducing Velero

Heptio Velero is a fantastic way to back up a Kubernetes environment, including the Persistent Volumes. You can find more information here.

In this exercise, we will use it to back up the OpenShift objects, stored in etcd.

Here is a diagram of how to backup process works:

The procedure to install Velero is available here. Follow the steps described in this link to install Velero in your OpenShift cluster.

Backing up OpenShift

Now, we are ready to run a backup. In this example, I will back up only the default project, where the ConfigMaps have been created.

Run the following command to back up the OpenShift project:

velero backup create my-backup --include-namespaces default

And you will see the following output:

patro:aws edu$ velero backup create my-backup --include-namespaces default
Backup request "my-backup" submitted successfully.
Run `velero backup describe my-backup` or `velero backup logs my-backup` for more details.

Very simple! If you look at your AWS S3 bucket, you will see there are 2 directories created:

patro:aws edu$ aws s3 ls patrocinio2
                           PRE backups/
                           PRE metadata/

Simulating loss

Now that we backed up the data using Velero, let’s simulate some data loss.

First, let’s count how many ConfigMaps we have in our cluster:

patro:scripts edu$ count=$(kubectl get cm | grep snake- | wc -l)
patro:scripts edu$ echo "Number of config maps: $count"
Number of config maps:       20

Run the following script to delete all the ConfigMaps we created:

kubectl delete configmap snake-{1..100}

(Most deletions will fail, but the command above will delete the 20 ConfigMaps.)

If we count the number of ConfigMaps, we will see the following:

patro:scripts edu$ count=$(kubectl get cm | grep snake- | wc -l)
No resources found.
patro:scripts edu$     echo "Number of config maps: $count"
Number of config maps:        0

Restoring OpenShift

OK, time to restore the environment.

Velero makes really simple to recover the OpenShift objects. Run the following command:

velero restore create --from-backup my-backup

You will see an output like this:

patro:openshift-backup edu$ velero restore create --from-backup my-backup
Restore request "my-backup-20190707134323" submitted successfully.
Run `velero restore describe my-backup-20190707134323` or `velero restore logs my-backup-20190707134323` for more details.

If we run the command indicated above, we see that the restore has been completed:

patro:openshift-backup edu$ velero restore describe my-backup-20190707134323
Name:         my-backup-20190707134323
Namespace:    velero
Labels:       <none>
Annotations:  <none>
Phase:  Completed
(...)

Nice! So, if we now look at the ConfigMaps, we see the following result:

patro:openshift-backup edu$ count=$(kubectl get cm | grep snake- | wc -l)
patro:openshift-backup edu$ echo "Number of config maps: $count"
Number of config maps:       20

Wonderful. So we could recover all the ConfigMaps (and any other object in the default namespace).

Conclusion

In this blog, I showed how to use Heptio Velero to back up and restore an OpenShift environment.

Velero makes really simple to back up the etcd data and the Persistent Volumes to an S3 Bucket.

Bring your plan to the IBM Garage.
Are you ready to learn more about working with the IBM Garage? We’re here to help. Contact us today to schedule time to speak with a Garage expert about your next big idea. Learn about our IBM Garage Method, the design, development and startup communities we work in, and the deep expertise and capabilities we bring to the table.

Schedule a no-charge visit with the IBM Garage.