Surviving disruption in a Kubernetes application

Kubernetes provides a great orchestration capability for Docker containers. Using Kubernetes ReplicaSet (and StatefulSet), we can declare that a certain Kubernetes Pod should be kept alive, and Kubernetes will take charge of monitoring and deploying the Kubernetes Pod as needed.

However, applications are not unscathed from providing resilience at the code level to ensure that the application will perform as expected.

Let’s take the application described at https://medium.com/@epatro/running-distributed-transactions-in-kubernetes-19a8fa27758b, particularly the diagram reproduced here:

Although this application is very simple, it already has a point of concern: when an account is started, it registers itself in the accountSystem microservice. However, if the accountSystem microservice dies, it looses its state.

This is a typical problem of a stateful component: it needs to store its state into a persistent storage, so that when the component crashes (or is killed), it can recover the state from the persistent storage.

Traditionally the storage used to be a Relational Database, where a schema needs to be created, and the application extended extensively to use the database. These days, there are many choices to persist the data, and I am going to use Redis as a simple database.

Here is the new solution:

The Redis component itself needs to use a Kubernetes Persistent Volume, not displayed in the diagram for simplicity.

Now if you deploy the application, you will see the Redis component:

./deployMicrobank.sh

Namespace set to microbank
release “microbank” deleted
NAME: microbank
LAST DEPLOYED: Wed Jan 3 11:36:07 2018
NAMESPACE: microbank
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME TYPE DATA AGE
microbank-redis Opaque 1 6s

==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
microbank-redis Pending 6s

==> v1/Service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
microbank-account-system 172.16.0.181 <nodes> 80:31913/TCP 5s
microbank-balance 172.16.0.101 <nodes> 80:32635/TCP 5s
microbank-john-account 172.16.0.53 <nodes> 80:31773/TCP 5s
microbank-mary-account 172.16.0.182 <nodes> 80:31461/TCP 5s
microbank-redis 172.16.0.150 <none> 6379/TCP 5s

==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
microbank-account-system 1 1 1 0 5s
microbank-balance 1 1 1 0 4s
microbank-john-account 1 1 1 0 4s
microbank-mary-account 1 1 1 0 4s
microbank-redis 1 1 1 0 5s

NOTES:
Happy banking!

You can open the NodePort number by running the following command:

kubectl describe svc microbank-account-system

Name: microbank-account-system
Namespace: microbank
Labels: chart=account-system-0.1.0
Selector: app=microbank-account-system
Type: NodePort
IP: 172.16.0.181
Port: account-system 80/TCP
NodePort: account-system 31913/TCP
Endpoints: 192.168.37.237:80
Session Affinity: None
No events.

Now, you can query the accountSystem microservice:

curl http://<proxy-ip>:<node-port>/accounts

You will see the following response:

{“accounts”:[“john-account”,”mary-account”]}

If you kill the accountSystem Pod, you will see that Kubernetes restarts it:

cd ./scripts

./killPod.sh account-system

Found microbank-account-system-4011826929-s0kr8
pod “microbank-account-system-4011826929-s0kr8” deleted

and the accounts are still available in the microservice:

curl http://<proxy-ip>:<node-port>/accounts

You will see the same response:

{“accounts”:[“john-account”,”mary-account”]}