How to configure Kubernetes audit log in IBM Cloud Private

2 min readJun 6, 2018

One of the new features of Kubernetes v1.10 is audit log (https://kubernetes.io/docs/tasks/debug-application-cluster/audit/), creating an event for every activity that matches the audit policy.

To enable audit logging in IBM Cloud Private (ICP), you can set the flag auditlog_enabled in the config.yaml file, and the installation will enable for you.

However, what hapens if you already deployed your ICP environment and you want to enable Kubernetes audit afterwards?

In this blog, I present a way to do it.

What needs to be done?

Well, it’s much simpler to enable the audit log before install, but it’s possible to do it without falter.

We need to pass the "audit-policy-file" flag to the apiserver hyperkube command. To do it, it will require doing some tricks:

Stop kubelet in master
Update master configuration
Restart kubelet in master

Stop kubelet in master

As hyperkube controls the api-server, we need to stop kubelet to edit the configuration.

Run the following command in the master nodes:

systemctl stop kubelet

Update master configuration

Now, we can edit the master configuration and add the audit flag.

In the master node, follow these steps:

edit the file /etc/cfc/pods/master.json
search for the ”name”: “apiserver”
In the command section, you should see the following line:

"--service-cluster-ip-range=10.0.0.1/24"

Replace it with the following contents:

"--service-cluster-ip-range=10.0.0.1/24",
"--audit-policy-file=/etc/cfc/conf/audit-policy.yaml",
"--audit-log-path=-"

In the same section, search for the following line:

"--feature-gates="

and replace with

"--feature-gates=AdvancedAuditing=true"

That’s it! We just need to re-enable kubelet in the server

Restart kubelet in master

Now, that we reconfigured in the apiserver to log the audit information, we need to re-enable kubelet. Run the following command:

systemctl start kubelet

Now the logs will be persisted in the log directory of the Docker container.

To see it, run the following command to get to Docker container ID:

docker ps | grep apiserver_k8s

and get the first column (CONTAINER ID).

Then run

docker logs <container-id>

You will see a lots of audit log information.

This log will be automatically captured by filebeat, sent to logstash and stored in ElasticSearch. The world is beautiful!

Almost…

This procedure captures tons of data and might fill your file system.

So you might want to look at the policies enabled in the file /etc/cfc/conf/audit-policy.yaml before leaving it running for days.

Conclusion

In this blog, I showed how to configured Kubernetes audit log after installing ICP.

The procedure was very surgical, but nevertheless simple.

Happy auditing!