How to implement MongoDB replication across multiple OpenShift clusters

Introduction

MongoDB has an interesting feature of providing replication across multiple instances, using replica set (This is mongo replica set, not the Kubernetes resource type with the same name). You can read more about MongoDB replication at https://docs.mongodb.com/manual/replication/.

Now, let’s talk about multiple clusters. One great benefit of using MongoDB replication in a multicluster environment is the fact that the mongo instances can be spread across multiple clusters. Neat!

In this article, I will show how to deploy a 3-instance MongoDB replica set across two OpenShift clusters, using the following architecture:

Step 1: Deploy Mongo1 (Primary)

On Cluster1, run the following command to create a new project:

oc new-project mongo

Now run the following command to create the PVC required to run MongoDB:

oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo1-pvc.yaml

You can see the result by running the following command:

oc describe pvc mongo1-pvc

Now, let’s create the Deployment by running the following command:

oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo1-deployment.yaml

You can see the result by running the following command:

oc describe deploy mongo1

Notice that we want to treat the mongo instances independently, so we are labeling the Pods differently. Now, let’s expose this Deployment with the following command:

oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo1-service.yaml

This command exposes the mongo1 instance as NodePort, port 30017.

You can check the result by running the following command:

oc describe svc mongo1

Step 2: Deploy Mongo2 (Secondary)

We need to do the same for the second instance of Mongo. As we are still working on the same OpenShift cluster, we will use a different PVC, Deployment, and Service. Run the following commands:

oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo2-pvc.yaml
oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo2-deployment.yaml
oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo2-service.yaml

Step 3: Deploy Mongo3 (Secondary)

Now, it’s time to move to a second OpenShift cluster.

After configuring the CLI to talk to the second OpenShift cluster, create the same project:

oc new-project mongodb

And the following commands:

oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo1-pvc.yaml
oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo1-deployment.yaml
oc apply -f https://raw.githubusercontent.com/patrocinio/guestbook/mcm-mongo/mongo1-service.yaml

Step 4: Configure MongoDB Replica Set

At this point, we have the three instances of mongo running in 2 OpenShift clusters. They are exposed as:

• <cluster1>:30017
• <cluster1>:30018
• <cluster2>:30017

We now have to configure the Mongo replica set.

Assuming that we are still connected to the second OpenShift cluster, run the following command:

kubectl exec -it $(kubectl get po -o jsonpath='{.items[0].metadata.name}') -- mongo

You will see an output like this:

MongoDB shell version v4.2.2
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
[...]
>

Run the following command to create the replica set:

rs.initiate()

You will see an output like this:

> rs.initiate()
{
"info2" : "no configuration specified. Using a default configuration for the set",
"me" : "mongo1-6c66dcc8c7-jkxv9:27017",
"ok" : 1,
"$clusterTime" : {
"clusterTime" : Timestamp(1578703551, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
},
"operationTime" : Timestamp(1578703551, 1)
}

Now run the following commands to reconfigure the instance hostname, replacing <cluster2> with the IP address or hostname of any worker in the cluster:

var cfg = rs.conf();
cfg.members[0].host="<cluster2>:30017";
rs.reconfig(cfg);

You can check the result by running the following command:

rs.status();

Now, let’s add the mongo instances running on the other cluster. Run the following commands:

rs.add("<cluster1>:30017");
rs.add("<cluster1>:30018");

Finally, run the following command to check the status:

rs.status();

You will see that the first instance (<cluster2>:30017) is the primary. You can exit the Pod, by running exit.

You should see 3 members in the replica set.

Step 5: Testing the connect to the replica set

Now, we need a mongo client to test the connection to the replica set.

I found Robo 3T (available at https://robomongo.org/download) very useful.

After downloading it, create a new Connection, with Type Replica Set. In the Members, specify the 3 instances defined above:

After defining the connection, select the connection name and click Connect. You will see the following screen:

Expand the Replica Set, right-click the first instance and select “Server Status”. You see the mongo instance is running fine.

Step 6: Simulate a disruption in one instance

Now, let’s simulate a disruption in one mongo instance.

Assuming you are still connected to the second OpenShift cluster, run the following command to stop the mongo Pod:

oc scale deploy mongo1 --replicas=0

Now, go back to Robo 3T and try to get the status of this instance. It returns with the following error:

Now, restart the Pod:

oc scale deploy mongo1 --replicas=1

And run the following command to get the status:

kubectl exec -it $(kubectl get po -o jsonpath='{.items[0].metadata.name}') -- mongo
rs0:SECONDARY> rs.status()

You see the prompt shows that this instance is now secondary, and one of the instances in the first OpenShift cluster became primary.

Conclusion

In this article, I showed how to create a MongoDB replica set with 3 instances in 2 OpenShift clusters.

This configuration enables high availability across clusters, using the native Mongo replication.