This guide will go through the steps needed to setup a local, container based, playground to test out Replicante Core and the official agents. By the end you should have a full system running that can be used to experiment with and demo all replicante features and behaviours.

Please note: this guide does not explain how to install and configure replicante for production use and it does not attempt to cover all possible configuration, installation and management details.

0. Overview

Replicante provides an opinionated development tool, replidev, that can be used to demo and experiment with replicante’s features as well.

In this guide we will use replidev to start a set of containers to run:

Once you are familiar with how the playgrounds setup works you can run any node. Only nodes with an agent attached to them will be exposed by the discovery server.

1. Get the code

There are two command line tools you will need to install:

Install replidev

First of all, replidev requires some dependencies:

This tools must be avaliable in $PATH for replidev to be able to use them.

Once dependencies are available fetch and compile replidev:

# Cargo install will fetch the git sources and make the compiled binary available in $PATH.
$ cargo install --git replidev

# Check that replidev is working:
$ replidev --version

With all the dependencies in place we can get the playgrounds repo, which provides the definitions for datastore nodes and Replicante Core pods to start everything easily:

$ cd /path/for/replicante/repos
$ git clone
$ cd playgrounds/

# Check replidev works and finds podman:
$ replidev play node-list

Install replictl

The replicante command line tool comes pre-built for some platforms. The official documentation has a section on installing from pre-built binaries you can try first.

If that does not work for you or you want to install the latest version from the repo:

# Cargo install will fetch the git sources and make the compiled binary available in $PATH.
$ cargo install --git replictl

# Check that replidev is working:
$ replictl --version

2. Start Replicante Core

Replicante Core uses TLS certificates to securely communicate with agents. Mutual TLS authentication is required to enable the action system on agents.

First, generate certificates to be used by core and agents:

$ replidev gen-certs
--> Generating CA certificates
--> Generating Server certificates
--> Generating Client certificates
--> Bundling some certs
CA Certificate:     ./data/pki/replidev/certs/replidev.crt
CA Private Key:     ./data/pki/replidev/keys/replidev.key
Client Bundle:      ./data/pki/replidev/bundles/client.pem
Client Certificate: ./data/pki/replidev/certs/client.crt
Client Private Key: ./data/pki/replidev/keys/client.key
Server Bundle:      ./data/pki/replidev/bundles/server.pem
Server Certificate: ./data/pki/replidev/certs/server.crt
Server Private Key: ./data/pki/replidev/keys/server.key

This step is only needed once on a new machine or when the certificates expire.

With certificates in place start the Replicante Core stack, including all dependencies, with:

$ replidev play replicore-start
--> Create pod play-replicore
--> Start container play-replicore-zookeeper
--> Wating 10s for play-replicore-zookeeper to start
--> Start container play-replicore-mongo
--> Wating 10s for play-replicore-mongo to start
--> Start container play-replicore-kafka
--> Start container play-replicore-app
--> Start container play-replicore-ui
--> Initialise play-replicore/mongo from play-replicore-mongo
==> Checking (and initialising) mongo replica set ...
MongoDB shell version v4.2.5
connecting to: mongodb://
Implicit session: session { "id" : UUID("37cb50ad-a3f4-483b-a712-00754db853e5") }
MongoDB server version: 4.2.5
---> Replica Set not initialised, correcting ...
---> Replica set configured, waiting for primary.
---> Checking Replica Set status ...
---> Replica Set Ready!
==> Ensuring all mongo indexes exist ...
MongoDB shell version v4.2.5
connecting to: mongodb://
Implicit session: session { "id" : UUID("9ea25960-0f73-4866-903a-a04d6c8ea869") }
MongoDB server version: 4.2.5

Once the above command completes you should have access to the WebUI at http://localhost:3000 and you can follow replicante core logs with podman logs -f play-replicore-app.

3. Start the playgrounds API server

At this point you should see an empty WebUI and, if you look at the logs, some errors like the one below:

{"msg":"Cluster discovery error","level":"ERRO","ts":"2020-04-26T09:23:08.957288273+00:00","module":"replicante::components::discovery::election","error_cause":"http://podman-host:9876/discover: error trying to connect: Connection refused (os error 111)","error_layers":2,"error_message":"HTTP request failed","error_name":"HttpRequest"}

This is because this Replicante stack is configured to discover clusters from the playgrounds API server running on your podman host. To start this service open a new console and run:

$ replidev play server
--> Server listening at

While the WebUI will stay empty, the error should be gone now!

4. Start a MongoDB cluster

It is now time to start our first datastore node. Any of the datastores in can be started but only those where an agent is available will be listed in replidev play server.

For our quick start we will create a MongoDB Replica Set node:

$ replidev play node-start mongo/rs
--> Starting mongo/rs node play-node-JfC9yNPm for cluster mongo-rs
--> Create pod play-node-JfC9yNPm
--> Start container play-node-JfC9yNPm-mongo
--> Start container play-node-JfC9yNPm-repliagent

If you wait up to a minute for replicante core to perform a discovery cycle you should see a new cluster and agent in the WebUI!

Since the node has not been initialised it should be in the NODE_DOWN state. Let’s fix that now:

# Note that the port set in the host attribute is set dynamically and may be different.
# Use replidev play node-list to check for the correct store port.
$ podman exec -it play-node-JfC9yNPm-mongo mongo --eval 'rs.initiate({_id: "mongo-rs", members: [{_id: 0, host: "podman-host:10000"}]})'

Once replicante refreshes the cluster, the node should be in the UP state.

But a one node replica set is not that useful, let us add two more nodes and wait for them to show up (as NODE_DOWN):

$ replidev play node-start mongo/rs
--> Starting mongo/rs node play-node-8bMJWWs0 for cluster mongo-rs
--> Create pod play-node-8bMJWWs0
--> Start container play-node-8bMJWWs0-mongo
--> Start container play-node-8bMJWWs0-repliagent
$ replidev play node-start mongo/rs
--> Starting mongo/rs node play-node-q3mKXaG4 for cluster mongo-rs
--> Create pod play-node-q3mKXaG4
--> Start container play-node-q3mKXaG4-mongo
--> Start container play-node-q3mKXaG4-repliagent

And add them to the initialised replica set:

$ podman exec -it play-node-JfC9yNPm-mongo mongo --eval 'rs.add("podman-host:10002");'
$ podman exec -it play-node-JfC9yNPm-mongo mongo --eval 'rs.add("podman-host:10004");'

5. Start a zookeeper cluster

To demo replicante’s cross-store features we are going to also start a zookeeper cluster.

This is very similar to what we did for MongoDB so you could skip it if you are not interested:

$ cat - > zookeeper.demo.json <<EOS
  "id": "0",
  "port": 10006,
  "election": 10007,
  "client": 10008
}, {
  "id": "1",
  "port": 10010,
  "election": 10011,
  "client": 10012
}, {
  "id": "2",
  "port": 10014,
  "election": 10015,
  "client": 10016
$ replidev play node-start zookeeper --var-file 'cluster=zookeeper.demo.json' --var 'my_id=0'
$ replidev play node-start zookeeper --var-file 'cluster=zookeeper.demo.json' --var 'my_id=1'
$ replidev play node-start zookeeper --var-file 'cluster=zookeeper.demo.json' --var 'my_id=2'

6. Schedule some actions

Before you can interact with the local Replicante Core stack you need to set up replictl:

$ replictl context login --context play
Replicante API address: http://localhost:16016/
(Optional) API CA certificate file:
(Optional) API client certificate bundle:

$ replictl context select --context play
$ replictl context list
*        play   http://localhost:16016/   Not set     Not set      Not set     Not set   Not set

# Set the namespace for the context once for all other commands.
$ replictl context change
Select a namespace (empty to clear selection): default
Select a cluster (empty to clear selection):
Select a node (empty to clear selection):

Actions are defined as YAML files. This is a bit much for an example ping action but provides many advantages for real use systems:

For our example we’ll schedule a ping action against any cluster/node:

$ cat - > ping.yaml <<EOS
kind: AgentAction
metadata: {}
$ replictl apply -f ping.yaml --cluster mongo-rs --node 'https://podman-host:10001'
$ replictl apply -f ping.yaml --cluster mongo-rs --node 'https://podman-host:10003'
$ replictl apply -f ping.yaml --cluster zookeeper --node 'https://podman-host:10009'

Note how the same action definition is applied to any nodes across any cluster.

Actions are scheduled during the next cluster refresh operation, and updated every refresh. As a result we’ll need two refresh cycles to happen before our ping action is marked as done.

While we could wait for these cycles to be scheduled, we can also request additional refresh tasks to be performed against a specific cluster. This is great for testing clusters and agents, like now, but when done too often it could increase the load on agents and nodes in the cluster.

$ replictl cluster refresh --cluster mongo-rs
Cluster refresh scheduled

7. Clean up everything

Once you are done experimenting with the playgrounds you can stop all processes and nodes.

Stop all the datastore nodes:

$ replidev play node-list
play-node-ou1ZPcVe   zookeeper   10014        10016         10017        Running   0dc4ed954b0b  
play-node-epjE9kI8   zookeeper   10010        10012         10013        Running   26c235601056  
play-node-RJZBqGQf   zookeeper   10006        10008         10009        Running   47a69ad89a1f  
play-node-q3mKXaG4   mongo-rs    10004        10004         10005        Running   42a03c999b98  
play-node-8bMJWWs0   mongo-rs    10002        10002         10003        Running   32593ddfdf9b  
play-node-JfC9yNPm   mongo-rs    10000        10000         10001        Running   8d2312ace6da

$ replidev play node-stop play-node-ou1ZPcVe play-node-epjE9kI8 play-node-RJZBqGQf play-node-q3mKXaG4 play-node-8bMJWWs0 play-node-JfC9yNPm
--> Stop pod play-node-ou1ZPcVe
--> Remove pod play-node-ou1ZPcVe
--> Stop pod play-node-epjE9kI8
--> Remove pod play-node-epjE9kI8
--> Stop pod play-node-RJZBqGQf
--> Remove pod play-node-RJZBqGQf
--> Stop pod play-node-q3mKXaG4
--> Remove pod play-node-q3mKXaG4
--> Stop pod play-node-8bMJWWs0
--> Remove pod play-node-8bMJWWs0
--> Stop pod play-node-JfC9yNPm
--> Remove pod play-node-JfC9yNPm

Stop the replicante core stack:

$ replidev play replicore-stop
--> Stop pod play-replicore
--> Remove pod play-replicore

Stop the replidev play server by hitting Ctrl-C on the terminal where it is running.

And finally you can delete all node and core data with:

$ replidev play node-clean-all --confirm
--> Clean data for all nodes (from ./data/nodes/)

$ replidev play replicore-clean --confirm
--> Clean data for play-replicore pod (from ./data/replicore)