This guide will go through the steps needed to setup a local, container based, playground to test out Replicante Core and the official agents. By the end you should have a full system running that can be used to experiment with and demo all replicante features and behaviours.

Please note: this guide does not explain how to install and configure replicante for production use and it does not attempt to cover all possible configuration, installation and management details.

0. Overview

Replicante provides an opinionated development tool, replidev, that can be used to demo and experiment with replicante’s features as well.

In this guide we will use replidev to start a set of containers to run:

Once you are familiar with how the playgrounds setup works you can run any node. Only nodes with an agent attached to them will be exposed by the discovery server.

1. Get the code

There are two command line tools you will need to install:

Install replidev

First of all, replidev requires some dependencies:

This tools must be avaliable in $PATH for replidev to be able to use them.

Once dependencies are available fetch and compile replidev:

# Cargo install will fetch the git sources and make the compiled binary available in $PATH.
$ cargo install --git https://github.com/replicante-io/replicante.git replidev

# Check that replidev is working:
$ replidev --version

With all the dependencies in place we can get the playgrounds repo, which provides the definitions for datastore nodes and Replicante Core pods to start everything easily:

$ cd /path/for/replicante/repos
$ git clone https://github.com/replicante-io/playgrounds
$ cd playgrounds/

# Check replidev works and finds podman:
$ replidev play node-list
NODE   CLUSTER   STORE PORT   CLIENT PORT   AGENT PORT   STATUS   POD ID

Install replictl

The replicante command line tool comes pre-built for some platforms. The official documentation has a section on installing from pre-built binaries you can try first.

If that does not work for you or you want to install the latest version from the repo:

# Cargo install will fetch the git sources and make the compiled binary available in $PATH.
$ cargo install --git https://github.com/replicante-io/replicante.git replictl

# Check that replidev is working:
$ replictl --version

2. Start Replicante Core

Replicante Core uses TLS certificates to securely communicate with agents. Mutual TLS authentication is required to enable the action system on agents.

First, generate certificates to be used by core and agents:

$ replidev gen-certs
--> Generating CA certificates
--> Generating Server certificates
--> Generating Client certificates
--> Bundling some certs
CA Certificate:     ./data/pki/replidev/certs/replidev.crt
CA Private Key:     ./data/pki/replidev/keys/replidev.key
Client Bundle:      ./data/pki/replidev/bundles/client.pem
Client Certificate: ./data/pki/replidev/certs/client.crt
Client Private Key: ./data/pki/replidev/keys/client.key
Server Bundle:      ./data/pki/replidev/bundles/server.pem
Server Certificate: ./data/pki/replidev/certs/server.crt
Server Private Key: ./data/pki/replidev/keys/server.key

This step is only needed once on a new machine or when the certificates expire.

With certificates in place start the Replicante Core stack, including all dependencies, with:

$ replidev play replicore-start
--> Create pod play-replicore
ab9fe546c021ba0a366a29d30a1996eea1a0d21d28be78bd5ea3f74c748a5c2a
--> Start container play-replicore-zookeeper
bb1206f6884fa85266a8c6276f29fb60ffd48aa03ebc03f03d7b2d5ecca595aa
--> Wating 10s for play-replicore-zookeeper to start
--> Start container play-replicore-mongo
0d7efa500edab710c6613a7bddb7b3934986fc71631839787ce8e4cbd2310566
--> Wating 10s for play-replicore-mongo to start
--> Start container play-replicore-kafka
e02fc11827131bc98ab9624b4dec87c0c2abfde70ff716570ec250afacc6d9b9
--> Start container play-replicore-app
89ae5c8e81dc620faa9181e69edcac1a276a21bb3b4dcfbc9b3dd90c28a6ab5b
--> Start container play-replicore-ui
e0daf45c61453d25e701cb4b641bf00357a9ca9cb3f2c32a31049efdd378e8c8
--> Initialise play-replicore/mongo from play-replicore-mongo
==> Checking (and initialising) mongo replica set ...
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("37cb50ad-a3f4-483b-a712-00754db853e5") }
MongoDB server version: 4.2.5
---> Replica Set not initialised, correcting ...
---> Replica set configured, waiting for primary.
---> Checking Replica Set status ...
---> Replica Set Ready!
==> Ensuring all mongo indexes exist ...
MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("9ea25960-0f73-4866-903a-a04d6c8ea869") }
MongoDB server version: 4.2.5

Once the above command completes you should have access to the WebUI at http://localhost:3000 and you can follow replicante core logs with podman logs -f play-replicore-app.

3. Start the playgrounds API server

At this point you should see an empty WebUI and, if you look at the logs, some errors like the one below:

{"msg":"Cluster discovery error","level":"ERRO","ts":"2020-04-26T09:23:08.957288273+00:00","module":"replicante::components::discovery::election","error_cause":"http://podman-host:9876/discover: error trying to connect: Connection refused (os error 111)","error_layers":2,"error_message":"HTTP request failed","error_name":"HttpRequest"}

This is because this Replicante stack is configured to discover clusters from the playgrounds API server running on your podman host. To start this service open a new console and run:

$ replidev play server
--> Server listening at http://0.0.0.0:9876

While the WebUI will stay empty, the error should be gone now!

4. Start a MongoDB cluster

It is now time to start our first datastore node. Any of the datastores in https://github.com/replicante-io/playgrounds/tree/main/stores can be started but only those where an agent is available will be listed in replidev play server.

For our quick start we will create a MongoDB Replica Set node:

$ replidev play node-start mongo/rs
--> Starting mongo/rs node play-node-JfC9yNPm for cluster mongo-rs
--> Create pod play-node-JfC9yNPm
8d2312ace6dab2e4fa5d4bbe4a8da166d9cbdda25caab2593f9416c2d896240a
--> Start container play-node-JfC9yNPm-mongo
b1eb87ba78ace5fa086eb9dfd931b9b9554a288a2a49fe6c56a5819fa3108d08
--> Start container play-node-JfC9yNPm-repliagent
c6ac1ae84034cf8a4dd3e203cb8a34851bedfe8faf1d6e635ae1ef7911255f91

If you wait up to a minute for replicante core to perform a discovery cycle you should see a new cluster and agent in the WebUI!

Since the node has not been initialised it should be in the NODE_DOWN state. Let’s fix that now:

# Note that the port set in the host attribute is set dynamically and may be different.
# Use replidev play node-list to check for the correct store port.
$ podman exec -it play-node-JfC9yNPm-mongo mongo --eval 'rs.initiate({_id: "mongo-rs", members: [{_id: 0, host: "podman-host:10000"}]})'

Once replicante refreshes the cluster, the node should be in the UP state.

But a one node replica set is not that useful, let us add two more nodes and wait for them to show up (as NODE_DOWN):

$ replidev play node-start mongo/rs
--> Starting mongo/rs node play-node-8bMJWWs0 for cluster mongo-rs
--> Create pod play-node-8bMJWWs0
32593ddfdf9b0242d6290f962eb16eded166281cb059fac784fb6ee1ad3b2353
--> Start container play-node-8bMJWWs0-mongo
85b43eb866746c03f78bcd4efd7408724d9031e1e4af762782cf79204268a25c
--> Start container play-node-8bMJWWs0-repliagent
9c31fea7effdc97298efbc1c4573f744c3b5b5a92a44643047e3849f7e8eb719
$ replidev play node-start mongo/rs
--> Starting mongo/rs node play-node-q3mKXaG4 for cluster mongo-rs
--> Create pod play-node-q3mKXaG4
42a03c999b989153ae8d4ee0bd9aa9ce7ff45d73cfb545aad11623254fc5d695
--> Start container play-node-q3mKXaG4-mongo
fda39e282598843251638c9392516b1c8cf12be564d172ca68929c18af033ba7
--> Start container play-node-q3mKXaG4-repliagent
74d65f53899bc3e6ad778e8c53ff9ab214935352d4058186e758cf248e6ea4b6

And add them to the initialised replica set:

$ podman exec -it play-node-JfC9yNPm-mongo mongo --eval 'rs.add("podman-host:10002");'
$ podman exec -it play-node-JfC9yNPm-mongo mongo --eval 'rs.add("podman-host:10004");'

5. Start a zookeeper cluster

To demo replicante’s cross-store features we are going to also start a zookeeper cluster.

This is very similar to what we did for MongoDB so you could skip it if you are not interested:

$ cat - > zookeeper.demo.json <<EOS
[{
  "id": "0",
  "port": 10006,
  "election": 10007,
  "client": 10008
}, {
  "id": "1",
  "port": 10010,
  "election": 10011,
  "client": 10012
}, {
  "id": "2",
  "port": 10014,
  "election": 10015,
  "client": 10016
}]
EOS
$ replidev play node-start zookeeper --var-file 'cluster=zookeeper.demo.json' --var 'my_id=0'
$ replidev play node-start zookeeper --var-file 'cluster=zookeeper.demo.json' --var 'my_id=1'
$ replidev play node-start zookeeper --var-file 'cluster=zookeeper.demo.json' --var 'my_id=2'

6. Schedule some actions

Before you can interact with the local Replicante Core stack you need to set up replictl:

$ replictl context login --context play
Replicante API address: http://localhost:16016/
(Optional) API CA certificate file:
(Optional) API client certificate bundle:

$ replictl context select --context play
$ replictl context list
ACTIVE   NAME   URL                       CA BUNDLE   CLIENT KEY   NAMESPACE   CLUSTER   NODE  
*        play   http://localhost:16016/   Not set     Not set      Not set     Not set   Not set

# Set the namespace for the context once for all other commands.
$ replictl context change
Select a namespace (empty to clear selection): default
Select a cluster (empty to clear selection):
Select a node (empty to clear selection):

Actions are defined as YAML files. This is a bit much for an example ping action but provides many advantages for real use systems:

For our example we’ll schedule a ping action against any cluster/node:

$ cat - > ping.yaml <<EOS
apiVersion: replicante.io/v0
kind: AgentAction
metadata: {}
spec:
  action: agent.replicante.io/test.ping
EOS
$ replictl apply -f ping.yaml --cluster mongo-rs --node 'https://podman-host:10001'
$ replictl apply -f ping.yaml --cluster mongo-rs --node 'https://podman-host:10003'
$ replictl apply -f ping.yaml --cluster zookeeper --node 'https://podman-host:10009'

Note how the same action definition is applied to any nodes across any cluster.

Actions are scheduled during the next cluster refresh operation, and updated every refresh. As a result we’ll need two refresh cycles to happen before our ping action is marked as done.

While we could wait for these cycles to be scheduled, we can also request additional refresh tasks to be performed against a specific cluster. This is great for testing clusters and agents, like now, but when done too often it could increase the load on agents and nodes in the cluster.

$ replictl cluster refresh --cluster mongo-rs
Cluster refresh scheduled

7. Clean up everything

Once you are done experimenting with the playgrounds you can stop all processes and nodes.

Stop all the datastore nodes:

$ replidev play node-list
NODE                 CLUSTER     STORE PORT   CLIENT PORT   AGENT PORT   STATUS    POD ID  
play-node-ou1ZPcVe   zookeeper   10014        10016         10017        Running   0dc4ed954b0b  
play-node-epjE9kI8   zookeeper   10010        10012         10013        Running   26c235601056  
play-node-RJZBqGQf   zookeeper   10006        10008         10009        Running   47a69ad89a1f  
play-node-q3mKXaG4   mongo-rs    10004        10004         10005        Running   42a03c999b98  
play-node-8bMJWWs0   mongo-rs    10002        10002         10003        Running   32593ddfdf9b  
play-node-JfC9yNPm   mongo-rs    10000        10000         10001        Running   8d2312ace6da

$ replidev play node-stop play-node-ou1ZPcVe play-node-epjE9kI8 play-node-RJZBqGQf play-node-q3mKXaG4 play-node-8bMJWWs0 play-node-JfC9yNPm
--> Stop pod play-node-ou1ZPcVe
0dc4ed954b0b1798a6d00d492fa20bc44339d3056bf03464ea2737abcf150564
--> Remove pod play-node-ou1ZPcVe
0dc4ed954b0b1798a6d00d492fa20bc44339d3056bf03464ea2737abcf150564
--> Stop pod play-node-epjE9kI8
26c235601056c575549bf6e2dae4f478c85e3192a0ec39c182b3ea36eb61f1b5
--> Remove pod play-node-epjE9kI8
26c235601056c575549bf6e2dae4f478c85e3192a0ec39c182b3ea36eb61f1b5
--> Stop pod play-node-RJZBqGQf
47a69ad89a1f3a94aeac616b32eaa070de69dd469956bbf2a62979b1c559769e
--> Remove pod play-node-RJZBqGQf
47a69ad89a1f3a94aeac616b32eaa070de69dd469956bbf2a62979b1c559769e
--> Stop pod play-node-q3mKXaG4
42a03c999b989153ae8d4ee0bd9aa9ce7ff45d73cfb545aad11623254fc5d695
--> Remove pod play-node-q3mKXaG4
42a03c999b989153ae8d4ee0bd9aa9ce7ff45d73cfb545aad11623254fc5d695
--> Stop pod play-node-8bMJWWs0
32593ddfdf9b0242d6290f962eb16eded166281cb059fac784fb6ee1ad3b2353
--> Remove pod play-node-8bMJWWs0
32593ddfdf9b0242d6290f962eb16eded166281cb059fac784fb6ee1ad3b2353
--> Stop pod play-node-JfC9yNPm
8d2312ace6dab2e4fa5d4bbe4a8da166d9cbdda25caab2593f9416c2d896240a
--> Remove pod play-node-JfC9yNPm
8d2312ace6dab2e4fa5d4bbe4a8da166d9cbdda25caab2593f9416c2d896240a

Stop the replicante core stack:

$ replidev play replicore-stop
--> Stop pod play-replicore
812ac9e658ca9c36119bfa2375624f4b57f292fa8675d6d1826ba9551234c2a6
--> Remove pod play-replicore
812ac9e658ca9c36119bfa2375624f4b57f292fa8675d6d1826ba9551234c2a6

Stop the replidev play server by hitting Ctrl-C on the terminal where it is running.

And finally you can delete all node and core data with:

$ replidev play node-clean-all --confirm
--> Clean data for all nodes (from ./data/nodes/)

$ replidev play replicore-clean --confirm
--> Clean data for play-replicore pod (from ./data/replicore)