Editing Patroni Cluster (section)

=== etcd ===
<pre>
sudo mkdir -p /var/lib/etcd/
sudo mkdir -p /etc/etcd
sudo groupadd --system etcd
sudo useradd -s /sbin/nologin --system -g etcd etcd
sudo chown -R etcd:etcd /var/lib/etcd/
sudo chmod -R a+rw /var/lib/etcd
</pre>

Create the configuration file at /etcd/etcd/etcd.yaml </br>
<pre>
name: 'debian1' 
data-dir: '/var/lib/etcd/data.etcd'
initial-advertise-peer-urls: http://192.168.59.101:2380 
listen-peer-urls: http://192.168.59.101:2380 
advertise-client-urls: http://192.168.59.101:2379 
listen-client-urls: http://192.168.59.101:2379,http://127.0.0.1:2379 
initial-cluster: "debian1=http://192.168.59.101:2380,debian2=http://192.168.59.102:2380,debian3=http://192.168.59.103:2380" 
initial-cluster-state: 'new' 
initial-cluster-token: 'token-01'
</pre>

Configure the systemd unit file </br>
<pre>
[Unit]
Description=etcd key-value store
Documentation=https://github.com/etcd-io/etcd
After=network-online.target local-fs.target remote-fs.target time-sync.target
Wants=network-online.target local-fs.target remote-fs.target time-sync.target

[Service]
User=etcd
Type=notify
Environment=ETCD_DATA_DIR=/var/lib/etcd
Environment=ETCD_NAME=%H
ExecStart=/usr/bin/etcd --config-file /etc/etcd/etcd.yml
Restart=always
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable etcd.service
sudo systemctl start etcd.service
</pre>

Test that everything is working </br>
<pre>
export ETCDCTL_ENDPOINTS="http://192.168.59.103:2379,http://192.168.59.101:2379,http://192.168.59.102:2379"
etcdctl member list -w table
etcdctl endpoint health -w table
etcdctl endpoint status -w table 
</pre>
<pre>
+----------------------------+------------------+---------+---------+-----------+-----------+------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+--------+
| http://192.168.59.102:2379 | 3cdde56ec82a0ca0 |  3.5.16 |   53 kB |      true |        13 |        145 |        |
| http://192.168.59.101:2379 | 406189a3bb67b8bc |  3.5.16 |   53 kB |     false |        13 |        145 |        |
| http://192.168.59.103:2379 | 939ed44c2e6d5e5b |  3.5.16 |   53 kB |     false |        13 |        145 |        |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+--------+
</pre>

Change the leader if necessary </br>
<pre>
etcdctl move-leader 939ed44c2e6d5e5b --endpoints=$ETCDCTL_ENDPOINTS
</pre>

==== Issues ====

One of the members would not connect to the cluster. </br>
The error message on the leader was “Prober detected unhealthy status …” </br>
On the problem member the message was “Failed to publish local member to cluster through raft” </br>
No network issues, solved by removing and adding the member. </br>
<pre>
etcdctl member list -w table
sudo systemctl stop etcd (On server being removed)
etcdctl member remove <ID>
etcdctl member add debian2 --peer-urls=http://192.168.59.102:2380
</pre>
Edit the etcd.yml files copying all information displayed, including the new order for ETCD_INITIAL_CLUSTER </br>
On the server being added ..
<pre>
sudo systemctl start etcd
</pre>