How To Setup A Private IPFS Cluster

I have been working on setting up a private IPFS cluster for our WeUseRTM Platform and figured I should document it since IPFS is such an integral part of not only the WeUseRTM platform but Raptoreum as well. IPFS is a peer-to-peer distributed file system and setting it up the way I will in this guide gives replication, and high availability of files added to assets on the WeUseRTM platform on a private network.

Here I will be setting up a three server IPFS cluster and using “raft” for consensus, with raft 50% of the cluster must be available at all times for it to continue functioning, so keep that in mind.

Installing IPFS Cluster

Pretty simple and straight forward, each server needs three components of IPFS:

1.) Main implementation of IPFS which is go-ipfs

2.) ipfs-cluster-service, this is the IPFS cluster peer

3.) ipfs-cluster-ctl needed for interaction with the cluster and cluster peer

These are are all their own system services and daemons. I recommend you don’t run as root, create a sudo user instead:

adduser username (follow the steps)
usermod -aG sudo

Change to user:

su - username

 

Grab IPFS, cluster-service, cluster-ctl Files And Wiggle Your Fingers

wget https://dist.ipfs.io/ipfs-cluster-service/v0.11.0/ipfs-cluster-service_v0.11.0_linux-amd64.tar.gz && tar -xzf ipfs-cluster-service_v0.11.0_linux-amd64.tar.gz
wget https://dist.ipfs.io/ipfs-cluster-ctl/v0.11.0/ipfs-cluster-ctl_v0.11.0_linux-amd64.tar.gz && tar -xzf ipfs-cluster-ctl_v0.11.0_linux-amd64.tar.gz
wget https://dist.ipfs.io/go-ipfs/v0.4.22/go-ipfs_v0.4.22_linux-amd64.tar.gz && tar -xzf go-ipfs_v0.4.22_linux-amd64.tar.gz
sudo cp ipfs-cluster-service/ipfs-cluster-service /usr/local/bin
sudo cp ipfs-cluster-ctl/ipfs-cluster-ctl /usr/local/bin
cd ~/go-ipfs
sudo ./install.sh

Confirm things are installed correctly:

ipfs-cluster-service help

ipfs-cluster-ctl help

ipfs help

 

Secret Key Setup

This is a private key and the secret key which is 32-bit hex encoded random string is what keeps it private. Only peers that have this key can communicate with the cluster. Generate it and display:

export CLUSTER_SECRET=$(od -vN 32 -An -tx1 /dev/urandom | tr -d ' \n')
echo $CLUSTER_SECRET

 

You will need this secret key for the other two peers so record it for now, we will use it in a bit. Now we need to initiate the ~/.ipfs-cluster folder:

ipfs-cluster-service init --consensus raft

Take note of the peer identity when running above command, you will need it when bootstrapping other peers.

Do the same for ipfs:

ipfs init

That will have created the needed folder and config file, we will come back to this. Let’s setup our firewall as well as Supervisor to monitor and restart the ipfs daemons if needed, such as after a reboot.

Install CSF (Config Server Firewall)

This is probably a little overkill but I prefer overkill rather than underkill with firewalls.

Few things required for CSF to be fully functional, on Ubuntu 18.04 fresh install this usually covers it:

apt install sendmail unzip dnsutils libwww-perl -y

CSF must be run with root so login as root or as sudo user do:

sudo su

cd /usr/src
rm -fv csf.tgz
wget https://download.configserver.com/csf.tgz
tar -xzf csf.tgz
cd csf
sh install.sh

Test it with:

csf -r

It should restart without any errors or complaints, now we have some config to do to make sure the cluster works smoothly.

nano /etc/csf/csf.conf

– Change TESTING value from 0 to 1
– Scroll down to the ports section and remove all ports from TCP and UDP for both IPv6 and IPv4 except the following TCP ports 22, 80, 443 (inbound and outbound). 22 for ssh and 443 for grabbing updates.
– Search (cntrl-w) for “IGNORE_ALLOW” and change its value from 0 to 1

Now restart csf for effect:

csf -r

Instead of leaving open ports that is needed by the cluster we close everything except the absolute essential and each server we whitelist the other server IPs which allows them through the firewall. Whitelist the other two server IPs like:

csf -a IPhere

One more step is to tell CSF never to ban the other servers IPs so we add the IPs to csf.ignore:

nano /etc/csf/csf.ignore

Install Supervisor And Setup Daemon Monitoring

sudo apt install supervisor -y

Add some configs for ipfs-cluster-service and ipfs:

sudo nano /etc/supervisor/supervisord.conf

Add these two parts:

[program:ipfs-cluster-service]
environment=IPFS_CLUSTER_PATH=/home/youruser/.ipfs-cluster
command=/usr/local/bin/ipfs-cluster-service daemon

[program:ipfs]
environment=IPFS_PATH=/home/youruser/.ipfs
command=ipfs daemon

Now lets let Supervisor know what’s up, this will start the service daemon but we need it stopped which the last command does. Reason is we want to see what it is doing and if it is starting like it should.

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl stop ipfs-cluster-service

Let’s start in direct in our session and make sure it is starting properly:

ipfs-cluster-daemon

After a minute you should see output stop with * CLUSTER READY *. Everything is good if you see that and you can start the service again:

sudo supervisorctl start ipfs-cluster-service

We are ready to move on to bootstrapping another peer and adding to the cluster, but if your like me and you lost track of the secret key you can grab it again like this:

cat .ipfs-cluster/service.json | grep secret

Bootstrapping Additional Peers (adding them to cluster)

Install and verify install exactly the same as we did the first time, but when you get to the secret key part you do:

export CLUSTER_SECRET=your_secret_key_from_first_peer

ipfs-cluster-service init --consensus raft

Now we run the daemon in current session with –bootstrap:
ipfs-cluster-service daemon –bootstrap /ip4/first_node_IP/tcp/9096/ipfs/peer_id

The peer identity is shown when you first run ipfs-cluster-service, if you are not sure what it is do:

sudo supervisorctl stop ipfs-cluster-service

ipfs-cluster-service

If everything is good you will see ** IPFS Cluster is READY ** followed by “Current Raft Leader”, then joined cluster. Kill it with cntrl-c add it to your Supervisor along with ipfs same as first node. You do not need to include –bootstrap flag unless the peer has been removed from cluster and is being re-added. Fire it up as normal:

sudo supervisorctl start ipfs-cluster-service

 

Time To Take It For A Test Drive!

 

Create a testfile:

mkdir test_file
echo WeUseRTM is going to roxxor soxxors! > smelly_soxx.txt
ipfs add smelly_soxx.txt

Check to see if it is on the other peers:

ipfs cat files_hash

If it is there it will return the contents of the file, in this case that is “WeUseRTM is going to roxxor soxxors!”. Also in the screen you see that I ran a search for the file hash just as a check to make sure the cluster was indeed private.

Accessing IPFS Private Cluster API From External App

WeUseRTM runs outside of the IPFS cluster and the Cluster by default makes the needed API accessible only on 127.0.0.1. It took us a few tries to figure out just which API we wanted but we found that this was the one: (.ipfs-cluster/service.json)

"api": {
"ipfsproxy": {
"listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",

Change 127.0.0.1 to your public IPv4 or to 0.0.0.0 to make it bind to all available IP, then restart the ipfs-ctl service. This SHOULD NOT be open to public make sure you whitelist only the needed IP for access to that port.

 

Upgrading The IPFS Cluster

 

These are my notes to upgrade, at some point I will make it into a proper script 😛

 

Note: version mismatch will make the other nodes unable to connect until they are up to date

wget https://dist.ipfs.io/ipfs-cluster-service/v0.12.1/ipfs-cluster-service_v0.12.1_linux-amd64.tar.gz && tar -xzf ipfs-cluster-service_v0.12.1_linux-amd64.tar.gz
wget https://dist.ipfs.io/ipfs-cluster-ctl/v0.12.1/ipfs-cluster-ctl_v0.12.1_linux-amd64.tar.gz && tar -xzf ipfs-cluster-ctl_v0.12.1_linux-amd64.tar.gz
wget https://dist.ipfs.io/go-ipfs/v0.4.23/go-ipfs_v0.4.23_linux-amd64.tar.gz && tar -xzf go-ipfs_v0.4.23_linux-amd64.tar.gz
sudo supervisorctl stop ipfs-cluster-service
sleep 3
sudo cp ipfs-cluster-service/ipfs-cluster-service /usr/local/bin
sudo cp ipfs-cluster-service/ipfs-cluster-service /usr/local/bin
sudo cp ipfs-cluster-ctl/ipfs-cluster-ctl /usr/local/bin
cd ~/go-ipfs
sudo ./install.sh
sudo supervisorctl start ipfs-cluster-service
ipfs-cluster-ctl version
ipfs version
ipfs-cluster-service version

Check all peers connectable from first node updated:

ipfs-cluster-ctl peers ls

If you see any of this “ERROR: protocol not supported” the related node is not running a good version.