IOTA Full Node (IRI) Server
Copy-Paste Installation Guide including Conditional Proxy, Monitoring, Visualization and Metrics



Grafana dashboard

This tutorial provides copy-paste Linux console commands to install and run a complete and stable IOTA full node within a few minutes. All you need is some time, patience, a rental server and basic console knowledge. I have consciously chosen not to use automated installation tools like Ansible or Chef. This tutorial is designed so that it changes your server installation as little as possible and therefore applications are installed under the home directory of the user `iota`.

Make sure to follow the instructions precisely.

Discord is the primary communication platform for the IOTA community. If you have not already joined use this link. In the channel #fullnodes you can get support in English and in the channel #fullnodes-ger in German.

Follow us on twitter to keep up to date and let the community know that you are about to install a full node server!     


1. Server

A VPS (Virtual Private Server) with 2 cores and 4GB RAM (8GB / 4 cores would perform better) is currently sufficient. An SSD hard disk is highly recommended but not mandatory.

On the right hand side you can see a small list of EU / German host providers who already have nodes running on their machines.

For the sake of simplicity we will focus on Ubuntu. Please consider the following points:

  • minimal Ubuntu server installation 18.04. or higher
  • SSH access with a key or password
    if you don't know what to do with keys, choose a password (default)
  • A public IP (IPv4) address

Ignore or disable other options such as firewall, plesk, support, etc. as it is unnecessary and can cause problems.

2. Settings

Some formalities for the beginning

Server IP (IPv4) Address

Enter the public IPv4 address of your server

Neighbors

We'll use Nelson which automatically takes care of neighbors. If you already have neighbors you can enter them here (space-separated) and continue to use them in conjunction with nelson otherwise leave the input field empty. If you don't want to install Nelson just add (3-5) neighbors here (see Discord #nodesharing) and skip step 9. Nelson. Neighbors must add each other using the same protocol: udp or tcp .


Node Name
Reverse proxy and Free SSL certificate (HTTPS)

It is recommended to run the node behind a reverse proxy with SSL not only for security reasons.
The official IOTA Trinity Wallet can only connect to nodes that are SSL secured.


3. Ubuntu Installation

Connect to your server using Secure Shell (SSH).

Windows users should use the PUTTY SSH client. If you want to modify commands or config files in a text editor on Windows you should use Notepad++. Ordinary editors (Notepad) can insert invisible, problematic special characters.

User / sudo

For servers that has installed ubuntu as main user we have to execute many commands with sudo. If your user is root you don't need to do that.
As sudo works for both types of users we simply use sudo for all commands.

Initial log-in

Depending on your host provider you have to log in either with ubuntu or root. On the first login you will probably get a hint about known hosts. You can simply confirm this with yes.

ssh root@[[ click here to enter your server ip address ]]

Update

We want to update the OS first. A kernel update sometimes requires a reboot - follow the instructions in the terminal.
Make a reboot if required by entering sudo reboot and reconnect with ssh.

sudo apt-get update -qqy --fix-missing && sudo DEBIAN_FRONTEND=noninteractive apt-get -o DPkg::options::="--force-confdef" -o DPkg::options::="--force-confold" upgrade -y && sudo apt-get clean -y && sudo apt-get autoremove -y --purge

Packages

We need some packages such as Java. Oracle has proven to be the better choice because many people have problems with OpenJDK.

echo "oracle-java11-installer shared/accepted-oracle-license-v1-2 select true" | sudo debconf-set-selections && echo "oracle-java11-installer shared/accepted-oracle-license-v1-2 seen true" | sudo debconf-set-selections && sudo apt-get install software-properties-common -y && sudo add-apt-repository ppa:linuxuprising/java -y && sudo apt-get update -yqq --fix-missing && sudo apt-get install oracle-java11-installer curl wget jq apache2-utils git apt-transport-https -y && sudo apt-get install oracle-java11-set-default -y

Settings

We want to tell our OS which java we want to use by default

sudo sh -c 'echo JAVA_HOME="/usr/lib/jvm/java-11-oracle" >> /etc/environment' && source /etc/environment

IOTA user

For security reasons we do not start our node with the root- (admin-) user but create an unprivileged user iota

sudo useradd -s /bin/bash -m iota

Directories

Create the directories for the node (IRI) application

sudo -u iota mkdir -p /home/iota/node/ixi /home/iota/node/mainnetdb

Increase open file limit

Under certain circumstances the default values are not sufficient

cat << EOF | sudo tee /etc/security/limits.conf > /dev/null
*    soft nproc  65535
*    hard nproc  65535
root soft nproc  65535
root hard nproc  65535
*    soft nofile 65535
*    hard nofile 65535
root soft nofile 65535
root hard nofile 65535
EOF

Load the settings without rebooting

sudo sysctl -p

IRI Installation

We download the official and current version from github into the iota node directory

4. Systemd Service

Of course we want our node to start automatically after a reboot or crash and therefore create a systemd service for it.
Just copy and paste everything into the console.

Add the IRI_JVM_OPTS environment variable to the file /home/iota/node/iota.env

sudo -u iota touch /home/iota/node/iota.env && echo "IRI_JVM_OPTS="`bash <(curl -s https://gist.githubusercontent.com/zoran/5ef05b2674a1acc1fc66940286d0fdfb/raw)` | sudo tee /home/iota/node/iota.env

Just so the IRI_VERSION environment variable.

echo "IRI_VERSION="`curl -s https://api.github.com/repos/iotaledger/iri/releases/latest | jq "[.assets][0][0].name" --raw-output` | sudo tee -a /home/iota/node/iota.env

Install the iota service

cat << 'EOF' | sudo tee /lib/systemd/system/iota.service > /dev/null
[Unit]
Description=IOTA (IRI) full node
After=network.target

[Service]
WorkingDirectory=/home/iota/node
EnvironmentFile=/home/iota/node/iota.env
User=iota
LimitNOFILE=65535
LimitNPROC=65535
TasksMax=infinity
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=120
ExecStart=/usr/bin/java ${IRI_JVM_OPTS} -Djava.net.preferIPv4Stack=true -jar ${IRI_VERSION} -c iota.ini
SyslogIdentifier=IRI
Restart=on-failure
RestartSec=120

[Install]
WantedBy=multi-user.target
Alias=iota.service

EOF

We are now enabling the new service. Every time you change something in this file you need to run sudo systemctl daemon-reload.

sudo systemctl daemon-reload && sudo systemctl enable iota.service

IRI Auto Update

We check every 15 minutes if there is a newer version of IRI and install it automatically.

echo '*/15 * * * * root bash -c "bash <(curl -s https://gist.githubusercontent.com/zoran/d1cd864801e68c89569f7e5e32eeae9f/raw)"' | sudo tee /etc/cron.d/iri_updater > /dev/null

5. IRI and Reverse Proxy with SSL Certificate

We need 3 ports that are freely configurable. PORT is used to communicate (locally) with the API of the IRI. UDP_RECEIVER_PORT and TCP_RECEIVER_PORT are used to communicate with our neighbors. There is no reason to change these ports here.

IRI config

cat << EOF | sudo -u iota tee /home/iota/node/iota.ini > /dev/null
[IRI]
PORT = 14267
UDP_RECEIVER_PORT = 14600
TCP_RECEIVER_PORT = 15600
API_HOST = 127.0.0.1
IXI_DIR = ixi
DEBUG = false
TESTNET = false
DB_PATH = mainnetdb
REMOTE_LIMIT_API = ""
ZMQ_ENABLED = true
ZMQ_PORT = 5556


#------------------------------------------------------------------------
# Local Snapshots Settings
#------------------------------------------------------------------------

# This parameter must be set to 'true' for the IRI to read any other 'LOCAL_SNAPSHOTS' parameters
LOCAL_SNAPSHOTS_ENABLED = true

# Enable the deletion of confirmed transactions from the ledger.
# Confirmed transactions are deleted if they were confirmed by a milestone that
# is older than the sum of 'LOCAL_SNAPSHOTS_DEPTH' + 'LOCAL_SNAPSHOTS_PRUNING_DELAY'
LOCAL_SNAPSHOTS_PRUNING_ENABLED = true

# Amount of milestone transactions to keep in the ledger.
# Default = 100. Minimum = 100
LOCAL_SNAPSHOTS_DEPTH = 100

# Amount of milestone transactions to keep in the ledger.
# Default = 40000. Minimum = 10000. Note: If you are operating a public facing
# node and use local snapshots, please set your 'LOCAL_SNAPSHOTS_PRUNING_DELAY'
# value to at least 40000
LOCAL_SNAPSHOTS_PRUNING_DELAY = 40000

# Interval, in milestone transactions, at which local snapshots are taken
# if the ledger is fully synchronized
LOCAL_SNAPSHOTS_INTERVAL_SYNCED = 10

# Interval, in milestone transactions, at which local snapshots are taken
# if the ledger is not fully synchronized
LOCAL_SNAPSHOTS_INTERVAL_UNSYNCED = 1000


#------------------------------------------------------------------------
# Static Neighbors
#------------------------------------------------------------------------
NEIGHBORS = 
EOF

Obtain and install a free Let’s Encrypt SSL certificate

Make sure you have entered a correct email address and domain name.

cd ~ && wget https://dl.eff.org/certbot-auto && \
  chmod a+x certbot-auto && \
  sudo mv certbot-auto /usr/local/bin && \
  sudo certbot-auto --noninteractive --os-packages-only && \
sudo certbot-auto certonly \
  --standalone \
  --agree-tos \
  --non-interactive \
  --text \
  --rsa-key-size 4096 \
  --email [[ mandatory email address not entered! click here ]] \
  --domains '[[ mandatory (FQDN) domain name not entered! click here ]]'

Install a script that automatically renews your certificate
With this you can ignore the "Let's Encrypt certificate expiration notice" emails.

echo "0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && /usr/local/bin/certbot-auto renew && /bin/systemctl reload openresty" | sudo tee /etc/cron.d/cert_renew > /dev/null

Install openresty (Nginx + Lua) which we use as a reverse proxy

Openresty (Nginx) config

sudo mkdir -p /usr/local/openresty/nginx/conf/ && cat << 'EOF' | sudo tee /usr/local/openresty/nginx/conf/nginx.conf > /dev/null
user                          iota;
worker_processes              auto;
worker_rlimit_nofile          40960;
error_log                     logs/error.log;
pid                           /usr/local/openresty/nginx/logs/nginx.pid;

events {
  worker_connections          40960;
  multi_accept                on;
}

http {
  default_type                application/json;
  keepalive_timeout           5m;
  send_timeout                6m;
  init_by_lua                 'require "cjson"';
  ssl_session_cache           shared:SSL:32m;
  ssl_session_timeout         5m;
  server_tokens               off;
  add_header                  X-XSS-Protection '1; mode=block';
  add_header                  X-Content-Type-Options nosniff;

  log_format                  main '$remote_addr - $remote_user [$time_local]  $status '
                                '"$request" $body_bytes_sent "$http_referer" '
                                '"$http_user_agent" "$http_x_forwarded_for"';

  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }

  upstream iri {
    server                    127.0.0.1:14267;
  }

  upstream grafana {
    server                    127.0.0.1:3000;
  }

  upstream prometheus {
    server                    127.0.0.1:9090;
  }

  upstream iota_exporter {
    server                    127.0.0.1:9311;
  }

  upstream ipm {
    server                    127.0.0.1:8888;
  }

  proxy_redirect              off;
  proxy_set_header            Host $host;
  proxy_set_header            X-Real-IP $remote_addr;
  proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_connect_timeout       2m;
  proxy_send_timeout          2m;
  proxy_read_timeout          2m;
  proxy_buffers               32 4k;

  client_max_body_size        1m;
  client_body_buffer_size     128k;

  limit_req_zone              $binary_remote_addr zone=iri:5m rate=50r/s;
  limit_req_zone              $binary_remote_addr zone=grafana:5m rate=120r/s;
  limit_req_zone              $binary_remote_addr zone=prometheus:5m rate=55r/s;
  limit_req_zone              $binary_remote_addr zone=iota_exporter:5m rate=55r/s;
  limit_req_zone              $binary_remote_addr zone=ipm:5m rate=15r/s;

  server {
    listen                    14265 default_server deferred;
    listen                    443 ssl http2 deferred;
    server_name               [[ mandatory (FQDN) domain name not entered! click here ]];


    ssl_certificate           /etc/letsencrypt/live/[[ mandatory (FQDN) domain name not entered! click here ]]/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/[[ mandatory (FQDN) domain name not entered! click here ]]/privkey.pem;
    ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers               HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    add_header                Strict-Transport-Security 'max-age=63072000; includeSubdomains';

    ssl_stapling              on;
    ssl_stapling_verify       on;
    ssl_trusted_certificate   /etc/letsencrypt/live/[[ mandatory (FQDN) domain name not entered! click here ]]/fullchain.pem;
    resolver                  1.1.1.1 8.8.8.8 8.8.4.4 9.9.9.9 valid=300s;
    resolver_timeout          5s;

    error_page 405 @error405;
    location @error405 {
      add_header Allow 'GET, HEAD, OPTIONS, POST' always;
    }

    location /grafana/ {
      limit_req               zone=grafana burst=150;
      limit_req_log_level     warn;
      limit_req_status        444;

      proxy_pass              http://grafana/;
    }

    location /prometheus/ {
      auth_basic              "Prometheus";
      auth_basic_user_file    /usr/local/openresty/nginx/conf/.htpasswd;

      limit_req               zone=prometheus burst=70;
      limit_req_log_level     warn;
      limit_req_status        444;

      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      sub_filter_once         off;
      sub_filter              '="/' '="/prometheus/';
      sub_filter              'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/prometheus";';

      rewrite                 ^/prometheus/?$ /prometheus/graph redirect;
      rewrite                 ^/prometheus/(.*)$ /$1 break;

      proxy_pass              http://prometheus/;
    }

    location /iota_exporter/ {
      auth_basic              "IOTA Prometheus Exporter";
      auth_basic_user_file    /usr/local/openresty/nginx/conf/.htpasswd;

      limit_req               zone=iota_exporter burst=70;
      limit_req_log_level     warn;
      limit_req_status        444;

      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      proxy_pass              http://iota_exporter/;
    }

    location /ipm/ {
      auth_basic              "IOTA Peer Manager";
      auth_basic_user_file    /usr/local/openresty/nginx/conf/.htpasswd;

      limit_req               zone=ipm burst=20;
      limit_req_log_level     warn;
      limit_req_status        444;

      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      sub_filter_once         off;
      sub_filter              '="/' '="/ipm/';
      sub_filter              'var PATH_PREFIX = "";' 'var PATH_PREFIX = "/ipm";';

      rewrite                 ^/ipm/(.*)$ /$1 break;

      proxy_pass              http://ipm/;
    }

    location /socket.io/ {
      auth_basic_user_file    /usr/local/openresty/nginx/conf/.htpasswd;

      limit_req               zone=ipm burst=20;
      limit_req_log_level     warn;
      limit_req_status        444;

      proxy_http_version      1.1;
      proxy_set_header        Upgrade $http_upgrade;
      proxy_set_header        Connection "upgrade";
      proxy_redirect          off;


      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;

      proxy_pass              "http://127.0.0.1:8888";
    }

    location / {
      limit_req               zone=iri burst=10;
      limit_req_log_level     warn;
      limit_req_status        444;

      if ( $request_method !~ ^(HEAD|OPTIONS|POST)$ ) {
        return 405;
      }

      if ( $request_method = OPTIONS ) {
        proxy_pass http://iri;
      }

      if ( $request_method = POST ) {
        set $upstream '';
        access_by_lua_block {
          ngx.req.read_body()
          local cjson            = require('cjson')
          local data             = ngx.req.get_body_data()
          local json_data        = cjson.decode(data)
          local req_command      = json_data["command"]
          local allowed_pub_commands = {
            'getNodeInfo',
            'getTips',
            'findTransactions',
            'getTrytes',
            'getInclusionStates',
            'getBalances',
            'getTransactionsToApprove',
            'attachToTangle',
            'interruptAttachingToTangle',
            'broadcastTransactions',
            'storeTransactions',
            'wereAddressesSpentFrom',
            'checkConsistency'
          }

          local function has_value (tab, val)
            for k, v in pairs(tab) do
              if v == val then
                return true
              end
            end

            return false
          end

          if has_value(allowed_pub_commands, req_command) then
            ngx.var.upstream = "iri"
          else
            ngx.exit(405)
          end
        }

        proxy_pass http://$upstream;
      }
    }
  }
}
EOF

Install dependecies and Openresty

sudo sh -c 'echo "export LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib" >> /etc/environment' && source /etc/environment && sudo ldconfig && \
sudo wget -qO - https://openresty.org/package/pubkey.gpg | sudo apt-key add - && \
sudo apt-get -y install zlib1g-dev libpcre3-dev libssl-dev software-properties-common && \
sudo add-apt-repository -y "deb http://openresty.org/package/ubuntu $(lsb_release -sc) main" && \
sudo apt-get update -yqq --fix-missing && sudo apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install -y openresty

Openresty (Nginx) logfiles should automatically rotate daily and be stored for 31 days.

cat << 'EOF' | sudo tee /etc/logrotate.d/openresty > /dev/null
/usr/local/openresty/nginx/logs/*.log {
  monthly
  missingok
  rotate 3
  compress
  delaycompress
  notifempty
  sharedscripts
  postrotate
    /bin/systemctl reload openresty
  endscript
}
EOF

Create an admin password file for restricted HTTP(S) access (Prometheus and IOTA Peer Manager)

sudo htpasswd -bc /usr/local/openresty/nginx/conf/.htpasswd admin [[ mandatory password not entered! click here ]]

6. IRI Database

It can take a long time to synchronize a database from scratch therefore we use a local snapshot to synchronize the node quickly

(Re-)Create the Database

You can also use this line to replace a corrupted database or if your node keeps falling back while synchronizing.

sudo systemctl stop iota && cd /home/iota/node && sudo rm -fr mainnetdb/* mainnet.snapshot.meta mainnet.snapshot.state mainnet.snapshot.gc spent-addresses-db spent-addresses-log && curl https://db.iota.partners/iri-mainnet-snapshot.tar.gz | sudo -s -H -u iota tar xz -C /home/iota/node/ && sudo systemctl start iota

Proceed to 7. Nodejs and PM2 Process Manager after you have executed the command.
If you have just replaced the database and your node is already fully setup you don't have to do anything.

Optional: Follow https://db.iota.partners/iri-mainnet-snapshot.tar.gz if you only want to download the database without following this tutorial.

Optionally also download the sha256 checksum for the database and check your download for integrity (all files must be in the same dir)

wget https://db.iota.partners/iri-mainnet-snapshot.tar.gz.sha256 && sha256sum --check iri-mainnet-snapshot.tar.gz.sha256

7. Nodejs and PM2 Process Manager

We install nodejs 9 but make sure that other versions are uninstalled first. Ignore npm warnings.

sudo apt-get purge nodejs -y && sudo curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add - && sudo sh -c "echo deb https://deb.nodesource.com/node_9.x bionic main > /etc/apt/sources.list.d/nodesource.list" && sudo apt-get update -yqq --fix-missing && sudo apt-get install -y nodejs && sudo npm i npm@latest -g

Install the pm2 process manager and the pm2 service that will take care of all our Node.js applications which looks like this:

pm2 list
sudo su - iota -c '
  mkdir -p /home/iota/.npm-global;
  npm config set prefix '~/.npm-global';
  npm config set loglevel="error";
  echo "export PATH=/home/iota/.npm-global/bin:/home/iota/.npm-global/lib/node_modules/pm2/bin:$PATH" >> /home/iota/.profile;
  source ~/.profile;
  npm install -g pm2;'
sudo env PATH=$PATH:/usr/bin /home/iota/.npm-global/lib/node_modules/pm2/bin/pm2 startup systemd -u iota --hp /home/iota

8. IOTA Peer Manager

Install the peer manager

sudo -i -u iota npm install -g iota-pm

Start and create a PM2 service for it

sudo -i -u iota pm2 start iota-pm -- -i http://127.0.0.1:14267 -p 127.0.0.1:8888 -r 5 && sudo -i -u iota pm2 save

Visit the IOTA Peer Manager dashboard. If you haven't entered any static neighbors you will only see some after the nelson installation.

User: admin, Password: [[ mandatory password not entered! click here ]]

https://[[ click here to enter your (FQDN) domain name ]]/ipm

9. Nelson

Nelson takes care of neighbors automatically.

Install Nelson and ignore possible warnings

sudo -i -u iota npm install -g nelson.cli

Nelson config

cat << EOF | sudo -u iota tee /home/iota/node/nelson.ini > /dev/null
[nelson]
name = 
cycleInterval = 240
epochInterval = 1200
apiPort = 18600
apiHostname = 127.0.0.1
port = 16600
IRIHostname = 127.0.0.1
IRIProtocol = any
IRIPort = 14267
TCPPort = 15600
UDPPort = 14600
dataPath = /home/iota/node/nelson/data/neighbors.db
incomingMax = 4
outgoingMax = 3
isMaster = false
silent = false
gui = false

getNeighbors = https://gitlab.com/semkodev/nelson.cli/raw/master/ENTRYNODES

; Protect API with basic auth
[nelson.apiAuth]
username = admin
password = 

EOF

Start Nelson

sudo -i -u iota pm2 start nelson -- --config /home/iota/node/nelson.ini && sudo -i -u iota pm2 save

10. Prometheus

We use Prometheus for long-term statistics and monitoring.
In the next steps we will also install so-called exporters to collect metrics from IOTA IRI or other applications.

Prometheus

Install Prometheus

curl -L https://github.com/prometheus/prometheus/releases/download/v2.7.1/prometheus-2.7.1.linux-amd64.tar.gz | sudo -i -u iota tar xz -C /home/iota/node

Configuration file

cat << EOF | sudo -i -u iota tee /home/iota/node/prometheus.yml > /dev/null
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'node'
    scrape_interval: 5s
    static_configs:
      - targets:
        - '127.0.0.1:9100'

  - job_name: 'iota_exporter'
    scrape_interval: 5s
    static_configs:
      - targets:
        - '127.0.0.1:9311'
EOF

Prometheus Service

cat << EOF | sudo tee /lib/systemd/system/prometheus.service > /dev/null
[Unit]
Description=Prometheus
Wants=network-online.target
After=network.target

[Service]
WorkingDirectory=/home/iota/node/prometheus-2.7.1.linux-amd64
User=iota
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=120
ExecStart=/home/iota/node/prometheus-2.7.1.linux-amd64/prometheus --web.enable-admin-api --web.listen-address="127.0.0.1:9090" --config.file=/home/iota/node/prometheus.yml
Restart=on-failure
RestartSec=120

[Install]
WantedBy=multi-user.target
Alias=prometheus.service

EOF

Install and start service

sudo systemctl daemon-reload && sudo systemctl enable prometheus.service && sudo systemctl start prometheus

Prometheus node exporter

Install the'node' exporter which provides us with server metrics.

curl -sL https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz | sudo -i -u iota tar xz -C /home/iota/node && sudo -i -u iota pm2 start /home/iota/node/node_exporter-0.17.0.linux-amd64/node_exporter -- --web.listen-address='127.0.0.1:9100' && sudo -i -u iota pm2 save

IOTA exporter (by crholliday)

Install git, clone and install the IOTA exporter from GitHub (ignore warnings / zmq errors)

sudo apt-get install build-essential git python gcc make g++ libc6-dev libzmq3-dev -y && sudo rm -fr /home/iota/node/iota-prom-exporter && sudo -i -u iota git clone --depth=1 https://github.com/crholliday/iota-prom-exporter.git /home/iota/node/iota-prom-exporter && sudo -i -u iota npm install -g /home/iota/node/iota-prom-exporter --silent

IOTA exporter config

cat << EOF | sudo -i -u iota tee /home/iota/node/iota-prom-exporter/config.js > /dev/null
let path = require('path')
global.rootPath = path.normalize(path.join(__dirname, '..', '..'))

module.exports = {
  iota_node_url: process.env.iota_node_url || 'http://127.0.0.1:14267',
  bind_address: process.env.bind_address || '127.0.0.1',
  bind_port: process.env.bind_port || 9311,
  zmq_url: process.env.zmq_url || '127.0.0.1:5556',
  zmq_restart_interval: process.env.zmq_restart_interval || 5,
  market_info_flag: process.env.market_info_flag || '',
  confirm_time_buckets: process.env.confirm_time_buckets || [300, 600, 1200, 2400, 3600, 7200, 21600, 43200],
  prune_interval_days: process.env.prune_interval_days || 1,
  retention_days: process.env.retention_days || 30
}

EOF

Install and start the pm2 service for the IOTA exporter

sudo -i -u iota pm2 start --name iota-prom-exporter /home/iota/node/iota-prom-exporter/app.js && sudo -i -u iota pm2 save

The exporter seems to have a problem at the moment, possibly a memory leak which can take more than 2GB RAM.
As work around we restart (reload) the exporter every hour which has no negative effects.

echo "0 * * * * iota bash -c '. /home/iota/.profile; pm2 reload iota-prom-exporter'" | sudo tee /etc/cron.d/iota-prom-exporter-restarter > /dev/null

11. Grafana

Grafana is the leading graph and dashboard builder for visualizing time series infrastructure and application metrics
First enter a password:  [[ mandatory password not entered! click here ]]

Install dependencies, download and install Grafana.

sudo apt-get install -y wget libfontconfig && cd /tmp && curl -sLO https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.4.3_amd64.deb && sudo dpkg -i grafana_5.4.3_amd64.deb && sudo rm grafana_5.4.3_amd64.deb

Grafana config

cat << EOF | sudo tee /etc/grafana/grafana.ini > /dev/null
app_mode = production
instance_name = [[ mandatory (FQDN) domain name not entered! click here ]]

[server]
http_port = 3000
domain = localhost
root_url = http://127.0.0.1/grafana
enable_gzip = true

[security]
# default admin user, created on startup
admin_user = admin
admin_password = [[ mandatory password not entered! click here ]]
login_remember_days = 365
cookie_username = grafana_user
cookie_remember_name = grafana_remember

[users]
allow_sign-up = false
allow_org_create = false

[auth.anonymous]
enabled = false

EOF

Enablethe Grafana service

sudo systemctl daemon-reload && sudo systemctl enable grafana-server

Start the Grafana server

sudo systemctl start grafana-server && sleep 15

Install the default datasource

Should this command fail you have to wait a few seconds until the grafana server is up and try again.

sudo curl -sH "Content-Type: application/json" -X POST -u admin:[[ mandatory password not entered! click here ]] -d '{"name":"Prometheus","type":"prometheus","typeLogoUrl":"","access":"proxy","url":"http://127.0.0.1:9090","basicAuth":false,"isDefault":true}' http://127.0.0.1:3000/api/datasources

Download the dashboards ('Host' and 'IOTA')

cd /etc/grafana && sudo wget -O /etc/grafana/prometheus-dashboard-node.json https://gist.githubusercontent.com/zoran/c4ddad312f081f5ca7d17b61b3532ce9/raw
cd /etc/grafana && sudo wget -O /etc/grafana/prometheus-dashboard-iota.json https://gist.githubusercontent.com/zoran/b4104df90d3cf762842ed663583952b0/raw

Install the dashboards ('Host' and 'IOTA')

sudo curl -sH "Content-Type: application/json" -X POST -u admin:[[ mandatory password not entered! click here ]] -d @/etc/grafana/prometheus-dashboard-node.json http://127.0.0.1:3000/api/dashboards/db
sudo curl -sH "Content-Type: application/json" -X POST -u admin:[[ mandatory password not entered! click here ]] -d @/etc/grafana/prometheus-dashboard-iota.json http://127.0.0.1:3000/api/dashboards/db

Use the IOTA instead of Grafana logo

sudo curl -s https://raw.githubusercontent.com/iotaledger/wallet/master/ui/img/iota-logo.svg --output /usr/share/grafana/public/img/grafana_icon.svg && sudo cp /usr/share/grafana/public/img/grafana_icon.svg /usr/share/grafana/public/img/icn-app.svg

Grafana Dashboards

Now visit your new 'IOTA' and 'Host' dashbobards. Refresh the page after a few seconds so that Grafana displays the new dashboard correctly.
The IOTA dashboard offers beside the IOTA also the most important host metrics. The Host dashboard, on the other hand, provides all the metrics your system provides. Depending on whether your server is virtual or bare metal, some metrics remain empty.
It can take several minutes until all metrics show a value, because some data has to be collected first.

User: admin, Password: [[ mandatory password not entered! click here ]]
https://[[ click here to enter your (FQDN) domain name ]]/grafana/dashboards

On the alerting page you can see if certain metrics are not within the expected range
https://[[ click here to enter your (FQDN) domain name ]]/grafana/alerting/list

12. Operation

If you have reached this step, your node is fully installed. Here you will find some useful commands.
Usually you can exit programs on the console with q, Esc or ctrl+c.

Monitoring

Browse the IRI logfile

sudo journalctl -u iota

Display IRI log output live (exit with CTRL+c)

sudo journalctl -u iota -f

Monitor the Nginx (Openresty) access and error logs

tail -f /usr/local/openresty/nginx/logs/*.log

IOTA Trinity Wallet

Use your node for your wallet (don't append the port :443 to the address)

https://[[ mandatory (FQDN) domain name not entered! click here ]]

IOTA.Partners perma node for the Trinity wallet

If you need a fast reliable perma node (cluster) with fast POW for your Trinity wallet
you can select it from the node list in Trinity or enter it manually (don't append the port :443 to the address):

https://perma.iota.partners

PM2 usage

pm2 commands can be executed directly without switching to the user iota in this way

sudo -u iota -H -s eval 'source ~/.profile; pm2 list'
sudo -u iota -H -s eval 'source ~/.profile; pm2 log < id or App name from pm2 list >'
sudo -u iota -H -s eval 'source ~/.profile; pm2 restart < id or App name from pm2 list >'
sudo -u iota -H -s eval 'source ~/.profile; pm2 stop < id or App name from pm2 list >'

IRI API

After a restart it can take 1-2 minutes until the API is available.

Check the latestMilestoneIndex and latestSolidSubtangleMilestoneIndex. Your node is synchronized if these two figures are identical.

curl -s http://127.0.0.1:14267 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1' -d '{"command": "getNodeInfo"}' | jq "{latestMilestoneIndex, latestSolidSubtangleMilestoneIndex}"

Show IRI status

non-HTTPS locally

curl -s http://127.0.0.1:14267 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1' -d '{"command": "getNodeInfo"}' | jq

non-HTTPS from internet

curl -s http://[[ mandatory (FQDN) domain name not entered! click here ]]:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1' -d '{"command": "getNodeInfo"}' | jq

HTTPS from internet

curl -s https://[[ mandatory (FQDN) domain name not entered! click here ]] -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1' -d '{"command": "getNodeInfo"}' | jq

List your neighbors

Locally only

curl -s http://127.0.0.1:14267 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1' -d '{"command": "getNeighbors"}' | jq

Prometheus HTTP API

The API allows direct access to raw Prometheus data that can be utilized by other (internet) applications. Here you can find some examples.
A complete reference can be found here: Prometheus HTTP API docs

Create graphs in browser

https://[[ mandatory (FQDN) domain name not entered! click here ]]/prometheus/graph?g0.range_input=1h&g0.expr=iota_node_info_total_tips&g0.tab=0

List IOTA exporter metrics in browser

https://[[ mandatory (FQDN) domain name not entered! click here ]]/iota_exporter/metrics

List Jobs (Targets) on the console

curl -sG -u admin: https://[[ mandatory (FQDN) domain name not entered! click here ]]/prometheus/api/v1/targets |jq ".data.activeTargets[].discoveredLabels.job"

List Job Metrics

curl -sG -u admin: https://[[ mandatory (FQDN) domain name not entered! click here ]]/prometheus/api/v1/targets/metadata \
  --data-urlencode 'match_target={job="iota_exporter"}' \
  --data-urlencode 'limit=2' | jq ".data[] | {metric, help}"

Query a Metric

curl -sG -u admin: "https://[[ mandatory (FQDN) domain name not entered! click here ]]/prometheus/api/v1/query?query=iota_node_info_total_transactions_queued" |jq ".data.result[].value"

Additional security

Disable login for the user iota

sudo usermod -s /usr/sbin/nologin iota

From this point on you can switch to the user 'iota' with

sudo -u iota -H -s eval 'cd ~; source ~/.profile; bash'

Managing neighbors (without node restart)

When the node is running you can add (or remove) new neighbors on-the-fly with API calls. However, you have to also add (or remove) the new neighbors in the /home/iota/node/iota.ini file otherwise the changes you made via the API calls are lost with the next restart. Note that adding or removing only works if nelson is stopped. Separate multiple neighbors by comma.

Add neighbors (one or more at the same time)

curl -sH 'X-IOTA-API-VERSION: 1' -d '{"command":"addNeighbors", "uris":[
  "tcp://ip-of-the-new-neighbor:12345", "udp://ip-of-the-new-neighbor:54321"
]}' http://127.0.0.1:14267

Remove neighbors (one or more at the same time)

curl -sH 'X-IOTA-API-VERSION: 1' -d '{"command":"removeNeighbors", "uris":[
  "tcp://ip-of-the-new-neighbor:12345", "udp://ip-of-the-new-neighbor:54321"
]}' http://127.0.0.1:14267

Donations

My IOTA donation address
WQQNGOC9RLYCFANHZKVUT9IUVRLVENRYFIOUUAJSIXMZTDUYYZ9TCSRJLUKOVNLZXKDPBJCDMTXLSHLGWBL99YOHQA

Contact

You can find me in the IOTA Discord channels (username: Zoran)