docs
Tangle Network
Node Operators
Monitoring
Loki Log Manager

Loki Log Management

The following is a guide outlining the steps to setup Loki for log management of a Tangle node. If you do not have Tangle node setup yet, please review the Tangle Node Quickstart setup guide here.

In this guide we will configure the following modules to scrape metrics from the running Tangle node.

  • Loki provides log aggregation system and metrics.
  • Promtail is the agent responsible for gathering logs, and sending them to Loki.

Let's first start by downloading the latest releases of the above mentioned modules (Loki, Promtail download pages).

This guide assumes the user has root access to the machine running the Tangle node, and following the below steps inside that machine.

1. Download Loki

AMD version:

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/loki-darwin-amd64.zip"

ARM version:

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/loki-darwin-arm64.zip"

2. Download Promtail

AMD version:

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/promtail-darwin-amd64.zip"

ARM version:

curl -O -L "https://github.com/grafana/loki/releases/download/v2.7.0/promtail-darwin-arm64.zip"

3. Extract the Downloaded Files:

unzip "loki-linux-amd64.zip" &&
unzip "promtail-linux-amd64.zip"

4. Copy the Extracted Files into /usr/local/bin:

sudo cp loki-linux-amd64 /usr/local/bin/ &&
sudo cp promtail-linux-amd64 /usr/local/bin/

5. Create Dedicated Users:

Now we want to create dedicated users for each of the modules we have installed:

sudo useradd --no-create-home --shell /usr/sbin/nologin loki &&
sudo useradd --no-create-home --shell /usr/sbin/nologin promtail

6. Create Directories for loki, and promtail:

sudo mkdir /etc/loki &&
sudo mkdir /etc/promtail

7. Change the Ownership for all Directories:

We need to give our user permissions to access these directories:

sudo chown loki:loki /usr/local/bin/loki-linux-amd64 &&
sudo chown promtail:promtail /usr/local/bin/promtail-linux-amd64

9. Finally, let's clean up these directories:

rm -rf ./loki-linux-amd64* &&
rm -rf ./promtail-linux-amd64*

The next series of steps will be configuring each service.

Configuration

If you are interested to see how we configure the Tangle Network nodes for monitoring check out https://github.com/webb-tools/tangle/tree/main/monitoring (opens in a new tab).

Loki

Loki's configuration details what ports to listen to, how to store the logs, and other configuration options. There are many other config options for Loki, and you can read more about Loki configuration at: https://grafana.com/docs/loki/latest/configuration/ (opens in a new tab)

Let’s create the file:

sudo touch /etc/loki/config.yml
sudo nano /etc/loki/config.yml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h


storage_config:
  boltdb:
    directory: /data/loki/index

  filesystem:
    directory: /data/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0

Promtail

The Promtail configuration details what logs to send to Loki. In the below configuration we are indicating to send the logs to Loki from the /var/log/dkg directory. This directory can be changed based on what logs you want to pick up. There are many other config options for Promtail, and you can read more about Promtail configuration at: https://grafana.com/docs/loki/latest/clients/promtail/configuration/ (opens in a new tab)

Let’s create the file:

sudo touch /etc/promtail/config.yml
sudo nano /etc/promtail/config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /data/loki/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/dkg/*log

Service Setup

Loki

Create and open the Loki service file:

sudo touch /etc/systemd/system/loki.service &&
sudo nano /etc/systemd/system/loki.service

Add the following lines:

[Unit]
  Description=Loki Service
  Wants=network-online.target
  After=network-online.target

[Service]
  User=loki
  Group=loki
  Type=simple
  ExecStart=/usr/local/bin/loki-linux-amd64 -config.file /etc/loki/config.yml

[Install]
WantedBy=multi-user.target

Promtail

Create and open the Promtail service file:

sudo touch /etc/systemd/system/promtail.service &&
sudo nano /etc/systemd/system/promtail.service

Add the following lines:

[Unit]
  Description=Promtail Service
  Wants=network-online.target
  After=network-online.target

[Service]
  User=promtail
  Group=promtail
  Type=simple
  ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /etc/promtail/config.yml

[Install]
WantedBy=multi-user.target

Great! You have now configured all the services needed to run Loki.

Starting the Services

Launch a daemon reload to take the services into account in systemd:

sudo systemctl daemon-reload

Next, we will want to start each service:

sudo systemctl start loki.service &&
sudo systemctl start promtail.service

And check that they are working fine, one by one:

loki:

systemctl status loki.service

promtail:

systemctl status promtail.service

If everything is working adequately, activate the services!

sudo systemctl enable loki.service &&
sudo systemctl enable promtail.service

Amazing! You have now successfully configured Loki for log management. Check out the Grafana documentation to create a Loki log dashboard!

Grafana DashboardCLI Options