March 03, 2021 ( last updated : March 03, 2021 )
rhel
grafana
loki
logging
I’ve been looking for a “light weight” method of getting aggregated logging across the home lab for a while. Loki has been interesting since its release, so I decided to have a play with it. I’ve been using Grafana heavily for years, so it seemed like the logical choice, as it is much lower on resources than Graylog, the other platform I was investigating.
The plan is to setup a single node in the lab as a log server, it has two disks associated with it - a boot device and a data device. The data device is a separate LV and will be configured to store logs on disk.
Note: This is a home lab installation, and is not configured with production grade security.
Sources: https://grafana.com/docs/loki/latest/installation/local/
# Install the requirements
dnf install nano wget unzip policycoreutils setroubleshoot
# Create a directory for Loki to run from
mkdir -v /opt/loki
# Create a user to run Loki as
useradd -s /sbin/nologin -d /opt/loki loki
# Provide access to the above directory to this user.
chown loki:loki /opt/loki
# Open the default loki firewall port
firewall-cmd --zone=public --add-port=3100/tcp
firewall-cmd --zone=public --add-port=3100/tcp --permanent
Download the Loki binaries and their respective configuration. There is a bug in the documentation that suggests the configuration should be pulled from the master branch, this is incorrect. The default configuration files should come from the same release branch as the binaries that are used.
cd /opt/loki
wget https://github.com/grafana/loki/releases/download/v2.1.0/loki-linux-amd64.zip
wget https://github.com/grafana/loki/releases/download/v2.1.0/promtail-linux-amd64.zip
unzip loki-linux-amd64.zip
unzip loki-linux-amd64.zip
rm *.zip
# Loki default configuration
wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/loki/loki-local-config.yaml
# Promtail default configuration
wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/promtail/promtail-local-config.yaml
At this point you should be able to run the example configuration and access it at http://{host_ip}:3100/metrics. By default Loki will listen on all interfaces, and with the firewall rules already open this will be accessible from your network.
./loki-linux-amd64 -config.file=loki-local-config.yaml
This should “work” in the most basic sense of the word. Depending on which user you are testing as you may encounter permissions issues. These can be rectified by ensuring the temporary directory exists and has the correct permissions. I have included the recursive flag on the chown in case this directory has been created already by running loki as root.
mkdir -pv /tmp/loki
chown -R loki:loki /tmp/loki
Whilst we are at it, lets create a file system location for log retention. I’m opting to use the file system backend at this time with a separate LV mounted at /data/, so I have used:
mkdir -pv /data/loki
chown loki:loki /data/loki
Once these were sorted I was able run Loki as the correct user with:
sudo -u loki loki-linux-amd64 -config.file=loki-local-config.yaml
At this point I was able to direct the Grafana Loki data source to my Loki server and add the data source without errors.
The final step is to adjust the configuration for local storage, more details can be found here: https://grafana.com/docs/loki/latest/configuration/examples/#complete-local-config. I’m not yet 100% convinced that I have the best configuration, but this seems to work.
#cat loki-local-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: filesystem
filesystem:
directory: /data/loki/chunks
compactor:
working_directory: /tmp/loki/boltdb-shipper-compactor
shared_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
ruler:
storage:
type: local
local:
directory: /data/loki/rules
rule_path: /tmp/loki/rules-temp
alertmanager_url: http://localhost:9093
ring:
kvstore:
store: inmemory
enable_api: true
Promtail is the tool that is used to ingest logs to Loki.
Sources:
Edit the default configuration file using your editor of personal or religous choice:
nano /opt/loki/promtail-local-config.yaml
As I particularly wish to use this for syslog aggregation, I have replaced the default job with the below configuration:
- job_name: syslog
syslog:
listen_address: 0.0.0.0:20514
idle_timeout: 60s
label_structured_data: yes
use_incoming_timestamp: yes
labels:
job: "syslog"
relabel_configs:
- source_labels: ['__syslog_message_hostname']
target_label: 'host'
- source_labels: ['__syslog_message_severity']
target_label: level
- source_labels: ['__syslog_message_app_name']
target_label: app_name
- source_labels: [__syslog_message_facility]
target_label: facility
This is the extent of default configuration required for using promtail.
NOTE: THE PORT OF 20514 IS NOT ARBITRARY! This port is part of the default SELinux rsyslog configuration that allows TCP communication. Another port can be used, but this requires further modification to the SELinux configuration. See https://support.logz.io/hc/en-us/articles/209486429-Troubleshooting-Rsyslog-SELinux-configuration for details on this process.
Add a firewall configuration to allow connections on this port.
firewall-cmd --zone=public --add-port=20514/tcp
firewall-cmd --zone=public --add-port=20514/tcp --permanent
Edit the local rsyslog configuration:
nano /etc/rsyslog.conf
Append the following to the end of the file:
action(type="omfwd" target="127.0.0.1" protocol="tcp" port="20514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted")
Save the file and exit.
At this point, you should be able to start Loki with:
sudo -u loki loki-linux-amd64 -config.file=loki-local-config.yaml
Then start promtail with the following:
sudo -u loki ./promtail-linux-amd64 -config.file=promtail-local-config.yaml
Finally, restart rsyslog with:
sudo systemctl restart rsyslog
Voila! If all has gone according to plan, you should now have syslog entries in Loki. This can be easily tested with Grafana.
Some legacy devices don’t provide much configurability in their remote syslog setup and so I would like my local rsyslog instance to also be able to listen on UDP 514 in order to forward to Loki.
Sources: https://rainer.gerhards.net/2019/10/rsyslog-configure-syslog-udp-reception.html
Modify the following comment lines to enable reception:
# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
Add a firewall configuration to allow connections on this port.
firewall-cmd --zone=public --add-port=514/udp
firewall-cmd --zone=public --add-port=514/udp --permanent
Create the following file:
sudo nano /etc/systemd/system/loki.service
[Unit]
Description=Grafana Loki service
After=network.target
[Service]
Type=simple
User=loki
ExecStart=/opt/loki/loki-linux-amd64 -config.file /opt/loki/loki-local-config.yaml
[Install]
WantedBy=multi-user.target
Create the following file:
sudo nano /etc/systemd/system/promtail.service
[Unit]
Description=Grafana Promtail service
After=network.target
[Service]
Type=simple
User=loki
ExecStart=/opt/loki/promtail-linux-amd64 -config.file /opt/loki/promtail-local-config.yaml
[Install]
WantedBy=multi-user.target
systemctl enable --now loki
systemctl enable --now promtail
This will likely fail due to a SELinux issue. The below type enforcement file should fix the identified issues.
Note: This was generated by setting permissive mode to be on setenforce 0 and then running grep loki /var/log/audit/audit.log | audit2allow -m loki > loki.te
# cat loki.te
module loki 1.0;
require {
type user_home_t;
type unreserved_port_t;
type init_t;
type unlabeled_t;
class file { execute execute_no_trans map open read };
class tcp_socket name_connect;
class dir add_name;
}
#============= init_t ==============
allow init_t unlabeled_t:dir add_name;
#!!!! This avc is allowed in the current policy
allow init_t unreserved_port_t:tcp_socket name_connect;
#!!!! This avc is allowed in the current policy
allow init_t user_home_t:file { execute execute_no_trans map open read };
This file can be converted into a module with:
checkmodule -M -m -o loki.mod loki.te
Before compiling as a policy package:
semodule_package -o loki.pp -m loki.mod
And installing:
semodule -i loki.pp
At this point, this can be tidied up with:
rm loki.mod loki.pp loki.te
Assuming you carried out the earlier service enabling, a restart of the services should yield success:
systemctl restart loki
systemctl restart promtail
Originally published March 03, 2021 (Updated March 03, 2021)
Related posts :