← All cheat sheets

RSYSLOG

Plain-text reference · 6 KB. Read it, search it (Ctrl-F) or print it.

rsyslog is the common Linux syslog daemon. In a SOC it is the workhorse for
COLLECTING local logs and FORWARDING them to a central collector/SIEM. This
sheet covers config basics, forwarding, TLS, and troubleshooting delivery.

FILES & SERVICE#

/etc/rsyslog.conf             main config
/etc/rsyslog.d/*.conf         drop-in configs (loaded in name order)
systemctl restart rsyslog     apply changes
systemctl status rsyslog
rsyslogd -N1                  validate config syntax WITHOUT starting
logger -t test "hello"        inject a test message into syslog

CONFIG STRUCTURE (three parts)#

1. MODULES (load inputs/features)
   module(load="imuxsock")    # local unix socket (/dev/log)
   module(load="imklog")      # kernel log
   module(load="imjournal")   # read the systemd journal
   module(load="imudp")       # receive syslog over UDP
   module(load="imtcp")       # receive syslog over TCP
   module(load="omfwd")       # forward output (often built-in)

2. INPUTS (listeners)
   input(type="imudp" port="514")
   input(type="imtcp" port="514")

3. RULES  = SELECTOR (facility.severity) + ACTION
   *.info;mail.none;authpriv.none  /var/log/messages
   authpriv.*                      /var/log/secure
   *.*                             @@collector.example.com:6514

SELECTORS: FACILITY.SEVERITY#

Facilities: auth, authpriv, cron, daemon, kern, mail, syslog, user,
            local0..local7 (local0-7 are for your own apps).
Severities (high->low): emerg, alert, crit, err, warning, notice, info, debug.
Examples:
  kern.crit            /var/log/kernel-crit
  auth,authpriv.*      /var/log/auth.log
  *.=err               only err-level (= means EXACTLY this severity)
  mail.*               /var/log/mail.log
  *.*                  everything

FORWARDING TO A CENTRAL COLLECTOR#

Legacy syntax:
  *.*  @collector:514      single @  = UDP  (lossy, no delivery guarantee)
  *.*  @@collector:514     double @@ = TCP  (reliable transport)

Modern action() syntax (preferred -- supports queues/TLS):
  action(type="omfwd"
         target="collector.example.com"
         port="6514"
         protocol="tcp")

DISK-ASSISTED QUEUE (don't lose logs if the collector is down)#

action(type="omfwd"
       target="collector.example.com" port="6514" protocol="tcp"
       queue.type="LinkedList"          # in-memory queue
       queue.filename="fwdRule1"        # enables disk-assisted spooling
       queue.maxdiskspace="1g"
       queue.saveonshutdown="on"
       action.resumeRetryCount="-1")    # retry forever
This buffers to disk when the collector is unreachable and drains on recovery.

TLS-ENCRYPTED FORWARDING (syslog over TLS, usually port 6514)#

# Load the TLS network driver (GnuTLS)
global(
  defaultNetstreamDriver="gtls"
  defaultNetstreamDriverCAFile="/etc/rsyslog.d/ca.pem"
  defaultNetstreamDriverCertFile="/etc/rsyslog.d/client-cert.pem"
  defaultNetstreamDriverKeyFile="/etc/rsyslog.d/client-key.pem"
)
action(type="omfwd"
       target="collector.example.com" port="6514" protocol="tcp"
       StreamDriver="gtls"
       StreamDriverMode="1"          # 1 = TLS
       StreamDriverAuthMode="x509/name"
       StreamDriverPermittedPeers="collector.example.com")

Receiver side similarly sets the CA/cert/key and
input(type="imtcp" port="6514").

TEMPLATES (control the on-wire / on-disk format)#

# RFC5424 structured format (good for SIEM ingestion)
template(name="rfc5424" type="string"
  string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")

# Dynamic file path per-host (useful on a collector)
template(name="PerHost" type="string"
  string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
action(type="omfile" dynaFile="PerHost")

# Use a template on forwarding
action(type="omfwd" target="collector" port="514" protocol="tcp"
       template="rfc5424")

Common property replacers: %msg% %HOSTNAME% %PROGRAMNAME% %syslogtag%
%TIMESTAMP% %PRI% %fromhost-ip% %syslogfacility-text% %syslogseverity-text%.

FILTER ON MESSAGE CONTENT (property-based)#

if $programname == 'sshd' then /var/log/sshd.log
if $msg contains 'Failed password' then @@collector:6514
if $fromhost-ip startswith '10.0.' then /var/log/internal.log
& stop        # stop processing further rules for this message

TROUBLESHOOTING LOG DELIVERY#

rsyslogd -N1                       validate config (no restart)
systemctl status rsyslog           is it running? recent errors?
journalctl -u rsyslog -n 50        daemon's own log
logger -t debugtest "ping 123"     inject a test message
# Then confirm it landed locally and on the collector:
grep "ping 123" /var/log/messages
# Network reachability to the collector:
ss -tanp | grep 6514               is the TCP session established?
# Verbose debug (temporary):
rsyslogd -dn                       run in foreground with debug output
# Common causes of "logs not arriving":
#  - SELinux/firewall blocking the port (check firewall + semanage port)
#  - UDP used (silent loss) -> switch to TCP + disk queue
#  - collector not listening / cert mismatch for TLS
#  - rate limiting: imjournal/imuxsock RateLimit.Interval/Burst dropping msgs
#  - disk full on the queue spool directory

GOOD PRACTICE FOR SOC#

- Forward over TCP (@@) or TLS, never bare UDP, for anything that matters.
- Always add a disk-assisted queue so a collector outage does not drop logs.
- Timestamp in RFC3339/UTC; include %HOSTNAME% and %fromhost-ip%.
- Protect log integrity: restrict who can write to log paths; forward
  off-host quickly so an attacker clearing local logs is caught centrally.

Defensive reference on CyberRamen. Offensive / red-team sheets live on OffensiveRamen.com.