Warning

License: Free — part of the universal collection tier (moved from Pro in 0.2.2).

Syslog Probe

The Syslog probe collects system logs and events by running a Syslog server that receives messages from network devices, servers, applications, and infrastructure components via UDP or TCP.

Quick Start

Basic Configuration

# probes.d/10-syslog.yaml — each file under probes.d/ is a YAML array of probes
- name: syslog
  type: syslog
  params:
    port: 514               # Syslog port (default: 514)
    protocol: udp           # Protocol: udp or tcp (default: udp)
    bind_address: "0.0.0.0" # Required to receive from remote senders (default: 127.0.0.1)

Minimal Configuration

# probes.d/10-syslog.yaml
- name: syslog
  type: syslog
  params: {}

The Syslog probe requires no mandatory parameters and works out-of-the-box with default settings (UDP port 514).

Supported Platforms

• Windows: Windows Server 2012+ / Windows 10+
• Linux: All modern distributions (Ubuntu, RHEL, CentOS, Debian, etc.)
• macOS: macOS 10.13+
• BSD: FreeBSD, OpenBSD, NetBSD

The Syslog probe is platform-independent. It listens on the loopback interface by default; receiving from remote senders requires setting bind_address explicitly (e.g. "0.0.0.0" or a specific interface IP).

Key Features Summary

Event Collection

Feature	Description	Details
RFC Compliance	Automatic format detection	RFC 3164 (BSD) and RFC 5424 (IETF)
Transport Protocols	UDP and TCP support	Configurable per probe instance
Event Metadata	Complete message parsing	Facility, severity, hostname, timestamp, content
Real-time Processing	Event-driven architecture	Zero-latency event forwarding
Multi-source	Centralized log aggregation	Network devices, servers, applications

Collected Event Fields

Field	Description	Example
facility	Syslog facility code (0-23)	1 (user-level)
severity	Syslog severity level (0-7)	3 (error)
hostname	Source system hostname	firewall-01.example.com
content	Log message content	Connection denied from 192.168.1.100
tag	Application/service identifier	sshd, kernel, nginx
client	Client IP address	192.168.1.50
priority	Combined facility/severity (PRI)	11 ((1*8)+3)
timestamp	Message timestamp	2025-10-13T14:23:45Z

Configuration Parameters

Parameter	Must set	Default	Description
port	In practice	514	Port to listen on; 514 needs root or CAP_NET_BIND_SERVICE
protocol	In practice	udp	Transport the listener accepts. One of udp, tcp
bind_address	In practice	127.0.0.1	Interface address to listen on; loopback when empty, so remote senders need 0.0.0.0 or an interface address. Example: 0.0.0.0

Example Configurations

Standard UDP Syslog (default):

# probes.d/10-syslog.yaml
- name: syslog
  type: syslog
  params:
    port: 514
    protocol: udp

TCP Syslog with custom port:

# probes.d/10-syslog.yaml
- name: syslog_tcp
  type: syslog
  params:
    port: 1514
    protocol: tcp

High-security environments (TCP + custom port):

# probes.d/10-syslog.yaml
- name: syslog_secure
  type: syslog
  params:
    port: 6514
    protocol: tcp

Multiple syslog listeners:

# probes.d/10-syslog.yaml
- name: syslog_standard
  type: syslog
  params:
    port: 514
    protocol: udp

- name: syslog_applications
  type: syslog
  params:
    port: 1514
    protocol: tcp

Monitoring Tool Integration

PRTG Network Monitor

Access syslog events in PRTG JSON format:

# All syslog events
curl http://localhost:8080/api/{agentkey}/prtg/metrics

# Configure PRTG HTTP Advanced Sensor:
# - URL: http://agent-host:8080/api/{agentkey}/prtg/metrics
# - Method: POST
# - Request body: {"probe": "syslog"}

PRTG Channels Available: - Syslog Events Received (count) - Events by Severity (Emergency, Alert, Critical, Error, Warning) - Events by Facility (Kernel, User, System, Security) - Event Rate (events/second)

Nagios

Access syslog metrics in Nagios format:

# Syslog event statistics
curl http://localhost:8080/api/{agentkey}/nagios/metrics?probe=syslog

# Example output:
# OK - Syslog events: 1234 received, 56 errors | events_total=1234c events_errors=56c rate=2.3/s

Nagios Performance Data: - syslog_events_total - Total events received (counter) - syslog_events_by_severity - Events by severity level - syslog_event_rate - Events per second

Web console

The Probes page of the console shows the probe's state and the counters it emits; the Sensor URLs tab of the HTTP output previews them as a poller sees them. The events themselves go to the outputs that read logs (OTLP, events); the console does not display them.

Use Cases

Centralized Log Aggregation

Collect logs from multiple sources into a single location: - Network devices (routers, switches, firewalls) - Linux/Unix servers - Windows systems (with syslog forwarders) - Applications and services - Security appliances

Configuration Example:

# probes.d/10-syslog.yaml — central syslog collector
- name: syslog_collector
  type: syslog
  params:
    port: 514
    protocol: udp

# strategies.d/00-event.yaml — forward to multiple destinations
event:
  type: event
  targets: ["senhub", "local_storage", "siem"]

Security Monitoring (SIEM Integration)

Forward security-critical logs to SIEM systems: - Authentication failures - Firewall denials - Intrusion detection alerts - Security policy violations

Filter by Security Facilities: - Facility 4: Security/authorization messages - Facility 10: Security/authorization messages (private) - Facility 13: Log audit

Compliance and Audit Logging

Maintain compliance with regulatory requirements: - PCI-DSS: Network and system logs - HIPAA: Access and security logs - SOX: Administrative activity logs - GDPR: Data access and processing logs

Retention Requirements: - PCI-DSS: 1 year minimum - HIPAA: 6 years minimum - SOX: 7 years minimum

Network Device Monitoring

Collect logs from network infrastructure: - Router and switch syslogs - Firewall policy logs - Load balancer events - VPN connection logs

Common Network Severities: - 0 (Emergency): System unusable - 1 (Alert): Immediate action required - 2 (Critical): Critical conditions - 3 (Error): Error conditions

Application Log Aggregation

Centralize application logs: - Web server access/error logs (nginx, Apache) - Database logs (MySQL, PostgreSQL) - Application server logs (Tomcat, JBoss) - Custom application logs

Common Application Facilities: - Facility 16-23: Local use (local0-local7)

Troubleshooting

No Events Received

Check network connectivity:

# Test UDP syslog
echo "<13>Oct 13 14:23:45 test-host test-app: Test message" | nc -u -w1 localhost 514

# Test TCP syslog
echo "<13>Oct 13 14:23:45 test-host test-app: Test message" | nc -w1 localhost 514

# Verify port is listening
netstat -an | grep 514
# or
ss -tulpn | grep 514

Check agent logs:

# View syslog probe debugging
senhub-agent run --filter probe.syslog

Verify probe configuration:

# Check configuration (multi-file layout)
grep -rA5 "type: syslog" /etc/senhub-agent/probes.d/

Permission Denied (Port < 1024)

Symptom: Error binding to port 514 on Unix/Linux systems

Solution: The Linux service runs as the unprivileged senhub account with every capability dropped, so it cannot bind a port below 1024. The error says so: bind: permission denied: port 514 is below 1024. Two ways out:

# Option 1: grant the one capability in a unit drop-in
sudo systemctl edit senhub-agent.service
#   [Service]
#   CapabilityBoundingSet=CAP_NET_BIND_SERVICE
#   AmbientCapabilities=CAP_NET_BIND_SERVICE
sudo systemctl daemon-reload && sudo systemctl restart senhub-agent

# Option 2: listen on a port above 1023 and point the senders at it
# /etc/senhub-agent/probes.d/10-syslog.yaml:
- name: syslog
  type: syslog
  params:
    port: 1514  # Non-privileged port

setcap on the binary does not help: the unit's empty capability bounding set removes file capabilities, and an update replaces the file. See Running the agent least-privilege.

Configure syslog sources to use alternate port:

# rsyslog configuration (/etc/rsyslog.conf or /etc/rsyslog.d/*.conf)
*.* @syslog-server:1514  # UDP
*.* @@syslog-server:1514 # TCP

Windows: Firewall Blocking

Symptom: Events not received from remote sources

Solution: Add Windows Firewall rule:

# Allow UDP Syslog
New-NetFirewallRule -DisplayName "SenHub Agent Syslog (UDP)" `
  -Direction Inbound -Protocol UDP -LocalPort 514 -Action Allow

# Allow TCP Syslog
New-NetFirewallRule -DisplayName "SenHub Agent Syslog (TCP)" `
  -Direction Inbound -Protocol TCP -LocalPort 514 -Action Allow

# Verify rules
Get-NetFirewallRule | Where-Object {$_.DisplayName -like "*Syslog*"}

UDP Packet Loss

Symptom: Missing events during high-volume periods

Solution: UDP is connectionless and can drop packets under load:

1. Switch to TCP for reliability:

   - name: syslog
     type: syslog
     params:
       protocol: tcp  # Use TCP instead of UDP
   

2. Increase UDP receive buffer:

   # Linux: Increase UDP buffer size
   sudo sysctl -w net.core.rmem_max=26214400
   sudo sysctl -w net.core.rmem_default=26214400
   
   # Make permanent
   echo "net.core.rmem_max=26214400" | sudo tee -a /etc/sysctl.conf
   echo "net.core.rmem_default=26214400" | sudo tee -a /etc/sysctl.conf
   

3. Rate limit syslog sources:

   # rsyslog rate limiting (/etc/rsyslog.conf)
   $SystemLogRateLimitInterval 10
   $SystemLogRateLimitBurst 500
   

Malformed Messages

Symptom: Events not parsed correctly or missing fields

Solution: The probe uses automatic format detection (RFC 3164/5424):

1. Verify syslog source format:

   # Capture raw syslog packets
   sudo tcpdump -i any -n port 514 -A
   

2. Check for proper RFC 3164 format:

   <PRI>TIMESTAMP HOSTNAME TAG: MESSAGE
   Example: <13>Oct 13 14:23:45 server01 sshd[1234]: Connection from 192.168.1.100
   

3. Check for proper RFC 5424 format:

   <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MESSAGE
   Example: <13>1 2025-10-13T14:23:45.000Z server01 sshd 1234 - - Connection from 192.168.1.100
   

High Memory Usage

Symptom: Agent consuming excessive memory during high event rates

Solution:

1. Implement event filtering at source:

   # rsyslog: Filter by severity (only errors and above)
   if $syslogseverity-text == 'err' or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or $syslogseverity-text == 'emerg' then @@syslog-server:514
   

2. Use rate limiting:

   # Limit events per source
   $SystemLogRateLimitInterval 60
   $SystemLogRateLimitBurst 1000
   

3. Increase storage flush interval (if using local storage)

Performance Considerations

Collection Overhead

The Syslog probe is event-driven with minimal overhead: - UDP: Near-zero overhead, fire-and-forget - TCP: ~1-2ms per connection establishment - Message parsing: ~0.1ms per message - Memory: ~1 KB per message in memory

UDP vs TCP Performance

Feature	UDP	TCP
Throughput	Very high (50,000+ msg/sec)	High (10,000-20,000 msg/sec)
Reliability	No delivery guarantee	Guaranteed delivery
Latency	Lowest (~0.1ms)	Low (~1-5ms)
Connection overhead	None	Connection establishment required
Ordering	No guarantee	Ordered delivery
Use case	High-volume, non-critical logs	Critical logs, compliance

Recommended Configurations

Use Case	Protocol	Rationale
Network device logs	UDP	High volume, some loss acceptable
Security events	TCP	Reliability critical
Application logs	UDP	High throughput required
Compliance logs	TCP	No data loss permitted
Mixed environment	Both	Run two probe instances

Advanced Configuration

Multiple Syslog Listeners

Run multiple syslog servers for different purposes:

# probes.d/10-syslog.yaml
# Standard UDP syslog (network devices)
- name: syslog_devices
  type: syslog
  params:
    port: 514
    protocol: udp

# TCP syslog for applications
- name: syslog_apps
  type: syslog
  params:
    port: 1514
    protocol: tcp

# Secure syslog for critical systems
- name: syslog_critical
  type: syslog
  params:
    port: 6514
    protocol: tcp

Integration with Other Probes

Correlate syslog events with system metrics:

# probes.d/00-host.yaml
# Syslog events
- name: syslog
  type: syslog
  params:
    port: 514
    protocol: udp

# System metrics for correlation
- name: cpu
  type: cpu
  params:
    interval: 30

- name: memory
  type: memory
  params:
    interval: 30

- name: network
  type: network
  params:
    interval: 30

This enables correlation like: - High CPU usage → Syslog errors - Memory exhaustion → OOM killer logs - Network saturation → Connection timeouts

Syslog Source Configuration

Linux rsyslog

# /etc/rsyslog.d/senhub.conf

# Send all logs via UDP
*.* @syslog-server:514

# Send all logs via TCP
*.* @@syslog-server:514

# Send only errors via TCP
*.err @@syslog-server:514

# Send security logs only
authpriv.* @@syslog-server:514

# Apply and restart
sudo systemctl restart rsyslog

Cisco IOS/IOS-XE

configure terminal
logging host 192.168.1.100 transport udp port 514
logging trap informational
logging facility local6
logging source-interface Loopback0
end
write memory

pfSense/OPNsense Firewall

System > Settings > Logging
  Remote Logging Options:
    - Enable Remote Logging: check
    - Source Address: [Interface IP]
    - Remote log servers: 192.168.1.100:514
    - Remote Syslog Contents: Everything

Windows Event Log Forwarding

Windows doesn't natively support syslog. Use a forwarder:

Option 1: NXLog Community Edition

# nxlog.conf
<Input eventlog>
    Module im_msvinevents
</Input>

<Output syslog>
    Module om_udp
    Host 192.168.1.100
    Port 514
    Exec to_syslog_bsd();
</Output>

<Route 1>
    Path eventlog => syslog
</Route>

Option 2: Windows Syslog Agent - Snare for Windows - Kiwi Syslog Agent - EventLog-to-Syslog

Syslog Format Reference

RFC 3164 (BSD Syslog)

<PRI>TIMESTAMP HOSTNAME TAG: MESSAGE

Example:
<13>Oct 13 14:23:45 server01 sshd[1234]: Connection from 192.168.1.100

The RFC 3164 timestamp carries no time zone. The probe reads it in the agent host's zone, which is right when the senders and the agent share one; a sender set to another zone arrives shifted by the difference. RFC 5424 timestamps carry their offset and are kept as sent.

RFC 5424 (IETF Syslog)

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MESSAGE

Example:
<13>1 2025-10-13T14:23:45.000Z server01 sshd 1234 ID47 [exampleSDID@32473 eventID="1"] Connection from 192.168.1.100

Messages without a PRI

Some senders start the line at the timestamp, with no <PRI>: the UniFi controller's activity log, for one.

Sep 29 18:08:37 unifi-pi CEF:0|Ubiquiti|UniFi Network|10.6.106|546|Config Modified|5|...

The probe still reads the timestamp, the hostname and the tag from such a line, and keeps the rest as the body. It sets no facility or priority, since the sender gave none. The severity is 5 (notice), unless the body is a CEF record: the severity then comes from the CEF header, 0-3 as informational, 4-6 as warning, 7-8 as error and 9-10 as critical. A message that carries a PRI keeps the severity of its PRI.

Priority (PRI) Calculation

PRI = (Facility * 8) + Severity

Examples:
- Facility 1 (user), Severity 3 (error): PRI = (1*8)+3 = 11
- Facility 4 (security), Severity 2 (critical): PRI = (4*8)+2 = 34

Authentication

The Syslog probe requires no authentication for incoming syslog messages. Access control should be implemented at the network level (firewall rules, VLANs).

Security Recommendations: - Restrict syslog traffic to trusted networks - Use TCP for sensitive logs - Consider TLS syslog for encryption (future enhancement) - Implement network segmentation - Monitor for unauthorized syslog sources

Requirements

Operating System

• Windows: Windows Server 2012+ or Windows 10+
• Linux: Any modern distribution with network stack
• macOS: macOS 10.13+
• BSD: FreeBSD, OpenBSD, NetBSD

Network

• Inbound UDP/TCP access on configured port
• Firewall rules allowing syslog traffic
• Network connectivity to syslog sources

Permissions

• Privileged ports (<1024): Root/Administrator or CAP_NET_BIND_SERVICE capability
• Non-privileged ports (>=1024): Standard user account sufficient

Syslog Sources

• RFC 3164 (BSD Syslog) or RFC 5424 (IETF Syslog) format
• UDP or TCP transport support

Metric reference

Every metric this probe can emit. Metric is the OpenTelemetry name the OTLP, Prometheus and Zabbix outputs derive theirs from. Name is what a Nagios check and the API metrics= filter match. PRTG channel is the label PRTG shows, placeholders filled from the series' tags.

Metric	Name	PRTG channel	Unit	Description
senhub.syslog.records_emitted	senhub.syslog.records_emitted	Syslog Records Emitted	#	Cumulative count of syslog messages this probe has relayed to the log rail
-	syslog_event	Syslog {facility_name} Severity	#	Syslog message severity level (RFC 3164)