Logging SSH connections from a Raspberry Pi bastion host using syslog-ng

Logging SSH connections from a Raspberry Pi bastion host using syslog-ng

October 20, 2022

Three common methods to connect to a private network include

  1. VPN
  2. Third-party services that proxy connections.
  3. Bastion hosts

Bastion hosts offer simplicity and a small attack surface.

The purpose of the bastion host is to have a jump host into your home lab and personal networks. Many developer tools have capabilities that work with bastion hosts to limit the inefficiency of having to actually jump from one host to another. For example, Pycharm (Datagrip) and DBeaver.io support the configuration of a jump box to provide the experience that you are directly connecting to a DB on a private network.

The simplicity and small attack surface of a bastion host comes from running a single public-facing service, an SSH server. By limiting SSH authentication to private key auth, you limit access to your backend to users that have a private key with a public key registered in the host’s authorized_keys file. This removes the opportunity for brute-force attacks or compromises using credential stuffing.

Bastion hosts can still be subject to slow loris like attacks; however, this is true of many systems, and there are some configurations that can reduce the impact. https://en.wikipedia.org/wiki/Slowloris_(computer_security)

It is useful to have bastion server auth logs that are remotely stored and searchable. An ideal logging configuration has at least a list of authorized users connected to your bastion host and when. For low-traffic systems, like home labs and personal networks, setup notifications to know when activity occurs on your bastion host.

This post covers Raspberry Pi as a bastion host with SSH server activity logged to the Log Center (syslog) on a Synology NAS. The core concepts are applicable to other prosumer syslog servers, such as QNAP and ASUSTOR.

Systems #

  • Synology NAS: syslog-ng server and notifications
    • syslog-ng is used for its TCP capabilities. TCP has transport guarantees not provided by the UDP implementation in syslog
  • Raspberry PI: Bastion host and Syslog client
    • hostname for exercise: bastionhost
  • Router: DMZ, port forward internet to bastion host
  • Host with private SSH key: Used for configuration on the LAN and remote access from the internet (outside LAN)
    • hostname for exercise: remotemachine

Synology #

Activities

Install / activate the syslog server following the documentation for your NAS appliance. Synology device’s use the Log Center application.

  • Configure Log Center
  • Set the Syslog transport protocol to TCP.
  • Note the server port. 514 is commonly used for transport. .
  • Note your NAS appliance IP address.
configure log center

The server port, server protocol, and NAS IP address will be referenced in the syslog-ng conf on the client Raspberry Pi.

Remote machine #

Copy your SSH public key to the bastion host using ssh-copy-id. You will be prompted for your password.

marmotstudios@remotemachine:~ $ ssh-copy-id marmotstudios@sst_labs

Raspberry PI #

Moving forward, the Raspberry PI may be referenced as the bastion host.

Recommendations

  • Use a recent version of the Raspberry Pi OS that prompts you to provide a username other than pi.
  • Use raspi-config to set a hostname and enable the SSH server

Install syslog-ng

marmotstudios@bastionhost:~ $ sudo apt install syslog-ng

Notes

syslog-ng.conf #

Backup configuration files before editing the files. Reverting to a previous state is far easier if there is a previous state.

marmotstudios@bastionhost:~ $ sudo cp /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf.bak

Open the syslog-ng config file for editing. Vim is not required, any CLI editor will work.

marmotstudios@sst_labs:~ $ sudo vim /etc/syslog-ng/syslog-ng.conf

Notes

  1. to access line numbers in vim, type :set number in vim’s command prompt. https://u.osu.edu/cstutorials/2018/10/02/vim-editor-set-number-default/
  2. Surrounding text included for readability.

Before

 75 destination d_xconsole { pipe("/dev/xconsole"); };
 76
 77 # Send the messages to an other host
 78 #
 79 #destination d_net { tcp("127.0.0.1" port(1000) log_fifo_size(1000)); };
 80
 81 # Debian only
 82 destination d_ppp { file("/var/log/ppp.log"); };

After

 75 destination d_xconsole { pipe("/dev/xconsole"); };
 76
 77 # Send the messages to an other host
 78 #
 79 destination d_net { tcp("192.168.1.5" port(514)); };
 80
 81 # Debian only
 82 destination d_ppp { file("/var/log/ppp.log"); };

Line 79 is the only edited line

  • The IP address referenced is the NAS running the syslog server, Log Center
  • The port is the configured port you configured in Log Center
  • The protocol, tcp, should match the protocol configured in Log Center

Editing continues below.

Before

154 # All messages send to a remote site
155 #
156 #log { source(s_src); destination(d_net); };
157
158 ###
159 # Include all config files in /etc/syslog-ng/conf.d/
160 ###

After

154 # All messages send to a remote site
155 #
156 log { source(s_src); destination(d_net); };
157
158 ###
159 # Include all config files in /etc/syslog-ng/conf.d/
160 ###

The only line edit in this section is line 156. Line 156 was uncommented.

Note: you can find references to s_src and d_net in the syslog-ng.conf that is being edited.

Exit vim and save the config file.

Reload the syslog-ng service.

marmotstudios@bastionhost:~ $ systemctl restart syslog-ng

sshd_config #

Backup the configuration file

marmotstudios@bastionhost:~ $ sudo sudo cp  /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

Open the sshd_config file for editing.

marmotstudios@bastionhost:~ $ sudo vim /etc/ssh/sshd_config

Notes

  • Changes were not required to the default sshd_config for syslog transport using Raspberry Pi OS Lite, Release date: September 22nd 2022. Changes are made to be explicit over implicit defaults and to disable root login.

Before

 24 # Ciphers and keying
 25 #RekeyLimit default none
 26
 27 # Logging
 28 #SyslogFacility AUTH
 29 #LogLevel INFO
 30
 31 # Authentication:
 32
 33 #LoginGraceTime 2m
 34 #PermitRootLogin prohibit-password
 35 #StrictModes yes
 36 #MaxAuthTries 6
 37 #MaxSessions 10
 38
 39 #PubkeyAuthentication yes

After

 24 # Ciphers and keying
 25 #RekeyLimit default none
 26
 27 # Logging
 28 SyslogFacility AUTH
 29 LogLevel INFO
 30
 31 # Authentication:
 32
 33 #LoginGraceTime 2m
 34 PermitRootLogin no
 35 #StrictModes yes
 36 #MaxAuthTries 6
 37 #MaxSessions 10
 38
 39 PubkeyAuthentication yes

Lines changed

  • Uncomment 28
  • Uncomment 29
  • Uncomment and alter 34
  • Uncomment 39

Editing continues below

Before

 54 #IgnoreRhosts yes
 55
 56 # To disable tunneled clear text passwords, change to no here!
 57 #PasswordAuthentication yes
 58 #PermitEmptyPasswords no
 59
 60 # Change to yes to enable challenge-response passwords (beware issues with

After

 54 #IgnoreRhosts yes
 55
 56 # To disable tunneled clear text passwords, change to no here!
 57 PasswordAuthentication yes
 58 PermitEmptyPasswords no
 59
 60 # Change to yes to enable challenge-response passwords (beware issues with

Lines changed

  • Uncomment 57
  • Uncomment 58

Exit vim and save the config file.

Reload the ssh server service

marmotstudios@bastionhost:~ $ sudo service restart ssh-server

Verify TCP port 22 activity is logged #

Telnet from the remote host on your LAN to port 22 on your bastion host (192.168.1.5).

marmotstudios@remotemachine:~ $ telnet 192.168.1.5 22
Trying 192.168.1.5...
Connected to 192.168.1.5.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u1
^]
telnet> Connection closed.

Press the CTRL + closing bracket key combination, then CTRL+d to exit.

The telnet activity interacts with the SSH server and creates logs that are sent to syslog-ng server. Example INFO and ERROR messages are viewable in the NAS Log Center > Logs.

Note: Filter your logs in Log Center by the hostname of your bastion server.

This level of logging means that connection activity from scans, brute forces, and authorized connections, and connection closes will generate logs. Mapping the bastion hosts SSH server directly to TCP port 22 on your public IP will result in significant log messages from scanning and brute-force activity. This issue is addressed in the router section.

Router #

Quick summary

  • SSH Authentication is enabled to your bastion host via SSH key.
  • Password and root auth should be disabled
  • Ecosystem: Port 22 is frequently targeted by scanners and brute-force attacks. Connections to port 22 on a public IP should be expected in minutes.

Set your router to port forward from a high port not in the Well Known or Registered port ranges to TCP port 22 on your bastion host. A high, non-standard port will typically reduce if not eliminate most connection attempts to your SSH server.

For example, port 49023. A list of commonly used ports is available at Wikipedia. https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

There is no guarantee that a high port will exclude activity. Should activity become a concern, you should consider altering your notifications or filtering what is logged.

Configure alerts #

Prerequisites - You have notifications already set up for your NAS Appliance. https://kb.synology.com/en-us/DSM/help/DSM/AdminCenter/system_notification_desc?version=7

Troubleshooting #

If verification fails

  1. NAS: Verify you have the correct IP, protocol, and port
  2. Raspberry Pi:
    1. Verify you have correctly set the destination and enabled the logging and restarted the services

To troubleshoot logs not visible in the log viewer for the NAS appliance

NAS Appliance

Check for any filters on hosts. Filters may exist by hostname, IP address, or other options.

Bastion host

Check syslog for session information related to the telnet attempts

marmotstudios@sst_labs:~ $  cat /var/log/syslog
Oct 19 14:29:41 bastionzero systemd[1]: session-6.scope: Succeeded.
Oct 19 14:29:41 bastionzero systemd[1]: session-6.scope: Consumed 2.763s CPU time.
Oct 19 14:29:44 bastionzero systemd[1]: Started Session 8 of user marmotstudios.
Oct 19 14:30:39 bastionzero systemd[1]: session-8.scope: Succeeded.

Use tcpdump to see if any log messages are being sent.

You can install tcpdump and check if logs are being sent. These logs will not be exclusive to the SSH service.

marmotstudios@sst_labs:~ $ sudo tcpdump -nni any port 601
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
14:30:21.826038 eth0  Out IP 192.168.1.6.41189 > 192.168.100.5.514: Flags [P.]
...truncated...

You can expand the tcpdump command to show payloads by updating the command to sudo tcpdump -s0 -nnXi any port 601

Appendix #

Out of scope #

  • Syslog-ng client filtering. There are options to adjust what is sent to your syslog server
  • Syslog-ng SSL
  • How to exit vim
  • expanded sshd_config settings for best practice

References #

  1. https://docs.qnap.com/nas-outdated/4.1/Home/en/index.html?syslog_server.htm , https://www.asustor.com/online/College_topic?topic=272#sl1
  2. https://www.rfc-editor.org/rfc/rfc5426 <514 transpo>
  3. https://www.raspberrypi.com/software/operating-systems/
  4. https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

Additional tools

  1. vim, not installed by default. sudo apt get install vim to install