This post will be edited over time, please feel free to come back and check for new content.
Last edit: 11-25-2024
Goal: A HomeLab setup that protects itself
This example HomeLab has at its core an OPNSense Router, smart switches with subnet zones, several VMs, a few Docker environments, and specifically for this version of this post: with cron updating near all Rules enabled Suricata, and after design of Zones, Firewall rules, Port Forwards, and the like, we installed OPNSense’s CrowdSec plugin.
Once installed, as we will have additional agents on the various services in your HomeLab, we will likely want to transition from the SQLite DB to a Postgres DB or something similar.
Near all additional services are actually deployed as Docker Stacks or Containers.
Further, we have designed this using MACVLAN as we will have an IDS involved and wish to parse the network traffic as is via Suricata. Extra to note, we designed this with an SSL Terminating Reverse Proxy in front of some of the publicly available ‘HomeLab Services’ allowing Suricata do see everything.
The DB has been setup on a subnet adjacent Docker environment, and I’ve used the ‘Custom LAPI’ config option in the OPNSense GUI CrowdSec plugin settings page, plus a persistent ‘/usr/local/etc/crowdsec/config.yaml.local’ that CrowdSec supports (it only replaces the configs in the original config.yaml with what you set in it). Will expand on this portion as it is likely of hot interest shortly as it also enables the rest of the Multi-Server setup. The ‘config.yaml.local’ and CrowdSec and a Postgres DB with Multi-Server should get you close on a Google search, I will update this detail on this page in time.
Now we are going to skip a few details about the beginning, many middle sections, and reach the most recent realization….
Suricata EVE without payload perfect for CrowdSec Parser
The ‘printable_payload’ element of the JSON output of eve.json is a bit much for most any parser, with this in mind, it is highly advisable to enable an additional EVE output from Suricata without this field for CrowdSec’s agent to easily parse the state and respond quickly.
In the example below we demonstrate doing so with an overwrite setting for the Reverse Proxy that is part of the network stack.
To do this, persistently, you must create or edit the following files on your OPNSense router
Step 1
The OPNSense CrowdSec plugin installs observing a few default logs from OPNSense (lighttpd/sshd/pf) but does not come configured for any Suricata log listening. A CrowdSec Acquis file must be created or modified to get the feature we are adding here and without this, the evexff.json file will go unobserved, you will also need the CrowdSec Hub elements to enable the parsing/alerting for Suricata, so console into the OPNSense and enter the following command (ssh in, select option ‘8’):
cscli collections install crowdsecurity/suricata
cscli collections install crowdsecurity/whitelist-good-actors
cscli parsers install crowdsecurity/whitelistsThere is an additional file that will help prevent you from banning yourself from yourself, make the next file and paste the content.
/usr/local/etc/crowdsec/postoverflow/s01-whitelist/mydomain-whitelist.yaml:
name: my/whitelist ## Must be unqiue
description: "Whitelist events from my dynamic IP"
whitelist:
reason: "My dynamic IP"
expression:
- evt.Overflow.Alert.Source.IP in LookupHost("my.domain.com")Then after creating the following file, it is my recommendation to hit ‘save’ on the CrowdSec plugin GUI of your OPNSense, this appears to reload instead of restarting the service as desired.
/usr/local/etc/crowdsec/acquis.d/suricata.yaml:
---
filenames:
- /var/log/suricata/evexff.json
labels:
type: suricata-evelogs
---Step 2
Now that CrowdSec is aware and ‘listening’ if you will, we will want to create, and rotate those evexff.json logs, let’s setup the rotation of ‘/var/log/suricata/evexff.json’ as a custom config
/usr/local/etc/newsyslog.conf.d/suricataxff.conf:
# logfilename [owner:group] mode count size when flags [/pid_file] [sig_num]
/var/log/suricata/evexff.json root:wheel 640 1 500000 $W0D23 B /var/run/suricata.pid 1Step 3
To note: It appears the ‘custom.yaml‘ file you will likely edit, needs the entire ‘output:’ stanza of the original (/usr/local/etc/suricata/suricata.yaml) Suricata config and then edit as desired – the example below is the working ‘custom.yaml’ with the upgraded edit being the additional EVE ouput.
Worst case currently, an admin will have to be aware of updates to the original and diff the two wisely, as I develop that I will share here. It appears like you replace at ‘stanza’ level, so the other features (threading/etc.) of Suricata appear to be performing as expected, but, my experience on this feels suddenly fresher than it used to. It would possibly be better to have this config at the surface of the OPNSense available to the same spaces as the ‘EVE’ logs for Suricata. Might try to develop/contribute in the future – barely have time for this post lol. ^_^
/root/custom.yaml (/usr/local/etc/suricata/custom.yaml – more on this later):
%YAML 1.1
---
vars:
address-groups:
HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
EXTERNAL_NET: "!$HOME_NET"
HTTP_SERVERS: "$HOME_NET"
SMTP_SERVERS: "$HOME_NET"
SQL_SERVERS: "$HOME_NET"
DNS_SERVERS: "$HOME_NET"
TELNET_SERVERS: "$HOME_NET"
AIM_SERVERS: "$EXTERNAL_NET"
DC_SERVERS: "$HOME_NET"
DNP3_SERVER: "$HOME_NET"
DNP3_CLIENT: "$HOME_NET"
MODBUS_CLIENT: "$HOME_NET"
MODBUS_SERVER: "$HOME_NET"
ENIP_CLIENT: "$HOME_NET"
ENIP_SERVER: "$HOME_NET"
SIP_SERVERS: "$HOME_NET"
INTERNAL_DEVICELIST: "[10.30.16.41,10.30.16.42,10.30.16.43]"
port-groups:
HTTP_PORTS: "[80,8080]"
SHELLCODE_PORTS: "!80"
ORACLE_PORTS: 1521
SSH_PORTS: 22
DNP3_PORTS: 20000
MODBUS_PORTS: 502
FILE_DATA_PORTS: "[$HTTP_PORTS,110,143]"
FTP_PORTS: 21
GENEVE_PORTS: 6081
VXLAN_PORTS: 4789
TEREDO_PORTS: 3544
SIP_PORTS: "[5060,5061]"
outputs:
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: evexff.json
xff:
enabled: yes
mode: overwrite
deployment: reverse
header: X-Forwarded-For
types:
- alert:
payload: no
payload-buffer-size: 100kb
payload-printable: no
metadata: yes # enable inclusion of app layer metadata with alert. Default yes
tagged-packets: yes
- eve-log:
enabled: yes
filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
filename: eve.json
pcap-file: false
community-id: true
community-id-seed: 0
xff:
enabled: yes
mode: extra-data
deployment: reverse
header: X-Forwarded-For
types:
- alert:
payload: no
payload-buffer-size: 100kb
payload-printable: yes
metadata:
app-layer: true
flow: true
rule:
metadata: true
raw: true
tagged-packets: yes
- alert:
payload: yes
payload-buffer-size: 100kb
payload-printable: yes
metadata: yes # enable inclusion of app layer metadata with alert. Default yes
metadata:
app-layer: true
rule:
metadata: true
raw: true
tagged-packets: yes
- frame:
enabled: no
- anomaly:
enabled: no
- eve-log:
enabled: yes
type: syslog
identity: "suricata"
facility: local5
level: Info
community-id: true
community-id-seed: 0
xff:
enabled: yes
mode: extra-data
deployment: reverse
header: X-Forwarded-For
types:
- alert:
payload: no
payload-buffer-size: 100kb
payload-printable: yes
metadata:
app-layer: true
flow: true
rule:
metadata: true
raw: true
tagged-packets: yes
- frame:
enabled: no
- anomaly:
enabled: no
types:
applayer: no
- unified2-alert:
enabled: no
- http-log:
enabled: no
filename: http.log
append: yes
- tls-log:
enabled: no # Log TLS connections.
filename: tls.log # File to store TLS logs.
append: yes
- tls-store:
enabled: no
- pcap-log:
enabled: no
filename: log.pcap
limit: 1000mb
max-files: 2000
compression: none
mode: normal # normal, multi or sguil.
use-stream-depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets
honor-pass-rules: no # If set to "yes", flows in which a pass rule matched will stop being logged.
- alert-debug:
enabled: no
filename: alert-debug.log
append: yes
- alert-prelude:
enabled: no
profile: suricata
log-packet-content: no
log-packet-header: yes
- stats:
enabled: yes
filename: stats.log
append: yes # append to file (yes) or overwrite it (no)
totals: yes # stats for all threads merged together
threads: no # per thread stats
- file-store:
version: 2
enabled: no
xff:
enabled: no
mode: extra-data
deployment: reverse
header: X-Forwarded-For
- file-store:
enabled: no
- tcp-data:
enabled: no
type: file
filename: tcp-data.log
- http-body-data:
enabled: no
type: file
filename: http-data.log
- lua:
enabled: no
scripts:
app-layer:
protocols:
telnet:
enabled: yes
rfb:
enabled: yes
detection-ports:
dp: 5900, 5901, 5902, 5903, 5904, 5905, 5906, 5907, 5908, 5909
mqtt:
enabled: yes
krb5:
enabled: yes
bittorrent-dht:
enabled: yes
snmp:
enabled: yes
ike:
enabled: yes
tls:
enabled: yes
detection-ports:
dp: 443
ja3-fingerprints: auto
pgsql:
enabled: yes
stream-depth: 0
max-tx: 1024
dcerpc:
enabled: yes
ftp:
enabled: yes
rdp:
ssh:
enabled: yes
http2:
enabled: yes
smtp:
enabled: yes
raw-extraction: no
mime:
decode-mime: yes
decode-base64: yes
decode-quoted-printable: yes
header-value-depth: 2000
extract-urls: yes
body-md5: no
inspected-tracker:
content-limit: 100000
content-inspect-min-size: 32768
content-inspect-window: 4096
imap:
enabled: detection-only
smb:
enabled: yes
detection-ports:
dp: 139, 445
nfs:
enabled: yes
tftp:
enabled: yes
dns:
tcp:
enabled: yes
detection-ports:
dp: 53
udp:
enabled: yes
detection-ports:
dp: 53
http:
enabled: yes
libhtp:
default-config:
personality: IDS
request-body-limit: 100kb
response-body-limit: 100kb
request-body-minimal-inspect-size: 32kb
request-body-inspect-window: 4kb
response-body-minimal-inspect-size: 40kb
response-body-inspect-window: 16kb
response-body-decompress-layer-limit: 2
http-body-inline: auto
swf-decompression:
enabled: no
type: both
compress-depth: 100kb
decompress-depth: 100kb
double-decode-path: no
double-decode-query: no
server-config:
modbus:
enabled: yes
detection-ports:
dp: 502
stream-depth: 0
dnp3:
enabled: yes
detection-ports:
dp: 20000
enip:
enabled: yes
detection-ports:
dp: 44818
sp: 44818
ntp:
enabled: yes
quic:
enabled: yes
dhcp:
enabled: yes
sip:
enabled: yes
asn1-max-frames: 256
datasets:
defaults:
rules:
host-os-policy:
windows: []
bsd: []
bsd-right: []
old-linux: []
linux: []
old-solaris: []
solaris: []
hpux10: []
hpux11: []
irix: []
macos: []
vista: []
windows2k3: []
netmap:
- interface: default
threads: auto
copy-mode: ips
disable-promisc: no # promiscuous mode
checksum-checks: auto
bpf-filter: not (( host 10.30.1.6 or 10.30.1.3 ) or ( net 10.30.14.0/24 or 10.30.21.0/24 ))
- interface: vlan0.1.10
copy-iface: vlan0.1.10^
- interface: vlan0.1.10^
copy-iface: vlan0.1.10
- interface: vlan0.1.50
copy-iface: vlan0.1.50^
- interface: vlan0.1.50^
copy-iface: vlan0.1.50
- interface: igb0
copy-iface: igb0^
- interface: igb0^
copy-iface: igb0
- interface: ix1
copy-iface: ix1^
- interface: ix1^
copy-iface: ix1
default-rule-path: /usr/local/etc/suricata/opnsense.rules
rule-files:
- suricata.rules
classification-file: /usr/local/etc/suricata/classification.config
reference-config-file: /usr/local/etc/suricata/reference.config
threshold-file: /usr/local/etc/suricata/threshold.configTo Note: My learning on “Netmap”, BPF, and Suricata is only just getting out of beginner/intermediate, not sure if the Netmap bits are doing things given that I have OPNSense set to IDS mode, but, I want to enable the BPF without having to modify the service file for Suricata in OPNSense, so far, this appears to be the best way. Do not blindly copy all of this as your “custom.yaml” the Netmap portion is going to be unique to YOUR router and you will have to ‘shim’ in the mod yourself.
Step 4
This part is the coming soon, I will provide a link to the Post at the Forum where I am figuring out how to resolve the nature of having more control over the Suricata configuration as well as, how to use suricata-update to keep the IDS rules files in order (and modified the way I want them).
OPNSense Forum:
https://forum.opnsense.org/index.php?topic=44128.0
Suricata Forum:
https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068