Elks? In my home network?
I've been running a home lab for around 6 months. More than any side project I've done, I've learned a lot. In terms of upskilling I'd put this experience somewhere north of completing 2 certifications and south of a year working in a tech startup.
My initial goals were humble:
- replace my google nest router with something a little more time-consuming
- move my side projects off my poor long-suffering EC2 t2.micro
- provide a rationale for stockpiling ThinkCentre M920qs
So I got started with proxmox and opnsense and, well, one fully-featured self-hosted data centre here I am. Older, wiser, and with a world-weary mistrust of strangers borne out of watching network logs in real-time.
On that note, flush with the success of implementing DHCP, firewall rules, DNS, VLANs, I decided I didn't like the out-of-the-box firewall log viewer in opnsense. On a whim I thought I'd just spin up a VM with the venerable ElasticSearch/Logstash/Kibana stack, add some custom logic (gist) to decode the opnsense rule output and add the rule name, and highlight in glaring red these sorts of outrageous home invasion attempts:
More than any other home lab decision, implementing a discoverability solution that cynics might say is a little over-engineered for my purposes, this has cost an outsize amount of time in configuration and debugging.
Here are some of the lessons I've learned:
Document everything... then read it
A few weeks after installing and configuring ELK, annoyed that my nginx logs were not being parsed as JSON, I spent a good few hours reconfiguring and testing filebeat, nginx and logstash. Nothing worked. Each service was working in isolation, and there weren't any network issues, but still my log was being displayed in kibana as a lump of ugly text. Eventually I found filebeat was working just fine - except it was pointed at the wrong address. So where were the logstash logs coming from?
Had I stopped to read my notes I would have recalled trying out syslog-ng as a filebeat replacement - and leaving it in place. Being a habitual note-taker, I'd put this down, but when trying to fix the issue, had jumped onto the machine, looked at the command history and started tinkering away.
Documentation is an essential part of networking engineering. And documentation works even better when read. I'm more mindful these days to stop and review before diving in.
Have a strategy for logging
This was quite the change of perspective. As a career QA, logs are frequently my first port of call. Splunk is the best platform I've used in this area but whatever the tool - Kibana, Athena, S3 Select, CloudWatch or a cheeky sed,awk or grep - scrutinising the logs will always reveal something you didn't know. I could never understand why companies didn't keep a good year of logs at hand (at least!) - think of the insights you could glean!
As an amateur network engineer, logs are like a fast-growing garden. Without a degree of regular care, and without a plan, it gets messy.
I initially devoted a 10GB drive to my elasticsearch store, with the idea of maybe provisioning more space in a couple of years. It filled it within the first week. If you're logging every request across your network, not only will it be rather a lot of data - it just won't be searchable. Many's the time I have re-ingested the index and heard the pitiable sound of the m920q's miniature fan cranking into high f#.
I've tamed my digital hoarder's desire to log all the things, and with logrotate and elastic search lifecycles it's a lot more manageable.
Start small
I'm astounded at the technology we have access to today. Not too long ago, a production-like testing web-server was likely a dedicated rack-mounted monster costing tens of thousands, and constant attention. These days it's containers and cloud computing in disposable environments costing next to nothing.
Proxmox takes that one step further - bare metal virtualisation meaning you can run a personal data center on minimal hardware specs from your living room. When you have industrial-grade big tech at your fingertips, it's easy to get a little too enthusiastic. Kubernetes, ElasticSearch, Jenkins, Kerberos... why not? Spin it up!
Well, there is a significant price tag to each. It would be hard to overestimate the time taken to implement even just my ELK stack. I'm glad I jumped into the self-hosting/home lab journey boots and all - it really has been an education - but after the initial investment I'll weigh up the time cost of adding that new component, no matter how useful it seems.