Painless and secure Windows Event Log delivery with Fluent Bit, Loki and Grafana
If you are like us and you are happily coding away on your new IaC projects, you’ve surely spent some time thinking about your new centralized logging infrastructure. If you’ve hit the jackpot and all the services you are deploying are "cloud native", there is nothing to worry about (apart from spending copious amounts of money). Excellent tools are waiting for you and deployment of logging solutions is often integrated.
Things become a little less shiny when the (oh so dreaded) legacy systems rise from long forgotten resting places in the backyard of your infrastructure. But fear not, fellow cloud warrior, the open source community has built some awesome integrations like fluentbit, fluentd or traefik. In this blog entry, we show how we integrated our legacy Windows Server (Active Directory) into our new cloud logging infrastructure.
Our cloud logging infrastructure is composed of the following components:
We also have Prometheus in the mix, but for this writeup, we will concentrate on logging and assume monitoring "just works".
All servers are secured with TLS, Loki is fronted with HAProxy to get TLS working. The syslog-ng service is also secured with TLS and certificate-based authentication.
Shipping logs from Windows and other legacy servers
(Cloud) Empires are not build in a day, and server infrastructure evolves over time. In foresight, we already deployed our syslog-ng server to gobble up all the logs we can’t process directly with Promtail. The syslog-ng server forwards the received logs directly to Promtail and they end up in Loki, just like all the other logs. There are some good syslog implementations for Windows, like rsyslog, but - after an initial trial - we decided to use Fluent Bit instead.
Reasons for this are:
rsyslog (and other syslog implementations) are mostly closed source and licenced
Information about fields and mappings often get lost in translation due to the "syslog way"
Fluent Bit ships directly to Loki and parses all the fields of the Windows Event Log for us nicely
How to set up Fluent Bit on Windows to securely send logs to Loki
For the installation of Fluent Bit, check the excellent instructions on their website.
The gist of it is:
Download the Fluent Bit
.zipfile on your windows server
Verify the checksum of the downloaded file with the one posted on the download page (important!)
Open a PowerShell window in the installation directory (as an Administrator, we will need it in the next steps)
Create a new file in the Fluent Bit
In this example, we will call it
win-to-loki.conf and fill it with the following contents:
[INPUT] section tells Fluent Bit to pull the logs from the Windows Event Log every second.
We are storing the position of the delivered logs in the
Note that if you delete this file all the logs are deliverd agin.
Channels specify which kind of events should be shipped.
Check your Windows Event Viewer for a complete List of channels on your system.
Security are default channels.
[OUTPUT] section defines our connection to Loki.
We use a secure connection to the HAProxy fronted Loki which is secured with basic auth.
Finally, you can define some labels.
You can test your log delivery first by replacing the
[INPUT] section with:
Now fire up Fluent Bit from the installation directory with:
The dummy log line should arrive in Loki.
If it does, change your configuration back to the
winlog input, and make sure you selected the channels you want to deliver to Loki.
Creating the Windows service
If you are all happy and excited now, it is time to wrap things up on the Windows side and install Fluent Bit as a Windows service. Just execute:
Note that the space after "binpath=".
This is mandatory!
If you want to know why, ask the good people over at stackoverflow
Now you can start the service from your Windows GUI or from the command line with
sc.exe start fluent-bit.
After you started the service, all selected logs will arrive in Loki and you can use the might of Grafana to process and filter them: