Painless and secure Windows Event Log delivery with Fluent Bit, Loki and Grafana

February 1, 2021 by Christian Eibl

#DevOps #Infrastructure #Cloud

If you are like us and you are happily coding away on your new IaC projects, you’ve surely spent some time thinking about your new centralized logging infrastructure. If you’ve hit the jackpot and all the services you are deploying are "cloud native", there is nothing to worry about (apart from spending copious amounts of money). Excellent tools are waiting for you and deployment of logging solutions is often integrated.

Things become a little less shiny when the (oh so dreaded) legacy systems rise from long forgotten resting places in the backyard of your infrastructure. But fear not, fellow cloud warrior, the open source community has built some awesome integrations like fluentbit, fluentd or traefik. In this blog entry, we show how we integrated our legacy Windows Server (Active Directory) into our new cloud logging infrastructure.

Logging Infrastructure

Our cloud logging infrastructure is composed of the following components:

Grafana as a frontend to query, slice & dice
Grafana Loki as our central log aggregator
Promtail, for shipping the logs to Loki
syslog-ng, when shipping via Promtail is not possible

We also have Prometheus in the mix, but for this writeup, we will concentrate on logging and assume monitoring "just works".

All servers are secured with TLS, Loki is fronted with HAProxy to get TLS working. The syslog-ng service is also secured with TLS and certificate-based authentication.

All our infrastructure is deployed via Terraform and Terragrunt and is currently hosted in AWS. This gives us easy service discovery for Loki (similar to Prometheus) and all our EC2 instances send their logs to Loki automatically.

Shipping logs from Windows and other legacy servers

(Cloud) Empires are not build in a day, and server infrastructure evolves over time. In foresight, we already deployed our syslog-ng server to gobble up all the logs we can’t process directly with Promtail. The syslog-ng server forwards the received logs directly to Promtail and they end up in Loki, just like all the other logs. There are some good syslog implementations for Windows, like rsyslog, but - after an initial trial - we decided to use Fluent Bit instead.

Reasons for this are:

rsyslog (and other syslog implementations) are mostly closed source and licenced
Information about fields and mappings often get lost in translation due to the "syslog way"
Fluent Bit ships directly to Loki and parses all the fields of the Windows Event Log for us nicely

How to set up Fluent Bit on Windows to securely send logs to Loki

Installation

For the installation of Fluent Bit, check the excellent instructions on their website.

The gist of it is:

Download the Fluent Bit .zip file on your windows server
Verify the checksum of the downloaded file with the one posted on the download page (important!)
Open a PowerShell window in the installation directory (as an Administrator, we will need it in the next steps)

Configuration

Create a new file in the Fluent Bit config directory. In this example, we will call it win-to-loki.conf and fill it with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


[SERVICE]
    flush     1
    log_level info

[INPUT]
    Name                   winlog
    Channels               System,Security
    Interval_Sec           1
    DB                     winlog.sqlite

[OUTPUT]
    name                   loki
    match                  *
    host                   loki.host.name
    port                   443
    tls                    On
    tls.verify             On
    tls.debug              0
    http_user              user
    http_passwd            password
    labels                 job=winlog

The [INPUT] section tells Fluent Bit to pull the logs from the Windows Event Log every second. We are storing the position of the delivered logs in the winlog.sqlite database. Note that if you delete this file all the logs are deliverd agin.

The Channels specify which kind of events should be shipped. Check your Windows Event Viewer for a complete List of channels on your system. The channels System and Security are default channels.

The [OUTPUT] section defines our connection to Loki. We use a secure connection to the HAProxy fronted Loki which is secured with basic auth. Finally, you can define some labels.

Testing

You can test your log delivery first by replacing the [INPUT] section with:

1
2
3
4


[INPUT]
    name      dummy
    dummy     {"key": 1, "value": "Knock, knock... who's there?"}
    samples   1

Now fire up Fluent Bit from the installation directory with:

1

.\bin\fluent-bit.exe -c .\conf\win-to-loki.conf

The dummy log line should arrive in Loki. If it does, change your configuration back to the winlog input, and make sure you selected the channels you want to deliver to Loki.

Creating the Windows service

If you are all happy and excited now, it is time to wrap things up on the Windows side and install Fluent Bit as a Windows service. Just execute:

1

sc.exe create fluent-bit binpath= "\your-path\td-agent-bit\bin\fluent-bit.exe -c \your-path\td-agent-bit\conf\win-to-loki.conf"

Note that the space after "binpath=". This is mandatory! If you want to know why, ask the good people over at stackoverflow Now you can start the service from your Windows GUI or from the command line with sc.exe start fluent-bit.

Results

After you started the service, all selected logs will arrive in Loki and you can use the might of Grafana to process and filter them: