Ingesting IIS logs into Elasticsearch with Geo tags

I’ve seen a lot of guides and posts regarding getting IIS logs into Elasticsearch (ES), before using it in Kibana/Grafana’n shit. And somehow I’ve managed to make all of them fail.

It’s normally on the GEO tagging that fails to be inserted the right way. But after some testing and debugging I finally got it working.

If you know your way around ES you can skip the rest and use the ingestion template and move along – it can be found here: https://github.com/SuperJoachim/BlogSamples/tree/master/ElasticSearch

For the rest of us:

 

Before we can get this working, we need a few things:

  • A working ES database – the easy way is to use https://www.elastic.co/cloud
  • ingest-geoip and ingest-user-agent plugin for ES installed.
  • filebeats installed on the IIS server – or some other tool to move data to ES.
  • Enable IIS logging and select the “right” fields if using my grok template.
  • Make filebeat log to the correct pipeline.
  • Access to the console for ES somehow – I’m using Kibana in this example.
  • Do some magic.

The first 3 steps I’ll skip since there’s 100000 of guides for these things on the fine internet!

 

IIS Logging

I’ve selected the following in my IIS with a month retention:

Fields to log in IIS

 

Make filebeat log to the correct pipeline

It’s a small thing – we just need to remember it:

In my filebeat.yml I’ve the following to make sure it’s ingesting into the correct pipeline:

It’s fairly simple.

 

Now the magic begins.

You need to create an index template in ES. Mostly because sc_bytes, time_taken won’t be integer and geoip.location will not be a geo_point.

Lucky for you – I’ve done all the work:

Put the following in to your console / dev tools in Kibana :

Source: https://github.com/SuperJoachim/BlogSamples/blob/master/ElasticSearch/indextemplate.json

 

Now we need to create the ingest pipeline. This will process the input from your filebeat input, delete some fields, do some geotagging and other magical stuff:

Run the following in console / dev tools in Kibana:

Source: https://github.com/SuperJoachim/BlogSamples/blob/master/ElasticSearch/IISIngestion.json

Under the “remove” section I’ve deleted some of the entries I don’t need. You can add/remove as you like. But remember, to be GDPR compliant, you need to remove the c_ip – unless you have a really good reason to keep it 🙂

And that’s it! Now you should be able to see some awesome data in your ES:

 

Happy logging!

Leave a Reply

Your e-mail address will not be published. Required fields are marked *