Written by Jamie Tanna
on 
CC-BY-NC-SA-4.0 Apache-2.0
2 mins

Parsing AWS ALB/ELB access logs into SQLite

Featured image for sharing metadata for article

When using AWS' Application Load Balancer (ALB)/Elastic Load Balancer (ELB), it's likely that at some point you'll want to perform some queries based on the raw access logs that AWS can provide.

It may be you're looking for some requests coming through from a certain User-Agent, or you want to find out the distribution of TLS ciphers in use.

I was sitting down to start and write something to convert these, as I didn't fancy using AWS Athena for the job.

Thinking I'd write a quick Go tool for it, I started by doing a cursory search online to see if anyone had a package that'd make interacting with SQLite easier, but I ended up finding a perfect solution for the problem, a tool for this, alblogs.

Not only does alblogs perform the translation of data to an SQLite database, but it also performs all the AWS calls for you, so you don't need to download bulk data, or calculate the path in the bucket you need.

For instance, we can query the jvt.me load balancer like so, which then drops us into an SQLite session, using a temporary location for the SQLite database:

./alblogs jvt.me
...
sqlite>

There are lots of handy CLI options, like specifying how many log files to look through.

Written by Jamie Tanna's profile image Jamie Tanna on , and last updated on .

Content for this article is shared under the terms of the Creative Commons Attribution Non Commercial Share Alike 4.0 International, and code is shared under the Apache License 2.0.

#blogumentation #sqlite #aws.

Also on:

This post was filed under articles.

Interactions with this post

Interactions with this post

Below you can find the interactions that this page has had using WebMention.

Have you written a response to this post? Let me know the URL:

Do you not have a website set up with WebMention capabilities? You can use Comment Parade.