Log Analyzer

Background.

 

Are you tired of reviewing log files? I love to develop software, but hate to spend hours checking logs to triage and fix defects. It took me years to realize that a text editor was not the right tool for my log analysis. One day I finally started looking for tools on the Internet. During my search found two main categories:

  • Server Log Analyzer: Analyzes logs to create a report that shows number of visits, navigation paths, service times and similar data.
  • SIEM Tools: (Security Information and Event Management) captures, organizes and analyze server logs looking for security threats.

I’m sure these tools are great, but didn’t work for me because they were designed to perform a different task. What I needed was an application log parser. More flexible than the server log analyzer but way simpler than the SIEM tools.

That is why I created my own log analysis software  (or log parser) with the following goals on mind:

  • Easily find key log messages.
  • Use those findings to browse the log to gather more details.
  • Remember search patterns to avoid typing.
  • Store key messages to document the failure’s root cause.

I’ve been using my log parser for several months to perform log analysis of serial port (RS232)  logs in order to  triage and fix embedded software defects. Since I started using my log parser  I noticed the following benefits:

  • Reduced log analysis time by half.
  • Improved diagnostic accuracy.
  • Simplified documentation and communication of root causes.

It worked so good for me that I decided to share it. I believe it could be useful to other colleagues because:

  • It is adaptable to any software. (Because it allows user to generate its own rules)
  • Can be used to debug any type of software (embedded or not) created with c, c++, Java, basic, python or any other programing language.
  • It can analyze logs from any source. (Captured with a digital analyzer, received trough USB or serial ports, written by a PC, etc)

 

What is it?

It is a graphical tool that uses user defined rules to index log files and browse trough the log in order to to isolate and understand failures. (I like to think about my log analysis tool as a combination of grep, sort & cat under a single graphical interface) I guess it could also be described as “universal log analyzer” or “generic application log parser”.

 

How does it work?

The LogAnalyzer uses three basic concepts:

  • Rule: Defines a list of strings that must be present on a log line for it to become a match.
  • Match: Represents a line of the log that matched a rule’s criteria.
  • Log: The file to be analyzed.

Each concept is reflected in a section of the log parser:

 log parser / log analyzer / graphical log analysis tool

  • Rule set: (left side) contains all the rules that can be applied to a log. User can define as many rules as needed and store them in different rule sets.
  • Index: (lower right) an ordered list of all the matches. It can be exported as text, in order to document and share failure signatures.
  • Log viewer: (upper right) displays the section of the section of the log file referenced by the selected index element.

 

Want to learn more?

Do you think my log analysis tool (or log parser) might be useful for you? You can learn more about it by: