What to log?

Print (or trace) debugging is the act of watching (live or recorded) trace statements, or print statements, that indicate the flow of execution of a process. This is sometimes called printf debugging, due to the use of the printf statement in C. This kind of debugging was turned on by the command TRON in the original versions of the novice-oriented BASIC programming language. TRON stood for, “Trace On.” TRON caused the line numbers of each BASIC command line to print as the program ran.

After more than 10 years of using logs to triage defects on my software I’ve identified two key elements to make this technique efficient:

  1. Selecting the trace statements wisely.
  2. Having a good method to analyze the logs.

   Defining the trace statements is a balancing act. On one side you want to add as few messages as possible to reduce the performance impact and keep the size of the logs manageable. But at the same time you want to make enough statements to understand  all the potential bugs.

A very useful technique is to combine the concepts of:

  • Debug channels.
  • Debug levels.

A debug channel typically has a short (3 or 4 characters) identifier and is used for all the trace statements of one of the modules of your software. Each channel has its own debug level (typically represented by a number). Each developer can define its own debugging levels. But I’ve found very useful to have at least the following ones:

  • Off: This channel will not add anything to the log.
  • Low: Only very important messages will be logged. (Like user interactions, execution of main algorithms and detection of errors)
  • Medium: A lot of extra information is added on top of the low messages. (Like entry and exit points of key functions)
  • High: Even more information is added on top of the medium messages. (Maybe changes on variables, or execution paths)

The keys to maximize the efficiency of this technique are to:

  • Set the level of each debug channel to low by default.
  • Provide “back door commands” to change the level of any channel at run time.

This allows developers to start the log analysis with the default logs (each channel reporting in low level) hoping that it will have enough information. And request the testers to change the report level of some of the channels in order to get more information if needed.

On embedded software (programmed in c or c++) it is common practice to use macros to implement the trace functionality. Mainly because loggin resources change between projects and macros are easy to compile out for release versions. But you can also use a tracing library or module instead of starting from scratch. This option becomes even more appealing if your software is running on a computer with more resources. You can find a tracing library for any modern high level programing language (.net, java, python, etc)

Choosing the right trace statements might seem overwhelming at the beginning, specially because it is really hard to anticipate (guess) what bugs you will end up triaging in the future. I personally find comfort in thinking about it as a process that will be tuned; where I start with my best guess and then I keep adding/removing trace statements as needed with every internal release.

Once the software is ready to be released to customers you want to turn your debug channels off to avoid confusing users and helping competitors. This can be achieved by:

  1. Compiling out all the trace statements on the release build.
  2. Setting the debug level of all channels to off on the release build.

The advantage of the first method is that you eliminate any performance impact produced by the trace statements. The advantage of the the second method is that you can enable the debug channels at run time if (when) you have to triage an issue at the customer site.

Keep in mind that having the information on the logs will be useless if you don’t have a good log analysis method . To optimize this task I created a log parser (or log analyzer). If you think this tool might be useful for you feel free to learn more about it or download it for a free 30 day trial.

This entry was posted on Friday, July 26th, 2013 at 7:11 pm and is filed under LogAnalyzer. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

Comments are closed.