Cleaning up Error logs and Dealing with vague Exceptions like “Object Reference not set to an instance of an Object”

In my experience in most cases when finding an issue with a dot NET app 90% of issues will have a corresponding exception, this exception might not be the root cause but can lead you to the root cause if you follow it, having clean error logs is essential if you want to get to the root cause fast.

The approach I take with getting to the root cause is pretty straight forward and systematic, but I find a lot that developers don’t seem to grasp troubleshoot concepts like “Process of Elimination” very well, which are common more in the Infrastructure and support side of IT, so I wanted to give a small overview for my Devs.

The steps I usually follow are:

  1. Check the line of code from the stack trace
  2. If not enough data, add more and wait for the exception to occur again
  3. If enough data then fix the problem

This is assuming you cant replicate the exception.

If you take this approach you will eventually get to the root cause and fix an exception, the next step after that is apply this to “every” exception and clean up your logs.

I will use an example of a recent issue I fixed.

I recently worked on one of our selenium testing projects, which was having some intermittent issues that we couldn’t trouble shoot and there was nothing in the Error Logs, when this happens my initial reaction is to look for code that is swallowing Exceptions,  and low and behold I find code like the below that I fixed up.

emptycatchexception

I fixed about 5 of these in key areas, now I start getting data into my logs, and I get the age old “NullReferenceException”

nullreferenceexception

Looking at the line of code, i can’t tell much, so we are using Serilog, so its easy just to throw some additional objects into the Log.Error method in order to better understand what was going on at run time. In this example the null reference exception was on the line of code that accesses the driverPool object, so I pass this in like below.

logtheobjectthatiscausingthenullreferenceexception

Now I wait for the exception to occur again and then check logs.

findingtherootcauseofexception

Now i can see the driver pool is a list of null objects. The reason for this will be earlier in the code, but this particular case is filling up the logs so we need to clean it up. So in this situation what i would do is do  null check then log a warning message and not attempt the code that is accessing it.

The reason for this is that this method does a “clean up” to dispose the object, but something else has already disposed it, there is not hard set rule here, you need to judge thee on a case by case basis.

fixingtherootcauseofexception

After waiting again my error logs are a lot more clean now and I can start to see other exception that I couldn’t see before, below you can visibly see the decrease in volume after I fixed this and a few other errors up as well, after 10:15 when my code was deployed i have 1 exception in the next hour, as opposed to hundreds per  minute. My warning logs are a bit more noisy, but that’s ok because I am trying to filter out the noise of the less important errors by using warnings.

errorlogvolume

 

Handling Client Side error in AngularJS and sending them Server Side

I did an example in my AzureDNS App today of how to add a ApiController that will log client side error to a server side store.

Coming from a C# background I am used to having a central logging store (e.g. SQL table or SEQ more recently) that all errors are dumped too. When running code on a server its easy for an app to throw its logs into something that is behind the firewall and catalogs the logs from all your apps nicely.

When you have code that is running in someone’s web browser that not always as easy, but what I’ve done with this example is just create a controller on /api/Log/Error within the running app that i can throw my JavaScript Exceptions at, then add a call from Angular’s global exception handler.

The Log Controller can be found here

https://github.com/HostedSolutions/AzureDNSUI/blob/master/src/HostedSol.AzureDNSUI.Web/Controllers/LogController.cs

The log error method is pretty straight forward, the logger object is passed in via DI from autofac


[Route("~/api/Log/Error")]
[HttpPost]
// POST: api/Log/Error
public void PostError([FromBody]string value)
{
_logger.Error(value);
}

The the global error handler in Angular is done as follows

https://github.com/HostedSolutions/AzureDNSUI/blob/master/src/HostedSol.AzureDNSUI.Web/App/Scripts/fac/loggerSvc.js


'use strict';
angular.module('AzureDNSUI') // This would be changed to your app name if reusing this code
.factory('$exceptionHandler', function($injector) {
return function(exception, cause) {
var $http = $injector.get("$http");
var $log = $injector.get("$log");
exception.message += ' (caused by "' + cause + '")';
$log.log(exception); // Logs to console
$http.post('/api/Log/Error', JSON.stringify(exception)); // Where the magic happens
throw exception;
};
});

An example error from Angular in our SEQ server below:

SEQExampleOutputFromAngularJS

There is a few fields that we can customize that I do by default from the Serilog config, some of these are not relevant for the javascript errors, for example the SourceContext will always be the same, I’ll do a follow-up post about collecting client information to pass through later.