Continuous Integration/Continuous Delivery Workshop

I hosted a Workshop on CI and CD on the weekend with the following overview of topics

  • Create Build definitions in TeamCity
    • C# net core 1.1 MVC/Web API Project
    • Typescript/Webpack
    • Unit Test NUnit and Mocha
    • Output Packages for Deployment
    • Update GitHub Pull Request status
  • Create deployments in Octopus
    • Deploy to Cluster (C# backend)
    • Deploy to CDN (React SPA)
    • Send an Alert to Slack

Below are the links:

Videos:

Slides:

https://github.com/tech-at-agoda/meetup-2017-05-13

 

Cleaning up Error logs and Dealing with vague Exceptions like “Object Reference not set to an instance of an Object”

In my experience in most cases when finding an issue with a dot NET app 90% of issues will have a corresponding exception, this exception might not be the root cause but can lead you to the root cause if you follow it, having clean error logs is essential if you want to get to the root cause fast.

The approach I take with getting to the root cause is pretty straight forward and systematic, but I find a lot that developers don’t seem to grasp troubleshoot concepts like “Process of Elimination” very well, which are common more in the Infrastructure and support side of IT, so I wanted to give a small overview for my Devs.

The steps I usually follow are:

  1. Check the line of code from the stack trace
  2. If not enough data, add more and wait for the exception to occur again
  3. If enough data then fix the problem

This is assuming you cant replicate the exception.

If you take this approach you will eventually get to the root cause and fix an exception, the next step after that is apply this to “every” exception and clean up your logs.

I will use an example of a recent issue I fixed.

I recently worked on one of our selenium testing projects, which was having some intermittent issues that we couldn’t trouble shoot and there was nothing in the Error Logs, when this happens my initial reaction is to look for code that is swallowing Exceptions,  and low and behold I find code like the below that I fixed up.

emptycatchexception

I fixed about 5 of these in key areas, now I start getting data into my logs, and I get the age old “NullReferenceException”

nullreferenceexception

Looking at the line of code, i can’t tell much, so we are using Serilog, so its easy just to throw some additional objects into the Log.Error method in order to better understand what was going on at run time. In this example the null reference exception was on the line of code that accesses the driverPool object, so I pass this in like below.

logtheobjectthatiscausingthenullreferenceexception

Now I wait for the exception to occur again and then check logs.

findingtherootcauseofexception

Now i can see the driver pool is a list of null objects. The reason for this will be earlier in the code, but this particular case is filling up the logs so we need to clean it up. So in this situation what i would do is do  null check then log a warning message and not attempt the code that is accessing it.

The reason for this is that this method does a “clean up” to dispose the object, but something else has already disposed it, there is not hard set rule here, you need to judge thee on a case by case basis.

fixingtherootcauseofexception

After waiting again my error logs are a lot more clean now and I can start to see other exception that I couldn’t see before, below you can visibly see the decrease in volume after I fixed this and a few other errors up as well, after 10:15 when my code was deployed i have 1 exception in the next hour, as opposed to hundreds per  minute. My warning logs are a bit more noisy, but that’s ok because I am trying to filter out the noise of the less important errors by using warnings.

errorlogvolume

 

Fluent C# for Unit Testing

I’ve been playing with different methods of this for a while now, ever since i got introduced to it by Shea when I was at Oztix.

Note:  in this example I am using the Shouldly library, which i recommend checking out.

I’ve seen a number of different implementations for Unit Testing and mocking specifically, but then all seem a bit complex, and also these days we run NUnit is massive degrees of parallel, and a lot of implementations use static, which causes issues in this instance.

So in introducing my new team to it I revisited it and came up with what i think is a rather simplified version using Moq, which we are using for mocking at Agoda – I miss NSubstitute 😦

When using fluent within Unit Testing the goal is, in my mind, to get your naming using the GivenWhenThen convention from BDD.  I don’t like creating “builder” classes as they dont selfdocument with good context of what you are doing.

So you test’s C# code should read something like this

GivenAPerson()
.WhenAngry()
.ThenHappinessLevelShouldBeLow();

So if we look at a more detailed example

I am going to create a person class with a happiness level property, and also something else to check, lets say Number of fingers.


public class Person
 {
 public Person()
 {
 //Set Defaults
 HappynessLevel = 5;
 NumberOfFingers = 10;
 }
 public int HappynessLevel{ get; set; }
 public int NumberOfFingers { get; set; }
 public void MakeAngry()
 {
 HappynessLevel = 1;
 }

public void MakeHappy()
 {
 HappynessLevel = 9;
 }
 }

Now I am going to create my test class in my unit test project to assist with Testing and mocking it.


 public class GivenAPerson
 {
 private Person _person;
 public GivenAPerson()
 {
 _person = new Person();
 }

public Person Then()
 {
 return _person;
 }

public GivenAPerson WhenAngry()
 {
 _person.MakeAngry();
 return this;
 }

public GivenAPerson ThenHappinessLevelShouldBeLow()
 {
 _person.HappynessLevel.ShouldBeLessThan(3);
 return this;
 }
 public GivenAPerson ThenNumberOfFingersShouldBeDefault()
 {
 _person.NumberOfFingers.ShouldBe(10);
 return this;
 }
 }

You can see that the methods all return “This” or the class itself, this is one of the core concepts in fluent C#. This allow us to achieve the desired outcome of style we are looking for above.

There is one method that doesn’t though, which is “Then”, I sue this in conjunction with shouldly, and I’ll get into why at the end.

So now lets look at putting it all together in the Test Fixture


 [TestFixture]
 public class TestFixtureExample
 {
 [Test]
 public void MyTest()
 {
 new GivenAPerson()
 .WhenAngry()
 .ThenHappinessLevelShouldBeLow()
 .ThenNumberOfFingersShouldBeDefault();
 }
 }

The first thing you will notice is that i use the word “new”, in the past I have used static to avoid this, but when using massive parallelism in NUnit you want to do this so you avoid static.

You will also note that I have added 2 checks to this test, this is to give an example of how you would use this to scale the framework.

Using this method you should be able to easily reuse your checks, in this example we verify that the persons number of fingers doesn’t change when they are angry, which is very odd, but it’s just to give an example how you can apply multiple Then or When to a single test.

Now what about the “Then()” method. I normally don’t write a method like what is in ThenNumberOfFingersShouldBeDefault(), because it is just a single value. Normally i use ThenBlahBlah() methods for complex checks, if the check is not complex i would write something like the following


 [Test]
 public void MyTest2()
 {
 new GivenAPerson()
 .WhenAngry()
 .ThenHappinessLevelShouldBeLow()
 .Then().NumberOfFingers.ShouldBe(10);
 }

This allows me a quick check for a single value at the end if I need to and avoids me writing methods.

 

 

Dependency Injection Recomendations

Recently started working with a project that has a class called “UnityConfiguration” with 2000 lines of this

container.RegisterType<ISearchProvider, SearchProvider>();

This fast becomes unmanageable, wait, I hear you say, not all Types are registered in the same way! Yes, and you won’t get away with a single line to wire-up your whole IoC container, but you should be able to get it way under 50 lines of code, even in big projects.

I prefer to go a bit Hungarian and file things via folders/namespaces by their types, then use the IoC framework to load in dependencies using this. This is because based on the Type is generally where you find the differences.

For example I put all my Attributes under a namespace called “Attributes”, with sub-folders if there is too many of course, as so on.

Below is an example of a WebApi application i have worked on in the past. This method is called assembly scanning and is in the Autofac doco here

var containerBuilder = new ContainerBuilder();
containerBuilder.RegisterAssemblyModules(
typeof(WebApiApplication).Assembly);

containerBuilder.RegisterAssemblyTypes(typeof(WebApiApplication).Assembly)
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Lib")).AsImplementedInterfaces();
containerBuilder.RegisterAssemblyTypes(typeof(WebApiApplication).Assembly)
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Attributes")).PropertiesAutowired();
containerBuilder.RegisterAssemblyTypes(typeof(WebApiApplication).Assembly)
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Filters")).PropertiesAutowired();

containerBuilder.RegisterApiControllers(Assembly.GetExecutingAssembly()).PropertiesAutowired();
_container = containerBuilder.Build();

You can see form the above code that things like the Attributes and filters require the Properties AutoWired as I use Property injection as opposed to the constructor injection, as these require a parameter-less constructor. So I end up with one line for each sub-folder in my project basically.

So as long as I keep my filing system correct I don’t have to worry about maintaining a giant “Configuration” class for my IoC container.

You can also make use of modules in Autofac by implementing the Module, I recommend using this for libraries external to your project that you want to load in. you can use the RegisterAssemblyModules method in Autofac in a similar way

 

 

 

 

 

GitHub Pull Request Merge Ref and TeamCity CI fail

GitHub has an awesome feature that allows us to build on the potential merge result of a pull request.

This allows us to run unit and UI tests against the result of a merge, so we know with certainty that it works, before we merge the code.

To get this working with TeamCity is a pain in the ass though.

Lets look at a basic workflow with this:

First we will look at two active pull request, and we are about to merge

MasterFeatureBranchGitHubFlow

Pull request 2 advertises the /head (actual branch) and /merge (result “if we merged”)

TeamCity say you should tie your builds to the /merge for CI, this will build the merge result, and I agree.

However lets look at what happens in GitHub when we merge in Feature 1.

MasterFeatureBranchGitHubFlowMergedFeature1

The new code goes into master, which will recalculate the merge result on Pull request 2. TeamCity correctly builds the merge reference and validates that the Pull Request will succeed.

However if we look in GitHub we will see the below

UpdateGitHubBranch

It now blocks you and prompts you to updates your branch.

After you click this, the /head and /merge refs will update, as it adds a commit to your branch and recalculates the merge result again, then you need wait for another build to validate the new commit on your branch.

MergeAndHeadRefOnGitHubBranchUpdate

This now triggers a second build.And when it completes you can merge.

The issues here is we are double building. There is two solutions as I see it,

  1. GitHub should allow you top merge without updating your branch
  2. TeamCity should allow you to trigger from one ref and build on a different one

I was able to implement the second result using a build configuration that calls the TeamCity API to trigger a build. However my preference would be number 1 as this is more automated.

BuildOffDifferentBranchFromTrigger

Inside it looks like this

BuildOffDifferentBranchFromTrigger1

Below is example powershell that is used in the trigger build, we had an issue with the SSL cert (even though it wasn’t self signed) so had to disable the check for it to work.

add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$buildBranch ="%teamcity.build.branch%"
Write-Host $buildBranch
$buildBranch = $buildBranch.Replace("/head","/merge")
$postbody = "<build branchName='$buildBranch'>
<buildType id='%TargetBuildType%'/>
</build>"
Write-Host $postbody
$user = '%TeamCityUser%'
$pass = '%TeamCityPassword%'

$secpasswd = ConvertTo-SecureString $pass -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($user, $secpasswd)

Invoke-RestMethod https://teamcity/httpAuth/app/rest/buildQueue -Method POST -Body $postbody -Credential $credential -Headers @{"accept"="application/xml";"Content-Type"="application/xml"}

You will see that we replace the branch name head with merge, so we trigger after someone clicks the update branch button only.

Also don’t forget to add a VCS trigger for file changes “+:.”, so that it will only run builds when there are changes.

VCSTriggerRule

We are running with this solution this week and I am going to put a request into GitHub support about option 1.

This is a really big issues for us as we have 30-40 open pull requests on our repo, so double building creates a LOT of traffic.

If anyone has a better solution please leave some comments.

 

 

 

 

TeamCity and avoiding redownloading of npm packages

I haven’t been able to find a better solution that this. My original post on stackoverflow here.

TeamCity clears out the node_modules folder on every build, so when running npm install it re-downloads them all. So I’ve moved to using a powershell step to backup and restore this folder and the end and start of each build respectively.

Here is the PS step i run at the start


param (
[string]$projId ,
[string]$WorkDir
)

if(Test-Path "c:\node_backup\$projId\node_modules")
{
Move-Item "c:\node_backup\$projId\node_modules" "$WorkDir\node_modules"
}

And here is the one at the end


param (
[string]$projId ,
[string]$WorkDir
)
If (!(Test-Path "c:\node_backup"))
{
mkdir "c:\node_backup"
}
If (!(Test-Path c:\node_backup\$projId))
{
mkdir "c:\node_backup\$projId"
}
If (Test-Path "c:\node_backup\$projId\node_modules")
{
Remove-Item "c:\node_backup\$projId\node_modules"
}

Move-Item "$WorkDir\node_modules" "c:\node_backup\$projId"

Then in the script arguments i pass this

-projId %system.teamcity.buildType.id% -WorkDir %NodeWorkingDirectory%

The NodeWorkingDir is a parameter I set in my projects that i use to tell node where my gulp file is.