Slack Bots – Merge Queue Bot

I recently did a talk about a Slack Bot we created to solve our issue of merging to master. Supaket Wongkampoo helped me out with the Thai translation on this one.

We have over 100 developers working on a single repository, so at any one time we have 20 odd people wanting to merge, and each need to wait for a build to complete in order to merge, then one merged the next person must get those changes and rerun. It quit a common scenario but I haven’t seen any projects doing this with this much frequency.

Slides are available here


Continuous Integration/Continuous Delivery Workshop

I hosted a Workshop on CI and CD on the weekend with the following overview of topics

  • Create Build definitions in TeamCity
    • C# net core 1.1 MVC/Web API Project
    • Typescript/Webpack
    • Unit Test NUnit and Mocha
    • Output Packages for Deployment
    • Update GitHub Pull Request status
  • Create deployments in Octopus
    • Deploy to Cluster (C# backend)
    • Deploy to CDN (React SPA)
    • Send an Alert to Slack

Below are the links:




Cleaning up Error logs and Dealing with vague Exceptions like “Object Reference not set to an instance of an Object”

In my experience in most cases when finding an issue with a dot NET app 90% of issues will have a corresponding exception, this exception might not be the root cause but can lead you to the root cause if you follow it, having clean error logs is essential if you want to get to the root cause fast.

The approach I take with getting to the root cause is pretty straight forward and systematic, but I find a lot that developers don’t seem to grasp troubleshoot concepts like “Process of Elimination” very well, which are common more in the Infrastructure and support side of IT, so I wanted to give a small overview for my Devs.

The steps I usually follow are:

  1. Check the line of code from the stack trace
  2. If not enough data, add more and wait for the exception to occur again
  3. If enough data then fix the problem

This is assuming you cant replicate the exception.

If you take this approach you will eventually get to the root cause and fix an exception, the next step after that is apply this to “every” exception and clean up your logs.

I will use an example of a recent issue I fixed.

I recently worked on one of our selenium testing projects, which was having some intermittent issues that we couldn’t trouble shoot and there was nothing in the Error Logs, when this happens my initial reaction is to look for code that is swallowing Exceptions,  and low and behold I find code like the below that I fixed up.


I fixed about 5 of these in key areas, now I start getting data into my logs, and I get the age old “NullReferenceException”


Looking at the line of code, i can’t tell much, so we are using Serilog, so its easy just to throw some additional objects into the Log.Error method in order to better understand what was going on at run time. In this example the null reference exception was on the line of code that accesses the driverPool object, so I pass this in like below.


Now I wait for the exception to occur again and then check logs.


Now i can see the driver pool is a list of null objects. The reason for this will be earlier in the code, but this particular case is filling up the logs so we need to clean it up. So in this situation what i would do is do  null check then log a warning message and not attempt the code that is accessing it.

The reason for this is that this method does a “clean up” to dispose the object, but something else has already disposed it, there is not hard set rule here, you need to judge thee on a case by case basis.


After waiting again my error logs are a lot more clean now and I can start to see other exception that I couldn’t see before, below you can visibly see the decrease in volume after I fixed this and a few other errors up as well, after 10:15 when my code was deployed i have 1 exception in the next hour, as opposed to hundreds per  minute. My warning logs are a bit more noisy, but that’s ok because I am trying to filter out the noise of the less important errors by using warnings.



Fluent C# for Unit Testing

I’ve been playing with different methods of this for a while now, ever since i got introduced to it by Shea when I was at Oztix.

Note:  in this example I am using the Shouldly library, which i recommend checking out.

I’ve seen a number of different implementations for Unit Testing and mocking specifically, but then all seem a bit complex, and also these days we run NUnit is massive degrees of parallel, and a lot of implementations use static, which causes issues in this instance.

So in introducing my new team to it I revisited it and came up with what i think is a rather simplified version using Moq, which we are using for mocking at Agoda – I miss NSubstitute 😦

When using fluent within Unit Testing the goal is, in my mind, to get your naming using the GivenWhenThen convention from BDD.  I don’t like creating “builder” classes as they dont selfdocument with good context of what you are doing.

So you test’s C# code should read something like this


So if we look at a more detailed example

I am going to create a person class with a happiness level property, and also something else to check, lets say Number of fingers.

public class Person
 public Person()
 //Set Defaults
 HappynessLevel = 5;
 NumberOfFingers = 10;
 public int HappynessLevel{ get; set; }
 public int NumberOfFingers { get; set; }
 public void MakeAngry()
 HappynessLevel = 1;

public void MakeHappy()
 HappynessLevel = 9;

Now I am going to create my test class in my unit test project to assist with Testing and mocking it.

 public class GivenAPerson
 private Person _person;
 public GivenAPerson()
 _person = new Person();

public Person Then()
 return _person;

public GivenAPerson WhenAngry()
 return this;

public GivenAPerson ThenHappinessLevelShouldBeLow()
 return this;
 public GivenAPerson ThenNumberOfFingersShouldBeDefault()
 return this;

You can see that the methods all return “This” or the class itself, this is one of the core concepts in fluent C#. This allow us to achieve the desired outcome of style we are looking for above.

There is one method that doesn’t though, which is “Then”, I sue this in conjunction with shouldly, and I’ll get into why at the end.

So now lets look at putting it all together in the Test Fixture

 public class TestFixtureExample
 public void MyTest()
 new GivenAPerson()

The first thing you will notice is that i use the word “new”, in the past I have used static to avoid this, but when using massive parallelism in NUnit you want to do this so you avoid static.

You will also note that I have added 2 checks to this test, this is to give an example of how you would use this to scale the framework.

Using this method you should be able to easily reuse your checks, in this example we verify that the persons number of fingers doesn’t change when they are angry, which is very odd, but it’s just to give an example how you can apply multiple Then or When to a single test.

Now what about the “Then()” method. I normally don’t write a method like what is in ThenNumberOfFingersShouldBeDefault(), because it is just a single value. Normally i use ThenBlahBlah() methods for complex checks, if the check is not complex i would write something like the following

 public void MyTest2()
 new GivenAPerson()

This allows me a quick check for a single value at the end if I need to and avoids me writing methods.



Dependency Injection Recomendations

Recently started working with a project that has a class called “UnityConfiguration” with 2000 lines of this

container.RegisterType<ISearchProvider, SearchProvider>();

This fast becomes unmanageable, wait, I hear you say, not all Types are registered in the same way! Yes, and you won’t get away with a single line to wire-up your whole IoC container, but you should be able to get it way under 50 lines of code, even in big projects.

I prefer to go a bit Hungarian and file things via folders/namespaces by their types, then use the IoC framework to load in dependencies using this. This is because based on the Type is generally where you find the differences.

For example I put all my Attributes under a namespace called “Attributes”, with sub-folders if there is too many of course, as so on.

Below is an example of a WebApi application i have worked on in the past. This method is called assembly scanning and is in the Autofac doco here

var containerBuilder = new ContainerBuilder();

.Where(t => t.IsInNamespace("Company.Project.WebAPI.Lib")).AsImplementedInterfaces();
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Attributes")).PropertiesAutowired();
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Filters")).PropertiesAutowired();

_container = containerBuilder.Build();

You can see form the above code that things like the Attributes and filters require the Properties AutoWired as I use Property injection as opposed to the constructor injection, as these require a parameter-less constructor. So I end up with one line for each sub-folder in my project basically.

So as long as I keep my filing system correct I don’t have to worry about maintaining a giant “Configuration” class for my IoC container.

You can also make use of modules in Autofac by implementing the Module, I recommend using this for libraries external to your project that you want to load in. you can use the RegisterAssemblyModules method in Autofac in a similar way






GitHub Pull Request Merge Ref and TeamCity CI fail

GitHub has an awesome feature that allows us to build on the potential merge result of a pull request.

This allows us to run unit and UI tests against the result of a merge, so we know with certainty that it works, before we merge the code.

To get this working with TeamCity is a pain in the ass though.

Lets look at a basic workflow with this:

First we will look at two active pull request, and we are about to merge


Pull request 2 advertises the /head (actual branch) and /merge (result “if we merged”)

TeamCity say you should tie your builds to the /merge for CI, this will build the merge result, and I agree.

However lets look at what happens in GitHub when we merge in Feature 1.


The new code goes into master, which will recalculate the merge result on Pull request 2. TeamCity correctly builds the merge reference and validates that the Pull Request will succeed.

However if we look in GitHub we will see the below


It now blocks you and prompts you to updates your branch.

After you click this, the /head and /merge refs will update, as it adds a commit to your branch and recalculates the merge result again, then you need wait for another build to validate the new commit on your branch.


This now triggers a second build.And when it completes you can merge.

The issues here is we are double building. There is two solutions as I see it,

  1. GitHub should allow you top merge without updating your branch
  2. TeamCity should allow you to trigger from one ref and build on a different one

I was able to implement the second result using a build configuration that calls the TeamCity API to trigger a build. However my preference would be number 1 as this is more automated.


Inside it looks like this


Below is example powershell that is used in the trigger build, we had an issue with the SSL cert (even though it wasn’t self signed) so had to disable the check for it to work.

add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$buildBranch =""
Write-Host $buildBranch
$buildBranch = $buildBranch.Replace("/head","/merge")
$postbody = "<build branchName='$buildBranch'>
<buildType id='%TargetBuildType%'/>
Write-Host $postbody
$user = '%TeamCityUser%'
$pass = '%TeamCityPassword%'

$secpasswd = ConvertTo-SecureString $pass -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($user, $secpasswd)

Invoke-RestMethod https://teamcity/httpAuth/app/rest/buildQueue -Method POST -Body $postbody -Credential $credential -Headers @{"accept"="application/xml";"Content-Type"="application/xml"}

You will see that we replace the branch name head with merge, so we trigger after someone clicks the update branch button only.

Also don’t forget to add a VCS trigger for file changes “+:.”, so that it will only run builds when there are changes.


We are running with this solution this week and I am going to put a request into GitHub support about option 1.

This is a really big issues for us as we have 30-40 open pull requests on our repo, so double building creates a LOT of traffic.

If anyone has a better solution please leave some comments.





TeamCity and avoiding redownloading of npm packages

I haven’t been able to find a better solution that this. My original post on stackoverflow here.

TeamCity clears out the node_modules folder on every build, so when running npm install it re-downloads them all. So I’ve moved to using a powershell step to backup and restore this folder and the end and start of each build respectively.

Here is the PS step i run at the start

param (
[string]$projId ,

if(Test-Path "c:\node_backup\$projId\node_modules")
Move-Item "c:\node_backup\$projId\node_modules" "$WorkDir\node_modules"

And here is the one at the end

param (
[string]$projId ,
If (!(Test-Path "c:\node_backup"))
mkdir "c:\node_backup"
If (!(Test-Path c:\node_backup\$projId))
mkdir "c:\node_backup\$projId"
If (Test-Path "c:\node_backup\$projId\node_modules")
Remove-Item "c:\node_backup\$projId\node_modules"

Move-Item "$WorkDir\node_modules" "c:\node_backup\$projId"

Then in the script arguments i pass this

-projId -WorkDir %NodeWorkingDirectory%

The NodeWorkingDir is a parameter I set in my projects that i use to tell node where my gulp file is.


Running up TeamCity build Agents on HyperV

I had to run up a bucket load of build servers on a HyperV environment the other day so decide to automate the process a little.

I have also started using RAM disks for the agents too, to speed them up.

Step for the build server load were:

  1. Install OS (Win Server 2012 R2)
  2. Install SQL 2012 Express
  3. Install VS 2015
  4. Install build agent and point it at the TC Build agent on Drive E:
  5. Install RAMDisk (here)
  6. Run windows updates

If you are unsure where to get the Build Agent installer from click the agents tab in TeamCity and there is a link in the top left.


At this point move the build agent to the RAM disk by:

  1. Stop Team City Build agent service
  2. Change the drive letter of E: to F:
  3. create a RAM drive as E: with image file on F and save to disk on shutdown
    1. I used 5Gb drive and it handles ok
  4. Copy the build agent folder from F to E
  5. Start Build agent service

After this i also installed IIS and disabled the firewall

Lastly sysprep the server and check OOBE/Shutdown

Then delete the VM I was using and the VHDX file  that was generated keep for using as a template, I backed it up to use on other hosts as well in future.

Then wrote the below script that run’s on the host, it does the following for me:

  1. Creates a new VHDX using a diff based on the original (makes run up really fast)
  2. Creates a VM that uses that VHDX
  3. Sets some values that I can set in the New-VM Command
  4. Changes the network config of the guest


Function Set-VMNetworkConfiguration {
Param (



[String[]]$DefaultGateway = @(),

[String[]]$DNSServer = @(),


$VM = Get-WmiObject -Namespace 'root\virtualization\v2' -Class 'Msvm_ComputerSystem' | Where-Object { $_.ElementName -eq $NetworkAdapter.VMName }
$VMSettings = $vm.GetRelated('Msvm_VirtualSystemSettingData') | Where-Object { $_.VirtualSystemType -eq 'Microsoft:Hyper-V:System:Realized' }
$VMNetAdapters = $VMSettings.GetRelated('Msvm_SyntheticEthernetPortSettingData')

$NetworkSettings = @()
foreach ($NetAdapter in $VMNetAdapters) {
if ($NetAdapter.Address -eq $NetworkAdapter.MacAddress) {
$NetworkSettings = $NetworkSettings + $NetAdapter.GetRelated("Msvm_GuestNetworkAdapterConfiguration")

$NetworkSettings[0].IPAddresses = $IPAddress
$NetworkSettings[0].Subnets = $Subnet
$NetworkSettings[0].DefaultGateways = $DefaultGateway
$NetworkSettings[0].DNSServers = $DNSServer
$NetworkSettings[0].ProtocolIFType = 4096

if ($dhcp) {
$NetworkSettings[0].DHCPEnabled = $true
} else {
$NetworkSettings[0].DHCPEnabled = $false

$Service = Get-WmiObject -Class "Msvm_VirtualSystemManagementService" -Namespace "root\virtualization\v2"
$setIP = $Service.SetGuestNetworkAdapterConfiguration($VM, $NetworkSettings[0].GetText(1))

if ($setip.ReturnValue -eq 4096) {

while ($job.JobState -eq 3 -or $job.JobState -eq 4) {
start-sleep 1

if ($job.JobState -eq 7) {
write-host "Success"
else {
} elseif($setip.ReturnValue -eq 0) {
Write-Host "Success"

Write-Host $MachineName

New-VHD –Path “D:\Hyper-V\Diff.$MachineName.vhdx” –ParentPath “D:\Hyper-V\agent_template2.VHDX” –Differencing

New-VM -Name $MachineName -MemoryStartupBytes 6024000000 -Generation 2 -BootDevice VHD -VHDPath “D:\Hyper-V\Diff.$MachineName.vhdx” -SwitchName "10.0.0.xx"

Set-VM -Name $MachineName -DynamicMemory -ProcessorCount 8
Write-Host $IPv4Address

Get-VMNetworkAdapter -VMName $MachineName -Name "Network Adapter" | Set-VMNetworkConfiguration -IPAddress $IPv4Address -Subnet -DNSServer -DefaultGateway

The above command takes a parameter of the new machine name and the IP you want to give the server, I have hard codded the subnet, gateway and DNS, but these should probably be made parameters too, depending on your environment.

After this i just have to login to the agents and domain join them after they are spun up. I used to use VMM that would domain join out-of-the-box, but it looked painful to do from a script on the host so have left it.

An also authorize them on the TeamCity server.


Connecting TeamCity to GitLab with a self-signed SSL

So I spent hours today beating my head against a wall and cursing JRE, so a pretty normally day for me.

I had to connect our TeamCity server to the GitLab server, the GitLab server uses a SSL cert that was generate from the AD Domain CA, so is trusted by all the domain machines. Our TC server is on the domain as well and when connecting to the https site it comes up as green.

However when connecting to git through TeamCity it is running inside JRE which for some reason doesn’t use the machine trusts, it has it’s own cert store you need to add the cert too.

Here’s the error i was facing:


List remote refs failed: PKIX path building failed: unable to find valid certification path to requested target


To test the trust from JRE you need to run this

java SSLPoke git.mycompany.local 443

Where git.mycompany.local is your gitlab server

You can get the sslpoke class here

if its untrusted you will see an error here.

You can use your web browser to export the public key.


Most docs tell me that you can export your root CA public cert, but this didn’t work for me, I actually had to export the specific cert for this site.

Then use this command line to import the cert into JRE and restart TeamCity.

C:\TeamCity\jre\bin>C:\TeamCity\jre\bin\keytool.exe -importcert -trustcacerts -file C:\MyGitLabSSLCert.cer -alias MyGitLabSSLCert -keystore “C:\TeamCity\jre\lib\security\cacerts”

After this we are in business!