Creating a Docker Container from a Scala/SBT project in TeamCity for Use with Octopus Deploy

I considered creating a series of blog posts about my journey into Scala titled “How much I hate Scala/SBT Part XX”, however i decide not to be that bitter. The language isn’t bad, its just the ecosystem around it sucks, I am more likely to find the source code for something from a google search, rather than a stack overflow post or even documentation.

So here’s where I started, I will assume your Scala project is already Packaging up the project with a TeamCity build and SBT step running compile and your ready to move to a docker container.

So the key things here is the version number for me, I use a custom variable called “Version” that i usually set to something like “1.0.%build.counter%”, for my dot NET projects i use the assembly info patcher and this is then used in the package version, so with your docker containers you can use the tag for the version. Octopus Deploy needs the version on the Tag to work effectively.

If you look internally how TeamCity’s SBT runner runs the comment you will see something like the following:

[11:49:38][Step 2/2] Starting: /usr/java/jdk1.8.0_121/bin/java -Dagent.home.dir=/opt/buildagent -Dagent.name=tt-xxx-2004 -Dagent.ownPort=9090 -Dagent.work.dir=/opt/buildagent/work -Dbuild.number=1.0.12 -Dbuild.vcs.number=411695cf560acb5b7e4b2eb837738660acf0e287 -Dbuild.vcs.number.1=411695cf560acb5b7e4b2eb837738660acf0e287 -Dbuild.vcs.number.Ycs_SupplyApiService_YcsSuppioScala1=411695cf560acb5b7e4b2eb837738660acf0e287 -Djava.io.tmpdir=/opt/buildagent/temp/buildTmp -Dsbt.ivy.home=/opt/buildagent/system/sbt_ivy -Dteamcity.agent.cpuBenchmark=627 -Dteamcity.agent.dotnet.agent_url=http://localhost:9090/RPC2 -Dteamcity.agent.dotnet.build_id=1022735 -Dteamcity.auth.password=******* -Dteamcity.auth.userId=TeamCityBuildId=1022735 -Dteamcity.build.changedFiles.file=/opt/buildagent/temp/buildTmp/changedFiles3407499404619574497.txt -Dteamcity.build.checkoutDir=/opt/buildagent/work/36dd69c049b3f712 -Dteamcity.build.id=1022735 -Dteamcity.build.properties.file=/opt/buildagent/temp/buildTmp/teamcity.build2492188537600221102.properties -Dteamcity.build.tempDir=/opt/buildagent/temp/buildTmp -Dteamcity.build.workingDir=/opt/buildagent/work/36dd69c049b3f712 -Dteamcity.buildConfName=DevelopPushDocker -Dteamcity.buildType.id=FFFGGHHH -Dteamcity.configuration.properties.file=/opt/buildagent/temp/buildTmp/teamcity.config6701741486713268575.properties -Dteamcity.projectName=APIService -Dteamcity.runner.properties.file=/opt/buildagent/temp/buildTmp/teamcity.runner3391739853422434247.properties -Dteamcity.tests.recentlyFailedTests.file=/opt/buildagent/temp/buildTmp/testsToRunFirst5160519002097758075.txt -Dteamcity.version=2017.1 (build 46533) -classpath /opt/buildagent/temp/agentTmp/agent-sbt/bin/sbt-launch.jar:/opt/buildagent/temp/agentTmp/agent-sbt/bin/classes: xsbt.boot.Boot < /opt/buildagent/temp/agentTmp/commands5523308191041557049.file
 I’ve highlighted the one I am after, but you can see that TeamCity is passing a lot of data to the SBT runner, it uses java to run from the command line instead of just running the SBT command itself.
There is something i am missing though, I need to know the branch name, becuase we have a convention that if is not built from the master branch we use a “-branchname” at the end. So to add this in you need to edit your SBT runner step in teamcity and add the below
SBTParametersFromCommandLineTeamCity
From this we can use this variable in our Build.Scala file like so, I also add a value for team that is used later.
val dockerRegistry = "MyprivateDockerReg.company.com"
 val team = "myteam"
val appName = "ycs-supply-api"
 var PROJECT_VERSION = Option(System.getProperty("build.number")).getOrElse("0.0.4")

 val BRANCH = Option(System.getProperty("teamcity.build.branch")).getOrElse("SNAPSHOT")
 if ( BRANCH != "master")
 {
 PROJECT_VERSION = PROJECT_VERSION + "-" + BRANCH.replace("-","").replace("/","")
 }
Now for docker, in you SBT plugins directory make sure you have this line to import the plugin
addSbtPlugin(“se.marcuslonnberg” % “sbt-docker” % “1.4.1”)
Then here is what our Build.sbt looks like
import scala.xml.{Elem, Node}

enablePlugins(JavaAppPackaging)

name := "MyAppBin"

dockerfile in docker := {
val appDir: File = stage.value
val targetDir = s"/opt/$team"

new Dockerfile {
from("java:8-jre-alpine")
maintainer(s"$team")

runRaw(s"mkdir -p $targetDir")
workDir(s"$targetDir")
copy(appDir, targetDir)
expose(80)
env("JAVA_OPTS" -> s"-Dappname=$appName -Dconfig.file=conf/application-qa.conf -Dlog.dir=log/ -Dlogback.configurationFile=conf/logback-test.xml -Xms256m -Xmx256m -server")
entryPoint(s"bin/${name.value}")
}
}

imageNames in docker := Seq(
// SPAPI Sets the latest tag
ImageName(s"$dockerRegistry/$team/$appName:latest"),
// SPAPI Sets a name with a tag that contains the project version
ImageName(s"$dockerRegistry/$team/$appName:${version.value}")
)

mainClass in Compile := Some("Boot")

buildOptions in docker := BuildOptions(
cache = true,
removeIntermediateContainers = BuildOptions.Remove.Always,
pullBaseImage = BuildOptions.Pull.IfMissing
)
This will get us building the container and saving it to the local docker cache. After this we need to push it to our private registry.
There is currently an open issue about SBT-Docker here so I couldn’t get docker login to run from SBT so I created a separate task in TeamCity to handle this.
To do this i want to keep a lot of my settings in the Build.Scala so that the experience on the local will be similar to the build server, but I don’t want to text parse, so what we can do is output some logs for SBT to tell TeamCity what settings to use.
Add these two lines in

println(s"##teamcity[setParameter name='DockerContainerCreated' value='$dockerRegistry/$team/$appName:${version.value}']")
println(s"##teamcity[setParameter name='SbtDockerRegistry' value='$dockerRegistry']")</div>
This Will make SBT output the format that TeamCity Reads to set parameters and allow us to create the next step as a command line step.
TeamCitySetParameterFromSBTCommandLine
Next add these parameters is as empty
EmptyParamterTeamCitySetParameterFromCommandLine
Then we can create a command line step that does the docker login/push
PushContainerToDockerRegistryWithVersionTagFromTeamCity
And we are done! you should see your container in the registry now, and if like us you are using Octopus Deploy you will see the container appear on searches and the version numbers will correct be associated with the containers

Slack Bots – Merge Queue Bot

I recently did a talk about a Slack Bot we created to solve our issue of merging to master. Supaket Wongkampoo helped me out with the Thai translation on this one.

We have over 100 developers working on a single repository, so at any one time we have 20 odd people wanting to merge, and each need to wait for a build to complete in order to merge, then one merged the next person must get those changes and rerun. It quit a common scenario but I haven’t seen any projects doing this with this much frequency.

Slides are available here https://github.com/tech-at-agoda/meetup-2017-04-01

 

Continuous Integration/Continuous Delivery Workshop

I hosted a Workshop on CI and CD on the weekend with the following overview of topics

  • Create Build definitions in TeamCity
    • C# net core 1.1 MVC/Web API Project
    • Typescript/Webpack
    • Unit Test NUnit and Mocha
    • Output Packages for Deployment
    • Update GitHub Pull Request status
  • Create deployments in Octopus
    • Deploy to Cluster (C# backend)
    • Deploy to CDN (React SPA)
    • Send an Alert to Slack

Below are the links:

Videos:

Slides:

https://github.com/tech-at-agoda/meetup-2017-05-13

 

Cleaning up Error logs and Dealing with vague Exceptions like “Object Reference not set to an instance of an Object”

In my experience in most cases when finding an issue with a dot NET app 90% of issues will have a corresponding exception, this exception might not be the root cause but can lead you to the root cause if you follow it, having clean error logs is essential if you want to get to the root cause fast.

The approach I take with getting to the root cause is pretty straight forward and systematic, but I find a lot that developers don’t seem to grasp troubleshoot concepts like “Process of Elimination” very well, which are common more in the Infrastructure and support side of IT, so I wanted to give a small overview for my Devs.

The steps I usually follow are:

  1. Check the line of code from the stack trace
  2. If not enough data, add more and wait for the exception to occur again
  3. If enough data then fix the problem

This is assuming you cant replicate the exception.

If you take this approach you will eventually get to the root cause and fix an exception, the next step after that is apply this to “every” exception and clean up your logs.

I will use an example of a recent issue I fixed.

I recently worked on one of our selenium testing projects, which was having some intermittent issues that we couldn’t trouble shoot and there was nothing in the Error Logs, when this happens my initial reaction is to look for code that is swallowing Exceptions,  and low and behold I find code like the below that I fixed up.

emptycatchexception

I fixed about 5 of these in key areas, now I start getting data into my logs, and I get the age old “NullReferenceException”

nullreferenceexception

Looking at the line of code, i can’t tell much, so we are using Serilog, so its easy just to throw some additional objects into the Log.Error method in order to better understand what was going on at run time. In this example the null reference exception was on the line of code that accesses the driverPool object, so I pass this in like below.

logtheobjectthatiscausingthenullreferenceexception

Now I wait for the exception to occur again and then check logs.

findingtherootcauseofexception

Now i can see the driver pool is a list of null objects. The reason for this will be earlier in the code, but this particular case is filling up the logs so we need to clean it up. So in this situation what i would do is do  null check then log a warning message and not attempt the code that is accessing it.

The reason for this is that this method does a “clean up” to dispose the object, but something else has already disposed it, there is not hard set rule here, you need to judge thee on a case by case basis.

fixingtherootcauseofexception

After waiting again my error logs are a lot more clean now and I can start to see other exception that I couldn’t see before, below you can visibly see the decrease in volume after I fixed this and a few other errors up as well, after 10:15 when my code was deployed i have 1 exception in the next hour, as opposed to hundreds per  minute. My warning logs are a bit more noisy, but that’s ok because I am trying to filter out the noise of the less important errors by using warnings.

errorlogvolume

 

Fluent C# for Unit Testing

I’ve been playing with different methods of this for a while now, ever since i got introduced to it by Shea when I was at Oztix.

Note:  in this example I am using the Shouldly library, which i recommend checking out.

I’ve seen a number of different implementations for Unit Testing and mocking specifically, but then all seem a bit complex, and also these days we run NUnit is massive degrees of parallel, and a lot of implementations use static, which causes issues in this instance.

So in introducing my new team to it I revisited it and came up with what i think is a rather simplified version using Moq, which we are using for mocking at Agoda – I miss NSubstitute 😦

When using fluent within Unit Testing the goal is, in my mind, to get your naming using the GivenWhenThen convention from BDD.  I don’t like creating “builder” classes as they dont selfdocument with good context of what you are doing.

So you test’s C# code should read something like this

GivenAPerson()
.WhenAngry()
.ThenHappinessLevelShouldBeLow();

So if we look at a more detailed example

I am going to create a person class with a happiness level property, and also something else to check, lets say Number of fingers.


public class Person
 {
 public Person()
 {
 //Set Defaults
 HappynessLevel = 5;
 NumberOfFingers = 10;
 }
 public int HappynessLevel{ get; set; }
 public int NumberOfFingers { get; set; }
 public void MakeAngry()
 {
 HappynessLevel = 1;
 }

public void MakeHappy()
 {
 HappynessLevel = 9;
 }
 }

Now I am going to create my test class in my unit test project to assist with Testing and mocking it.


 public class GivenAPerson
 {
 private Person _person;
 public GivenAPerson()
 {
 _person = new Person();
 }

public Person Then()
 {
 return _person;
 }

public GivenAPerson WhenAngry()
 {
 _person.MakeAngry();
 return this;
 }

public GivenAPerson ThenHappinessLevelShouldBeLow()
 {
 _person.HappynessLevel.ShouldBeLessThan(3);
 return this;
 }
 public GivenAPerson ThenNumberOfFingersShouldBeDefault()
 {
 _person.NumberOfFingers.ShouldBe(10);
 return this;
 }
 }

You can see that the methods all return “This” or the class itself, this is one of the core concepts in fluent C#. This allow us to achieve the desired outcome of style we are looking for above.

There is one method that doesn’t though, which is “Then”, I sue this in conjunction with shouldly, and I’ll get into why at the end.

So now lets look at putting it all together in the Test Fixture


 [TestFixture]
 public class TestFixtureExample
 {
 [Test]
 public void MyTest()
 {
 new GivenAPerson()
 .WhenAngry()
 .ThenHappinessLevelShouldBeLow()
 .ThenNumberOfFingersShouldBeDefault();
 }
 }

The first thing you will notice is that i use the word “new”, in the past I have used static to avoid this, but when using massive parallelism in NUnit you want to do this so you avoid static.

You will also note that I have added 2 checks to this test, this is to give an example of how you would use this to scale the framework.

Using this method you should be able to easily reuse your checks, in this example we verify that the persons number of fingers doesn’t change when they are angry, which is very odd, but it’s just to give an example how you can apply multiple Then or When to a single test.

Now what about the “Then()” method. I normally don’t write a method like what is in ThenNumberOfFingersShouldBeDefault(), because it is just a single value. Normally i use ThenBlahBlah() methods for complex checks, if the check is not complex i would write something like the following


 [Test]
 public void MyTest2()
 {
 new GivenAPerson()
 .WhenAngry()
 .ThenHappinessLevelShouldBeLow()
 .Then().NumberOfFingers.ShouldBe(10);
 }

This allows me a quick check for a single value at the end if I need to and avoids me writing methods.

 

 

Dependency Injection Recomendations

Recently started working with a project that has a class called “UnityConfiguration” with 2000 lines of this

container.RegisterType<ISearchProvider, SearchProvider>();

This fast becomes unmanageable, wait, I hear you say, not all Types are registered in the same way! Yes, and you won’t get away with a single line to wire-up your whole IoC container, but you should be able to get it way under 50 lines of code, even in big projects.

I prefer to go a bit Hungarian and file things via folders/namespaces by their types, then use the IoC framework to load in dependencies using this. This is because based on the Type is generally where you find the differences.

For example I put all my Attributes under a namespace called “Attributes”, with sub-folders if there is too many of course, as so on.

Below is an example of a WebApi application i have worked on in the past. This method is called assembly scanning and is in the Autofac doco here

var containerBuilder = new ContainerBuilder();
containerBuilder.RegisterAssemblyModules(
typeof(WebApiApplication).Assembly);

containerBuilder.RegisterAssemblyTypes(typeof(WebApiApplication).Assembly)
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Lib")).AsImplementedInterfaces();
containerBuilder.RegisterAssemblyTypes(typeof(WebApiApplication).Assembly)
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Attributes")).PropertiesAutowired();
containerBuilder.RegisterAssemblyTypes(typeof(WebApiApplication).Assembly)
.Where(t => t.IsInNamespace("Company.Project.WebAPI.Filters")).PropertiesAutowired();

containerBuilder.RegisterApiControllers(Assembly.GetExecutingAssembly()).PropertiesAutowired();
_container = containerBuilder.Build();

You can see form the above code that things like the Attributes and filters require the Properties AutoWired as I use Property injection as opposed to the constructor injection, as these require a parameter-less constructor. So I end up with one line for each sub-folder in my project basically.

So as long as I keep my filing system correct I don’t have to worry about maintaining a giant “Configuration” class for my IoC container.

You can also make use of modules in Autofac by implementing the Module, I recommend using this for libraries external to your project that you want to load in. you can use the RegisterAssemblyModules method in Autofac in a similar way