It seems that a lot of companies have become a fan of virtualization. I can understand why. Instead of having 5 single machines — and as such 5 sets of hardware — all doing one task, you can have one machine run 5 virtual servers running their one task. It saves space, power consumption, and in the end money.
There’s a problem though. If those servers are chugging along doing their thing and needing to utilize CPU cycles or I/O to hardware, things slow down. A lot. We’ve run into this issue lately since we’ve started doing a lot of parallel development — developing on multiple projects that are releasing at different times on the same product. We have code check-ins happening at all times of the day — we also have an offshore team hard at work — and as such we have continuous replication builds firing off all the time as well. If multiple products or projects are being worked on, then all those builds queue up, which normally isn’t a problem. It’s once we need to move a product to another environment (ie: Integration Testing) that things become evident that there is a problem. Once we start a build which also stages files on a different server (but happens to just be another virtual on the same physical machine) everything creeps to a halt. Coupled with other builds trying to run while large amounts of files are being transfered from one virtual to another things get even slower.
Solution time. To help take the load off the Dev environment server, we thought of getting setting up a new build server on its own physical machine. Makes total sense. But we’d have to go through all the bureaucracy of getting approved hardware, putting in requests, blah blah blah. Developers don’t have time for that kind of thing! So we thought about what we could do, and it struck us! Why not use our own machines? Release management has three members in it, that could be three extra build machines!
So I used myself as a Guinea pig and installed the build service on my machine. I configured it and with a few minor tweaks was able to get builds running on my machine without too much impact on performance. Since then, we’ve planned to take our workstations and dedicate them to one product each. This way we can have 3 CI (Continuous Integration) builds running at a time. We’re still planning on keeping the staging builds local until we get all issues ironed out — because I’m sure there will be issues that will come up.
So far, so good though. Everything seems to be running well, and our Integration Testing push times have been cut in half.
Links to pages i used:
How to: Set up and Install Team Foundation Build (for TFS 2008)
How to: Add the Build Service Account to the Build Services Security Group
How to: Create and Manage Build Agents
Some notes:
- If you have existing builds that use special build properties (ie: use VB6 to compile an executable) make sure that VB6 is in the same location on all build machines that will be used for such builds.
- If your initial build server has special properties for MSBuild, make sure you copy the MSBuild directory to your build machines. This will help with issues due to certain Imports not being available.