Fast, Automated Builds For Unity

We’ve recently been putting work into quick automated builds for our Unity projects at A Stranger Gravity. Our builds, which run Unity tests and build a Windows standalone, are now down to 2 minutes, using a combination of Docker, Google Cloud, and Azure Pipelines.

This kind of work involves stumbling through arcane documentation and using services that really don’t want you to have 20 Gb of data flying around for every build. I thought it would be helpful to consolidate information into something like an automated build bible. Hopefully someone in the future will find this helpful as they tackle this process themselves.

chrome_ChBOCqSbsE.png


Table of Contents:

  1. High Level Workflow

  2. Why..

    1. Why Docker?

    2. Why Azure and GCP?

  3. Quirks

    1. Unity Quirks

    2. Docker Quirks

    3. Azure Quirks

The Workflow

At a high level, our workflow is similar to traditional software development. We build within Docker images on Google Cloud instances, with a agents managed by Azure Pipelines.

  1. Builds trigger with Azure Pipelines from Git commits made to master and by Github PR checks.

  2. A Linux-box on Google Cloud Platform performs the build, running the Azure agent.

  3. Builds execute within a custom Docker image, with the Library folder pre-populated.

Our “Quick Check” build runs in 5 minutes, executing tests and making a Windows executable.

Why Docker?

We use docker because it makes it easy, and very quick to boot up a clean environment for our builds. In the past I’ve maintained build servers that maintain state between builds, and inevitably these servers drift into a bad state. Docker gives us the peace of mind that our builds are clean. If your change works on the build server, then it’s a local issue, end of story.

Docker makes it incredibly easy to debug issues. You can simply boot up and exact replica of the build machine on your local computer to debug. You can very quickly boot up a different cloud instance if you need extra build capacity, or you want to try out a more powerful machine.

Lastly, Docker is extremely fast. It uses clever caching to “boot” a new machine environment in just seconds, even if the docker image is many Gb of data. This means we can load a clean machine for every build, essentially for free.

If you’re not familiar with docker, I’d suggest reading a bit about it. It’s not a common tool in the game development sphere, but is incredibly useful.

Why Azure and Google Cloud?

A CI like Azure means that all of our build artifacts and logs are easily accessible on the web. You don’t have to figure out how to run a Jenkins instance securely on the public web — it just works. Azure is not particularly special, but we chose it primarily because of the strong .NET support (Unity) and that it had a reliable backend from a large company. Azure additionally had a plentiful free tier for getting started.

As for the build agents themselves, you can get away with using the hosted agents, but you’ll need self-hosting for any meaningful performance. Azure agents, like other CI services, are limited to roughly 10gb of disk space — nowhere near enough for a Unity project. The machines they run on are also quite weak.

We use a GCP instance setup with a Local SSD to run our builds. It’s a bit pricey. We had a bunch of GCP credits sitting around from an old venture, and it was a good way to make use of them. An easy secondary option is just to set up a build machine at home. Any machine can be a build agent. A cloud instance is nice, however, because:

  1. Server maintenance is real. Things break, machines don’t connect, etc. We’re a remote company and having a reliable instance in the cloud is a big plus.

  2. Easier upgrades. We can swap disks / images / wipe the whole machine in minutes. This keeps us closer to a clean slate for our builds.

  3. Power costs money! A personal server isn’t free — power costs can be quite high if you’re not careful.

We’ve tried different instance types. Unity importing is largely dependent on disk speed and CPU performance, so the more cores you can afford, the better. As a note, the “High CPU” instances on GCP are not actually faster for single-threaded performance. They do offer better multi-threaded performance, which is only useful for parts of the Unity import process and shader compilation. We’ve found that a high-end personal machine will often beat out some of the beefiest cloud instances.

With these systems setup, we execute a fairly simple pipeline, invoking the Unity editor from the command line. There are a lot of gotchas, however with this process:

Unity Build Quirks:

Mono doesn’t support 64bit Android.
Mono doesn’t support building a standalone player on 64 bit ARM. This is just a limitation of Mono, which we use to avoid IL2CPP build times (for quick debug builds).

Cloud Instances don’t have displays.
Unity expects a display by default. You must pass -nographics to Unity to avoid hitting issues on headless servers. Note that in the past, this flag did not work on Linux before 2019.3, and we had to use xvfb-run --auto-servernum --server-args='-screen 0 640x480x24' $UNITY_EXE as a workaround, which runs the editor with a fake display. As of 4/2020, using xvfb is still necessary when running our Unity tests.

Getting Unity to log output to stdout on Linux.
Supposedly, you can pass the argument -logFile with no logfile specified and Unity will output the logfile to stdout. This hasn’t ever worked for us, and we’re not sure why. It may have been deprecated as of 2019.3. There are two solutions. If you’re running the build as root user, you can simply pass -logFile /dev/stdout. AZP agents are not root users, so instead we setup a unix pipe, and run Unity in the background with &: 

mkfifo unity_output
/path/to/Unity -logFile unity_output (.. other arguments) &
cat unity_output   # blocks until the pipe is closed by Unity!

Unity must be able to open the logfile as write/read. On our system, /dev/stdout was only permissioned for write.

Disable the assembly updater.
By default Unity runs the assembly updater, which tries to update out-of-date code and assemblies. This can block a build if it hits something. For a CI service, it’s better to just fail the build and alert the user. Pass the argument -disable-assembly-updater to unity.

Burst does not cross compile.
We run all of our builds in Linux, but the new Burst compiler doesn’t cross compile. Supposedly you can disable burst with the --burst-disable-compilation Unity flag. This has never worked for us. Instead we run an extra script in our C# build script to manually change the burst settings json file, just before building.

Unity hangs on ‘Cleanup Mono’ after build (Unsolved)
Sometimes, Unity will hang at the end of a standalone build. The last entry of the log file is “cleanup mono”. This is after the build has succeeded. For some reason, Unity fails to exit gracefully and just hangs. We’ve been unable to find a graceful way to handle this case. For our standalone builds we simply kill the process after a successful build:

Process.GetCurrentProcess().Kill();

It’s messy, but it gets the job done. Because we’re running within Docker, we don’t have to worry about corrupting the state of the editor.

Docker Quirks

Unity really wasn’t built to run on Docker. There’s no official support. That said, in a Linux docker image, Unity executes fine, once you work around some strange quirks. That said, for Windows release builds you will need a non-docker, Windows instance. Here’s two big issues we’ve faced:

Unity doesn’t run in a Windows Docker container!
Inside a Windows Docker container, Unity.exe simply fails to start, with no output. This is regardless of the the arguments you pass it, activation, etc. It just doesn’t seem like this is supported, although some people claim to have gotten it working. Windows Docker containers are also, unfortunately, a bit of a second-class citizen for Docker in general. Docker is heavily Linux focused.

Windows Release builds can’t run on Linux.
This is because the new Burst compiler can’t cross compile. This means a windows release build must be made on a Windows machine. If Unity functioned within a Windows Docker image, we could run that machine through docker. Unfortunately the combination of these two issues means that we’re stuck with:

  1. Most development and release builds run on Linux in a Docker Image

  2. The Windows release builds are built locally.

In the future we may use a Windows box on GCP to build these release builds, but as all developers have a Windows machine, there’s not a big need.

Azure CI Quirks

Build Agents don’t run as root.
Most CI services run their build agents as root, except Azure. Worse, the default Ubuntu docker image no longer includes sudo as an installed tool. We had to install sudo with a RUN command in our custom Docker image.

The Cache task is slow.
We initially tried caching LFS and the Library folder using Azure Pipelines caching. Unfortunately when these folders are many Gb of data, uploading and downloading those caches became the dominant factor in our builds. Instead we embed a stale version the LFS and Library folders directly into our docker image, and then only download / import the changes for every build.

Set workspace: clean for self-hosted agents
Azure doesn’t set this flag by default on a self-hosted agent. Make sure to set it or your builds won’t be cleaned between runs!


Reach out to me with questions on Twitter.

Technical, ArticleJohn AustinBuild Tools, Unity