Lattice now compiles to .NET IL
Lattice is a high-performance visual scripting system targeting Unity ECS. Read more here.
I’ve tried several times to write blog posts about Lattice, and each time I’ve gotten lost in the weeds. It’s hard to pick a point to start. So instead, I’ve resolved to just start writing — quantity over quality, as they say.
Lattice has met a major milestone this week: programs now fully compile to .NET IL! If you’ve never written a programming language before, this may not mean much to you, but there’s something beautiful about seeing a relatively complex program compile down into flat assembly instructions.
I’m very proud of this work. One of the goals of the Lattice project was to design a visual language that could compile down into fast linear code — effectively just the procedural code that you would write if you were coding it by hand. This is very much inspired by Rust’s tradition of “zero-cost abstractions”. With the new compilation pipeline, the node-graph representation is stripped away entirely. Outputs of nodes become local variables and bodies of nodes become plain static methods.
I’ve talked in the past about a plan for Lattice to generate C#, but IL was a much better option for a few reasons:
Generating C# requires a Domain Reload to compile which is a non-starter if you're editing scripts while the game runs.
Generating C# would require shipping a C# compiler at runtime if you want to modify scripts during standalone play.
C# is a sloppy thing to generate. There are a lot of syntactical concerns you get bogged down in just trying to get something valid.
IL, as it turns out, is trivially easy to emit in-process with Reflection.Emit, and is actually really simple to work with. I use the Sigil library which is a validating wrapper which catches a number of type errors, etc, during generation. Plus, IL can do several things that C# can't like calling private methods, etc, which just makes the whole process smoother.
This is possible because of the IR representation I added to the compiler a month or two back. For example, the following graph is represented under the hood as a larger graph of simple primitives:
The view of the authoring experience in Lattice. This is my Integration Test graph.
The IR representation of the graph. A single authored node becomes several IR nodes.
This IR is critical because it reduces the complexity of the next step of the compilation: code generation. While there may be hundreds of nodes available for user scripts, in the IR there are only 7 distinct types of operators:
Function (a handle to a static C# method)
Previous (allows referencing a node value from the previous frame)
Entity (returns a handle to the current Entity)
QualifierTransform (allows referencing other entities dynamically)
Barrier (execution barrier, waits for all inputs to finish)
Collect (collects several values into an array)
Malformed (a stub node that returns an error. Used for syntax errors)
The IL Generation step only needs to implement generation for these 7 operators, dramatically reducing the complexity. In fact, the IL Generator is only ~500 lines of code. The final code looks something like:
With the new backend, Lattice is now quite fast — as fast as C#! I still need to do comparative profiling, but I’m fairly confident Lattice is by far the fastest visual scripting system in Unity by a long shot. Bolt, NodeCanvas, and Playmaker all interpret their node graphs. Lattice emits a single static method that executes all lattice scripts in the game in a single pass. The .NET and Mono JITs eat pure static methods for breakfast.
This work also enables some interesting next steps for the compiler:
Automatic parallelization & jobification (Unity Job system)
Burst compilation of Lattice Graphs
However, Lattice is now plenty fast for my needs. So for the time being, I’m pivoting back to working on gameplay workflows and UX. Stay tuned for more updates.