All right, hello everyone. Thanks for coming. My name is David Branstill and this is
a talk on why our Android builds so damn slow. So just a little bit about me. I spent
about 10 years working at Google on the Android operating system. I was an engineer on the Android
runtime team working on optimizations in the Java stack and Android security else
the technical lead for the hypervisor and virtualization framework projects. And now it's
sort of the deaf. We work with device manufacturers who use Android in their product and
we try to optimize their workflows so they're more efficient. And the number one complaint
we keep hearing from the ecosystem is Android builds and checkouts are just crazy crazy slow.
Just to give you some context, I mean you just heard from Stefan how huge the Android code
base is checking it out, downloading the source code to your computer easily takes between
15 and 20 minutes. Most of this is just because it's so damn huge. But the other factors
that the AOSP mirror can be quite slow and a good prototype is to use the authenticated access
because it's going to get around some of the usage quota. Once you have the source code you
want to build it that typically takes around two hours or roughly four and a half dollars
depending on where you get your machines from. And again this comes down to the fact that
it's so damn huge. There's about 250,000 build steps that the machine needs to go through.
And obviously if you spend more in your hardware it's going to go faster but the costs are also
going to go up. So there's a trade-off that you have to balance. And then as you develop you're
going to go through incremental builds and those can take 10 minutes, 30 minutes, it all depends
on what you've changed and how much of the code base needs to be rebuilt at that point.
But there's also another problem with Android and that's the fact that it doesn't have a good
enforcement of the dependency declarations. So sometimes you end up changing something and it doesn't
get correctly rebuilt and it uses an older version of the build artifact. So at that point most
engineers just wipe the build directory and start from scratch and it's another two hours to get
back to a working state. I'm going to be quoting some numbers here. I'm already M. Just to make it
reproducible. I picked a virtual machine on Google Cloud 32V CPU 120 gigs of RAM and VMSSD
costs about $2 an hour and all of the numbers are for the latest Android 16 QPR 2 release and it's
building the AOSP cutoffish ARM 64 phone target. So if you want to try it, sorry.
Now, okay I'll speak louder. So if you want to reproduce these numbers you should be able to buy
by creating a similar environment. So historically the answer to how are we going to fix this has
that Android will eventually move to Bazel. Bazel's the build system that's used in the
Google Monorequo. It has been open sourced. It has all these great properties of
permeticity, reproducibility. It has build and test caching. It scales really well.
So the original plan from AOSP has been to gradually migrate from the old make files to Bazel
in a bunch of steps. Step one, develop a build system. It's now called soon, which uses the Bazel
file format, but it's backwards compatible with make. Step two, gradually convert all your make files
to these new BP files. You saw that in Stefan's chart as well. And then start gradually building
some parts of the system with Bazel and gradually phase out soon and the original make.
Step four, profit. I get all the benefits from Bazel and the associated infrastructure.
The effort culminated around Android 14 early 2023 when most of the AOSP codebase has been migrated
to soon. Give or take some device config files. But the adoption downstream has remained
a lot lower, especially in BSPs. You still see a lot of make files.
But in early 2023, we started seeing Bazel builds on CI. Android.com. So it's some parts of the system
started to be built with Bazel. They're all looked great. And then in typical Google fashion,
we got a notification that Bazel isn't a supported build system anymore. The migration has been
halted in late 2023. And outside of building the Android kernel, soon is the main build system
from now on. So that kind of reason isn't a bind. We're in a state where the migration hasn't
been completed. We have soon, but soon isn't Bazel. It doesn't have those great properties of
hermeticity and reproducibility. It's sort of tries. There are some mechanisms to sanitize the
path force. Certain tool change to be used. There's a sandbox for gen rules. But under the hood,
it's still compatible with make. And you can do anything with a make file. We can bypass all these
great sandboxing mechanisms. So the reason why Bazel builds are fast is that it builds on these
properties and uses them in a really clever way. But because soon doesn't have those properties,
you can't really do that. So in Bazel, all inputs and outputs are declared upfront. You don't have
that with soon. It's easy to send box tasks in force, dependencies, capture results. You can't do that
with soon. It's easy to upload computation to another machine. If you have a build farm,
special in a large organization that can be really helpful, you can't do that with soon. So the
question is, where in this state, what can we actually do? How much of these sorts of features can
we get with the build system that Android currently uses? And we'll probably use for the foreseeable
future. I'm also going to reference a talk from Chris Simmons at embedded LSS 2020-3 where he
deep dives into how soon and all the other components work. I'm not going to go through that here.
I'm just going to tell you the important bits. But if you're interested, go and check out that video.
But in short, the way that the build system works is that we have two kinds of build files,
Android BP, that those are the ones that are very similar to Bazel's StarLark language.
And these are just templates and all the build logic actually lives in Go. And then we have the
legacy Android MK files that are still used in a bunch of places. And the way that the build works
is that you have four main components. SumUI is the top level process. It drives the whole thing.
It also renders the progress bar. And then you have sum underscore build, which takes the blueprint
files and compiles them down to Ninja and catchy, which does the same thing for make files.
And that's how the two systems can coexist. They both compile down to the same low level format
and they can reference each other if necessary. And this process, if you've ever built Android,
is really annoying because depending on how fast your SSD is, this can take anywhere between two
and seven minutes. And every time you touch one of these files, you have to go through the process again.
All right, the reason why it's taking so long is because it loads everything into RAM. There's a lot
of analysis. It uses about 40 gigs of RAM and then writes about seven gigabytes of data on disk.
All right, so if you're changing the config files, the build files often, you're going to be running
into this over and over. But once this is done, you get a set of the build.binger files and you
execute Ninja and then just goes through roughly 250,000 individual build commands and executes them
with as much parallel as it gets. So that's the bulk of the of the build.
And if you're ever trying to debug while your builds are slow, one indispensable tool is looking
at the build traces. This is a file that's generated during the build. It's in out slash build.trace.gz.
And you can take this file and open it in any perf viewer. The one I like is Profito.dev.
You can just upload the file there and it gives you this beautiful UI where you can see everything
that's happening. And we can see those stages here. So this red one that's building the blueprint
files. This is building make files that on this machine took two and a half minutes and the rest
is Ninja compiling all the little bit and pieces that you actually need for the for the system.
And before we dive into caching, just some quick tips for how to make it run faster.
So obviously, nobody has unlimited budgets. We have to make some trade-offs. So what's a good trade
of in terms of hardware? For the CPU, more cores is always better. You can get so much
parallelism out of that main Ninja step that if you can get more cores and get more cores.
2432 is a good baseline, but 6428 is just going to make it go so much faster.
More recent architectures are usually better, like they're always better, but it's not as much of a
win as just raw parallelism. Ram, you want about two gigabytes per logical core. So if you have
32 cores, you want 64 gigs of RAM, but that's typically the minimum. There are some spikes,
so you do want to have a swap roughly the same size just to just be able to absorb those spikes
every now and then otherwise the next might decide to kill your kill your build. But the most
important one is storage. Low latency is the most important aspect in a build. It doesn't matter
how fast is the transfer speed latency is the king and the reason is because the working directory
has about four million files and the median size is about 300 bytes. So these are millions and
millions of teeny tiny files and you just need to be able to create them on the file system as quickly
as you can. It's the same with the soon underscore build phase. It keeps scanning the code base
and it's about one million paths, but only reads a handful of those files. So again, just how quickly
your SSD can respond is the most important aspect here. So back to caching and recapping some of the
features that Basil has but soon doesn't. So if you just want a local cache, there's the good old
ccache, everybody heard of that and Android has supported pretty much since the very beginning.
At some point there was a pre-built binary in the code base that's now gone but you can still bring
your own. If you just install ccache in Ubuntu and you point Android at that binary it's going to
work just fine. The way you set it up is you need these three environment variables,
the top of the terminal, uccache equals one just enables it. Cache, ccache dare is the place where you
store the data and ccache exact is the binary itself of ccache and then ccache dash capital m
sets the maximum size of the cache. 20 gigs is enough for aosp16 and then you just run m to build
Android and at the end it's going to tell you how long it took. If you remember in the beginning
I said the baseline was two hours, it's about our 57, in this case ccache fully populated
finished in 47 minutes. So we spent 20 gigs of local storage and we get 58% improvement in
build times and if we print the statistics from ccache it will tell us that it had a cache hit
on 126,000 targets out of a total of 244,000 so it's pretty much exactly one half of
the whole belt is cnc++ that can be cached with ccache. So this is a really low effort
thing that you can do and it's going to make a huge difference. Sorry, no.
But the state of the art in aosp in terms of build caching is a component called RE client.
This is similar to ccache in the sense that it's a wrapper around a compiler but in this case
it connects to bezels RE infrastructure and compared to ccache which is only for cnc++, RE client
has support for a lot more different kinds of compilers and tools. So there's client, there's
a linker, javasy, meta-lava, RE, D8, sine APK and a bunch of smaller, smaller tools and our
client can take those and and make them cacheable. So for this you need an RE backend,
backend has two kinds of services. The first one is that it's a distributed cache so clients can
upload results of their computation and then either they themselves later or somebody else can
download the same result later on and in a larger organization there's also the support for remote
executions. So you might have a build farm and you can submit those individual jobs to the build farm
and get the results back. There are a lot of different backend solutions. This is a standard protocol.
The open source ones all have very similar names, build barn, build farm and build grid.
They all are under Apache 2.0 and if you go to the official bezel website there's a list of
other ones and commercial ones and so on. So how does RE client work? It has this command called RE
wrapper and you prefix your your commands with that. Android would do it for you and RE wrapper
its job is to create this data structure called action. It's part of the standard of the protocol
and that describes what is about to be executed. But here we hit the difference between bezel and
soon because in bezel you have all the information available upfront. It's all statically declared
and you can find it immediately whereas RE wrapper needs to do a bit of work to fill in the gap.
So for the command and environment that's statically available if it parses the command line that's
being executed it can figure out which compiles being used with our domain source files.
So it can populate some information about the inputs. But for example to figure out which
header files will be included by the compiler laid on it needs to run the preprocessor.
So it's a bit of work to get to that information and finally those input files need a hash.
So it needs to hash those files before it can port can talk to the back end.
Once it has the action struct it sends it to the back end the back end depending on how it's
one figure it will either execute there or look in the cache and it gives back another data structure
code action result. That's the same thing but it describes the outputs of that particular action
and it's just an exchange you know you do this a few hundred thousand times a build and that's how
that's how the caching works. So to set this up on your own machine
uh first you need a RBE back end I'm not going to go through that here.
Depends on that particular project and it's quite well documented but on the client site and
Android you need to create a config file. There's an example in build soon docs RBE the JSON really
well hidden and you need to populate things like the IP address credentials then you to select
the tool chains that you want to use this for and unfortunately I haven't been able to find
one place where they would just have a list but if you go to code search cs.android.com
and you just search for RBE underscore you'll find all the strings and finally you need to select
the exec cution strategy so that's what happens on a cache miss and there are three options
remote local fallback so execute remotely for back to local if it fails local that's the one I'm
using in the example which is no remote execution just treat the cache as a distributed cache nothing
else and finally my favorite racing which means you you tell the remote server to execute but
you also do it locally and whoever gets their first that result gets used and once you have this
config file then you just need to again set some environment variables to point and draw it at that
config file so the way this looks is the two variables at the top the first one is the directory
the other one is the name of the file minus the tot JSON for some reason and with that you can run
m first time in the top of the cache second time it's going to be fully populated
and prints the stats at the end so from this we can see that there were 156,900 local executions so
that's the number of times that the wrapper was invoked and of those 144,700 were cache hits
and that's out of 247,000 targets it for some reason adds 3,000 targets so this is 64% coverage
of the build and the build completed in 30 minutes 25 seconds which is 73% faster compared to the baseline
and this is a clear improvement over C cache but to me it's actually quite disappointing because we've
gone through all this work of you know a standard protocol distributed cache support for 10 different
two chains and all we get out of it is another 14% of coverage and another 15% speed up so it kind of
shows you that there's a limit to how much you can do with compiler wrapping and this general approach
of retrofitting the RBE protocol to AOSP and but this is the state of the art this is the best
that you can do with with AOSP today so if that what's next what can we actually do better here
well ideally we'll get better coverage less local pre-processing but that's that's tricky
because this is a heterogeneous code base every project is a little different it has a million
different programming languages this is probably not unique to Android the Octo is very similar in
this in this regard so it's some sort of generic mechanism for isolating and monitoring those
individual build commands so that we can we can move them around cache them find the
inputs and outputs it will also nice to speed up the ninja file generation like we said it's
somewhere between two and seven minutes you run into it every now and then ideally the algorithm
itself would be faster and anecdotally I I feel like in the past couple of releases it's got a
little better but it's also quite you know ramintensive and I'll have a so there's probably a limit
to how much how much you can do there the other option here is extending caching to this early stage
to cache the result of soon build and catties so that if you're an organization somebody else has
been through this or if you're switching between branches and you have a local cache you don't have
to go through this every single time and you will have noticed that we haven't really talked about
checkouts that's that's also still a problem especially in CI where you don't have that persistent
state or if you're an engineer who keeps you know switching between Android branches because you
need to cherry pick fixes into different releases and so on in that set up you keep going through
that checkout over and over and it's also quite painful but again going back to Stefan's talk
actually about 50% of the disk usage is coming from pre-builds and pre-builds have all sorts of
like versions of client and they're built for Linux and Darwin and really what you need during the
build is one version for one architecture so a lot of that time during the checkout is actually
completely wasted and even in the source code you don't always rebuild everything you don't need
all the definitions for all the devices you don't need cts when you're just building the the
operating system itself so there's there's a lot that we're downloading for no good reason
there actually used to be a solution for this githvfs is a Microsoft project for a virtual
file system forget which would allow us to do you know on demand downloading of of these files
but unfortunately that project has been deprecated so that's also so kind of stuck in that sense
and that's that's the end of my talk um these are the questions that we're asking ourselves
in search of the death and we were having an announcement but if you want to see a demo of how we
do check out in 30 seconds and built in five minutes then we have a we have a video for you
and yeah if you if you're interested in what we're doing
we're hiring it doesn't evil and yeah that's the it's the end thank you