> ## Documentation Index
> Fetch the complete documentation index at: https://site.aspect.build/llms.txt
> Use this file to discover all available pages before exploring further.

# AOSP build benchmark on Aspect Workflows

> Measured results for a full AOSP android-14.0.0_r22 build on a self-hosted Aspect Enterprise deployment, with complete infrastructure configuration and links to the public CI runs.

This page reports measured results for a full Android Open Source Project (AOSP) build on
Aspect Workflows remote execution, the infrastructure it ran on, and links to the public CI
runs so every number can be checked against its source.

## What was built, and how

* **The build:** AOSP `android-14.0.0_r22`, target `aosp_arm64-userdebug`, 176,702 ninja
  actions. AOSP builds with its own build system, Soong and ninja, not Bazel.
* **How actions reach remote execution:** through reclient, AOSP's remote execution client,
  which sends C++ compile and link, Java and packaging actions to the deployment's remote
  execution fleet over the standard Remote Execution API. Every other action runs on the CI
  machine.
* **Where it ran:** a GitHub Actions workflow on a Workflows runner, in a
  self-hosted [Aspect Enterprise](/docs/aspect-workflows/enterprise/overview) deployment in AWS (`us-east-2`). Each run syncs the source with `repo sync`,
  runs Soong analysis, then runs ninja.
* **The build definition:** the script, the reclient configuration and the workflow are in
  [aspect-build/basic-aosp-build](https://github.com/aspect-build/basic-aosp-build).

## Summary

Four configurations, from the worst case (a fresh machine with nothing warm) to the steady
state (no changes):

|                  | Configuration                                            | Build time  |
| ---------------- | -------------------------------------------------------- | ----------- |
| Worst case       | Fresh runner: cold source sync + full local build        | **1h 16m**  |
| Baseline         | Warm runner, full uncached build, single 96-vCPU machine | **34m 07s** |
| Remote execution | Warm runner, full uncached build, 512-slot fleet         | **24m 38s** |
| Steady state     | Warm runner, no changes                                  | **2m 38s**  |

* **Full, fully-uncached build: 24m 38s** with remote execution, versus **34m 07s** on a
  96-vCPU machine alone: **81,463 of 81,481 actions executed on the remote fleet, zero
  failures**. On the part of the build that can be parallelized, remote execution is
  **3.9x faster** (14m 12s down to 3m 36s); the end-to-end ratio is diluted by AOSP's
  serial packaging tail, which no platform can parallelize.
* **Incremental build with no changes: 2m 38s**, most of which is `repo sync`. Warm,
  persistent runners keep the source tree between builds: the first-ever run on a fresh
  machine paid **39m 21s** of `repo sync`; every run since has paid under two minutes.
* **Larger builds:** [Extrapolating to larger AOSP builds](#extrapolating-to-larger-aosp-builds)
  models how these results scale to a production-sized build. Those figures are modeled, not
  measured.

The deployment measured here runs entirely inside one AWS account; fleet size, per-action
memory ceilings, worker pools and the container image are all per-deployment settings.

## Results

The four benchmarks below use the same build on the same infrastructure. What varies is
what was already warm (the source tree, the output directory, the caches) and whether
remote execution was enabled.

| Benchmark                                         | Build step     | Soong timer | Actions executed |
| ------------------------------------------------- | -------------- | ----------- | ---------------- |
| First build: fresh runner, cold sync (worst case) | **1h 16m 06s** | 35:58       | 176,702          |
| Full build, single 96-vCPU machine                | **34m 07s**    | 32:25       | 176,702          |
| Full build, remote execution                      | **24m 38s**    | 22:38       | 176,702          |
| Incremental build, nothing changed                | **2m 38s**     | 01:01       | 576              |

In the remote-execution row, cache reuse was disabled, so every action executed. **81,463 actions ran on the remote
fleet**, 18 fell back to the local machine, and none failed.

The two warm full builds differ in exactly one thing: whether remote execution is on. It
takes the build step from 34m 07s to 24m 38s, a **1.38x** end-to-end improvement. Isolating ninja, it goes from 29m 10s to
19m 24s, a **1.50x** improvement.

Those figures understate what remote execution does to the work it can affect.
Roughly 5 minutes of every run is source sync and Soong analysis, and 15 minutes is a serial
tail that runs on the CI machine in both configurations. **On the parallelizable part of the
build, remote execution is 3.9x faster**: 14m 12s down to 3m 36s.

### Phase-by-phase comparison

The two warm full builds execute an identical action graph, and the last 10% is the same
17,669 actions in each, so the phases compare directly.

| Component                      | Remote execution | Single machine | Ratio             |
| ------------------------------ | ---------------- | -------------- | ----------------- |
| `repo sync` and Soong analysis | 5m 02s           | 4m 47s         | 1.0x (fixed cost) |
| **First 90% of actions**       | **3m 36s**       | **14m 12s**    | **3.9x faster**   |
| Last 10%: the serial tail      | 15m 36s          | 14m 54s        | 0.95x             |
| **Build step total**           | **24m 38s**      | **34m 07s**    | **1.38x**         |

The serial tail is marginally slower under remote execution, because
those actions form a dependency chain in which each one pays a network round trip that no
amount of parallelism recovers. The tail accounts for about 44% of the remote-execution run's
wall-clock and is unaffected by fleet size, cache state, or worker count.

### Execution profile

Within the remote-execution run, the fleet absorbs the parallel bulk of the build almost
immediately:

| Progress | Elapsed from ninja start | Sustained rate    |
| -------- | ------------------------ | ----------------- |
| 10%      | 0.3 min                  | \~1,100 actions/s |
| 30%      | 0.8 min                  | \~1,600 actions/s |
| 50%      | 1.7 min                  | \~490 actions/s   |
| 70%      | 2.5 min                  | \~770 actions/s   |
| 90%      | 3.6 min                  | \~430 actions/s   |
| 100%     | 19.4 min                 | —                 |

**90% of all actions complete in the first 3.6 minutes.** The remaining time is AOSP's
serial tail (R8/dex, APEX hiddenapi encoding, image assembly, and signing), a dependency
chain of large, mostly single-threaded steps that runs on the CI machine.

<Note>
  The serial tail is a property of the AOSP build graph, not of the execution platform. It is
  unaffected by fleet size or cache state, and is present in every configuration, including a
  purely local build. It is the reason a larger fleet doesn't reduce total build time beyond
  a point; see <b>Sizing guidance</b> below.
</Note>

All four benchmarks are public:

| Scenario                               | Public CI run                                                                                |
| -------------------------------------- | -------------------------------------------------------------------------------------------- |
| First build, fresh runner (worst case) | [run 31758099068](https://github.com/aspect-build/basic-aosp-build/actions/runs/31758099068) |
| Full build, remote execution           | [run 31847444276](https://github.com/aspect-build/basic-aosp-build/actions/runs/31847444276) |
| Incremental build                      | [run 31850749685](https://github.com/aspect-build/basic-aosp-build/actions/runs/31850749685) |
| Full build, single machine             | [run 31851206170](https://github.com/aspect-build/basic-aosp-build/actions/runs/31851206170) |

## First build on a fresh runner (worst case)

The cold-start benchmark: a newly-launched runner with no source tree, no output directory,
and nothing warm anywhere. This is what a team's first build looks like, and what every
build would look like without persistent runners.

| Phase                               | Duration       |
| ----------------------------------- | -------------- |
| `repo sync` (from nothing)          | **39m 21s**    |
| Soong analysis and ninja generation | 3m 27s         |
| ninja execution                     | 32m 40s        |
| **Build step total**                | **1h 16m 06s** |

More than half the wall-clock is the cold `repo sync`: the cost runner persistence
removes. Every subsequent run on this page synced the same tree in under two minutes.

## Full build with remote execution

Every action executed remotely, with no action-cache reuse of any kind.

| Phase                                     | Duration    |
| ----------------------------------------- | ----------- |
| Prerequisites                             | 3s          |
| `repo sync` (source tree already present) | 1m 49s      |
| lunch / build configuration               | 8s          |
| Soong analysis and ninja generation       | 3m 13s      |
| ninja execution                           | 19m 24s     |
| **Build step total**                      | **24m 38s** |

### Remote execution outcome

Measured from `rbe_metrics.txt`, uploaded as a workflow artifact on the run:

| Metric                         | Value                     |
| ------------------------------ | ------------------------- |
| Actions dispatched to reclient | 81,481                    |
| Executed remotely              | **81,463** (99.98%)       |
| Local fallback                 | 18 (0.02%)                |
| Failures                       | 0                         |
| Cache hits                     | 0 (disabled for this run) |

Action mix by ninja edge description: approximately 47,600 `clang++`, 18,300 `clang`, 1,100
`javac`, 1,000 `turbine`, plus `jar`, `d8`, `signapk` and `zip` steps.

The 18 local fallbacks are actions that exceeded the configured per-action memory ceiling
(large C++ links) and were retried on the CI machine, a deliberate trade-off described
under *Sizing guidance*.

## Full build on a single machine

The control: an identical source tree, an identical emptied output directory and the same
96-vCPU CI machine, with remote execution disabled. All 176,702 actions ran locally.

| Phase                               | Duration    |
| ----------------------------------- | ----------- |
| `repo sync`                         | 1m 32s      |
| Soong analysis and ninja generation | 3m 15s      |
| ninja execution                     | 29m 10s     |
| **Build step total**                | **34m 07s** |

Sync and analysis cost the same as the remote-execution run, to within a few seconds: they're
unaffected by where actions execute. The difference is entirely in ninja: 29m 10s
locally against 19m 24s with the fleet.

A 96-vCPU machine is already a strong local baseline. The comparison understates
what remote execution offers a team whose CI machines are smaller, since the remote figure
barely depends on the size of the machine driving it; see *Sizing guidance*.

## Incremental build

The tree is fully built and nothing has changed. Soong detects no work and only the
packaging ninja re-runs.

| Phase                                        | Duration   |
| -------------------------------------------- | ---------- |
| Prepare build directory                      | 8s         |
| `repo sync`                                  | 1m 34s     |
| Soong analysis                               | 2s         |
| ninja (`no work to do`, 576 packaging edges) | 57s        |
| **Build step total**                         | **2m 38s** |

No actions reached the remote fleet; there was nothing to execute. That leaves `repo sync`
as 60% of a build that did nothing: in the incremental loop the cost is syncing source, not
compiling.

## Configuration

### CI runner

The machine that drives the build, runs Soong, and executes the serial tail.

| Property      | Value                                    |
| ------------- | ---------------------------------------- |
| Instance type | `c6id.24xlarge`                          |
| vCPU / memory | 96 vCPU / 192 GiB                        |
| Local storage | 2 x 1,425 GB NVMe SSD (instance store)   |
| Root volume   | 2,048 GB EBS                             |
| Network       | 37.5 Gbps                                |
| OS            | Ubuntu 24.04.4 LTS                       |
| CI system     | GitHub Actions, self-hosted runner group |

The AOSP workspace lives on the instance-store NVMe rather than EBS, so source sync, the
output tree and all local I/O are on directly-attached storage.

### Remote execution fleet

| Property                      | Value                          |
| ----------------------------- | ------------------------------ |
| Worker instance type          | `m6id.4xlarge`                 |
| Per worker                    | 16 vCPU / 64 GiB / 950 GB NVMe |
| Workers (maximum)             | 32                             |
| Concurrent actions per worker | 16                             |
| **Total execution slots**     | **512**                        |
| Memory ceiling per action     | 3,584 MiB                      |
| Warm pool                     | 2 pre-initialized workers      |
| Scaling                       | 0 to 32, demand-driven         |

Workers run the container image AOSP's build system requests, so remote actions execute in
the toolchain environment they expect.

### Remote cache and storage tier

| Property      | Value                                        |
| ------------- | -------------------------------------------- |
| Engine        | NVMe storage nodes                           |
| Shards        | 4, mirrored                                  |
| Instance type | `im4gn.large` (2 vCPU / 8 GiB / 937 GB NVMe) |
| Usable cache  | \~3.7 TiB                                    |
| Frontend      | 3–18 tasks, auto-scaled                      |

A full cold AOSP build writes roughly 250–400 GB into the content-addressable store.
Repeated builds deduplicate to nearly nothing.

### AOSP build configuration

| Setting                  | Value                                                                                    |
| ------------------------ | ---------------------------------------------------------------------------------------- |
| Branch                   | `android-14.0.0_r22`                                                                     |
| Target                   | `aosp_arm64-userdebug`                                                                   |
| Platform version         | Android 14 (`UQ1A.240205.002`)                                                           |
| `repo sync` parallelism  | 4 jobs                                                                                   |
| Local ninja parallelism  | 96                                                                                       |
| Remote ninja parallelism | 800                                                                                      |
| ccache                   | Disabled (`USE_CCACHE=0`) on the full-build benchmarks; enabled on the incremental build |

Tools routed to remote execution through reclient: C++ compile, C++ link, javac, turbine,
R8, D8, jar, zip, signapk, ABI dumper, ABI linker and clang-tidy. Metalava runs locally.
Every tool is configured for local fallback, so any action the fleet cannot serve is retried
on the CI machine rather than failing the build.

## Sizing guidance

**Fleet size stops mattering after the parallel phase, for a build this size.** With 90%
of actions complete in the first 3.6 minutes, adding workers shortens that window, not the
serial tail that follows. For the OSS build, 512 slots is already past the point of
diminishing returns and a smaller fleet would produce a similar total time. The economics
invert for larger builds: a production build's parallel phase runs for hours rather than
minutes, keeping a bigger fleet busy long enough to pay for itself; see *Scaling the fleet
beyond 512 slots* under the extrapolation section.

**The CI machine can be much smaller.** The measurements above use a 96-vCPU runner for both
configurations, which flatters the local baseline: it is close to the largest single machine
worth pointing at an AOSP build. With remote execution, that machine's job is to run Soong,
drive the graph and execute the serial tail, none of which needs 96 cores. A substantially
smaller runner should land near the same total time.

**Memory ceiling versus slot count.** Each worker provides 64 GiB across 16 concurrent
actions. A 3.5 GiB per-action ceiling keeps all 16 slots usable, at the cost of a small
number of very large C++ links falling back to the CI machine (18 of 81,481 actions here).
Workloads with many large links can trade slots for headroom: for example, 8 slots at 7 GiB.

**Cold-start behavior.** Scaling a fleet from zero costs several minutes before full
capacity is available, because cloud capacity provisioning ramps progressively. A small
minimum fleet size or a warm pool removes this from the critical path. The runs above were
measured with the fleet already warm.

**Warm, persistent runners remove the largest fixed cost.** The very first build on this
deployment ran on a fresh machine and paid **39m 21s** of `repo sync`
([run 31758099068](https://github.com/aspect-build/basic-aosp-build/actions/runs/31758099068));
cold syncs against AOSP's servers range 35–50 minutes with rate limiting. Aspect Workflows
runners persist between builds (configurable idle timeout; 60 minutes here), so every
benchmark run on this page paid 1m 32s to 1m 49s instead: the source tree survives on the
runner's local NVMe and syncs incrementally. For the incremental loop, where `repo sync` is
60% of a 2m 38s build, runner persistence is the optimization that pays off most.

## Extrapolating to larger AOSP builds

**The figures in this section are modeled, not measured.** They apply the phase profile
measured above to a larger build.

The open-source AOSP build measured above is small enough that the two components a fleet can't accelerate,
source sync plus Soong analysis (\~5 min) and the serial packaging tail (\~15 min), account
for **58% of the remote run's wall-clock**. Production AOSP-derived builds (vendor HALs,
platform apps, product-specific stacks) are commonly three to five times the compile volume,
and nearly all of that added volume is parallelizable compile and link work. The serial tail
does not grow with it: a product build still assembles one set of images, and its
packaging-phase action count is similar to the OSS build's, growing only modestly with the
number of APKs.

That inverts the proportions. Modeling a production build that takes **\~4 hours** on the
same class of CI hardware where the OSS build takes \~1 hour, using the phase behavior
measured above:

| Phase                        | OSS build (\~1 h local) | Production build (\~4 h local) | Production with remote execution    |
| ---------------------------- | ----------------------- | ------------------------------ | ----------------------------------- |
| Source sync + Soong analysis | \~8 min                 | \~15 min                       | \~15 min                            |
| Parallelizable actions       | \~25 min (42%)          | \~195 min (**81%**)            | **\~50 min** (at the measured 3.9x) |
| Serial packaging tail        | \~26 min                | \~30 min                       | \~31 min                            |
| **Total**                    | \~60 min                | **\~4 h**                      | **\~1.5 h (\~2.5x)**                |

Model assumptions, stated so they can be challenged:

1. **Added volume is parallel-phase work.** What distinguishes a product build from OSS is
   compile and link volume: the exact work measured at 3.9x above.
2. **The serial tail is roughly constant in absolute terms**, give or take 15–20% growth for
   additional APK dexing and signing, because packaging structure does not scale with source
   volume.
3. **The parallel-phase speedup holds at 3.9x.** This is conservative: in the OSS run the
   512-slot fleet was idle after the first 3.6 minutes, so a longer parallel phase amortizes
   the same fleet better, and the fleet itself can be sized to the workload. With a fleet
   matched to a 4-hour build and output-download pruning enabled, the parallel-phase ratio
   has room above 3.9x.

Sensitivity: doubling the assumed tail still yields \~2.0x; holding the tail constant and
achieving 5x on the parallel phase yields \~2.8x. The justified range is **2.2–2.8x
end-to-end for a 4-hour-class build, \~2.5x central**, against the 1.38x measured on the
small OSS build. The direction of the relationship is the important part: **the larger the
AOSP build, the larger the share of it that remote execution accelerates.**

### Scaling the fleet beyond 512 slots

For the small OSS build, a larger fleet buys nothing: the 512 slots were already idle after
3.6 minutes. A 4-hour-class build is different: its \~195-minute parallel phase keeps a fleet
busy long enough that fleet size becomes a real lever. Scaling the measured configuration by
2–4x, with a stated scheduling-efficiency discount per step (90% / 85% / 80% of linear, to
account for narrowing graph width, cache-tier load, and client pipeline depth):

| Fleet                       | Slots | Parallel phase | Build total  | vs \~4 h local |
| --------------------------- | ----- | -------------- | ------------ | -------------- |
| 1x (measured configuration) | 512   | \~50 min       | **\~1h 36m** | **2.5x**       |
| 2x                          | 1,024 | \~28 min       | **\~1h 14m** | **3.3x**       |
| 3x                          | 1,536 | \~20 min       | **\~1h 06m** | **3.6x**       |
| 4x                          | 2,048 | \~16 min       | **\~1h 02m** | **3.9x**       |
| Limit (unbounded fleet)     | —     | 0              | \~46 min     | \~5.2x         |

Two readings of that table matter equally. First, **a fleet sized to the build pushes a
4-hour build toward one hour**: the 2x step alone recovers another 22 minutes. Second, the
returns diminish exactly as Amdahl predicts: the fixed \~46 minutes of sync, analysis and
serial tail is the floor no fleet reaches past, so each doubling buys roughly half what the
previous one did. The economically sensible operating point for a 4-hour-class build is
likely 2–3x the measured fleet; beyond that, effort is better spent on the serial tail
itself.

Scaling prerequisites, all configuration rather than architecture: the client's remote
pipeline depth (`NINJA_REMOTE_NUM_JOBS`) scales with slot count; the cache/storage tier is
sized at deployment time to match (its shard topology is fixed once data is resident); and
output-download pruning keeps the CI machine's network from becoming the funnel at higher
action rates.

Two further effects compound this in production use: real CI runs are not fully uncached, so
unchanged subtrees hit the shared remote cache and land under the uncached figure; and a
no-change run took 2m 38s (measured above).

## Cost considerations

This page's focus is build time; cost depends on fleet utilization and cache hit
rate, so what follows is a planning model with its assumptions stated. Two facts frame it: a
remote fleet adds compute that a local build does not consume, and a large dedicated runner
spends most of a long build under-utilized: dozens of vCPUs busy for the few minutes of
wide parallelism, then near-idle through the serial tail. Which effect wins is a question
of utilization, and utilization is set by build concurrency and caching.

### A modeled scenario: one production Android repo

Assume caching is already in place: warm persistent trees and a shared build cache. This
models what *adding remote execution* is worth on top of it.

The load is a typical day for a large organization shipping a production Android platform:
**\~300 presubmit delta builds, \~40 postsubmit integration builds, and \~15 full builds**
(the nightly target matrix plus release candidates), roughly 355 builds a day with dozens
in flight at peak hours.

Today's architecture is dedicated 64-vCPU runners. The comparison moves the same load to
16-vCPU runners plus one shared, demand-scaled fleet (up to 2,048 slots, the 4x row of the
fleet-scaling table above). Percentages are computed from on-demand list prices for the
instance types in this report; only the relative results are shown:

| Build type                       | Per day | Today (64-vCPU runners, cached) | Adding remote execution (16-vCPU runners + shared fleet) |
| -------------------------------- | ------- | ------------------------------- | -------------------------------------------------------- |
| Presubmit (delta, warm tree)     | \~300   | \~25 min median                 | **\~15 min median**                                      |
| Postsubmit integration           | \~40    | \~40 min                        | **\~22 min**                                             |
| Full build (nightly matrix, RCs) | \~15    | \~4 h                           | **\~1 h 10 m**                                           |

The gains concentrate where work actually executes. A cached presubmit only compiles its
delta, so remote execution trims it rather than transforms it; a full build executes
everything, and that's where 4 hours becomes \~70 minutes.

The outcome, on this load:

* **Compute cost: \~50% lower** than the same load on fully autoscaled large runners, and
  **\~90% lower** than a static pool sized for peak. Real AOSP runner groups sit between
  those bounds: a cold runner pays a \~39-minute source sync before its first build
  (measured above), which pushes teams toward keeping runners warm rather than scaling
  to zero.
* **One demand-scaled fleet serves everything.** The whole day is \~2,500–3,000 slot-hours
  of remote execution, absorbed at 60–70% fleet utilization; peak presubmit bursts queue
  for minutes, not hours.
* **\~100 hours of cumulative build waiting removed per day** across the team.

The basis: the OSS benchmark measured \~31 slot-hours of remote execution for a full build
(512 slots busy for 3.6 minutes), so a 4x-volume full build is \~125 slot-hours cold, and
\~60% of that on typical nightlies since the existing cache covers unchanged subtrees.
Cached presubmit deltas average \~4 slot-hours. The runner shrinks from 64 to 16 vCPU
because under remote execution it only runs Soong, drives the graph, and executes the
serial tail.

What the model depends on:

* **The fleet only pays for executed actions.** Caching already covers repeated work in
  both columns; remote execution buys speed on the work that remains, and its cost scales
  with that work rather than with the fleet's size on paper.
* **Load keeps the fleet utilized.** 355 builds a day is comfortably enough; the same fleet
  serving a handful of builds a week would be idle-heavy, and the comparison inverts.
* **Runner downsizing compounds across the pool**: every concurrent build needs a runner,
  and each one is a quarter the size.

## Verifying these results

Every run is public and reproducible:

* **Build logs**: the "Build AOSP" step of each linked run contains full ninja output with
  per-action timestamps.
* **Artifacts**: every run of the
  [AOSP Build workflow](https://github.com/aspect-build/basic-aosp-build/actions/workflows/aosp-build.yaml)
  uploads its full build logs as downloadable artifacts: the build log and the complete
  reclient logs, including `rbe_metrics.txt` with per-action completion statuses and a
  per-action record log. Download them from any run's page for full build details beyond
  what this page summarizes.
* **Build definition**: [aspect-build/basic-aosp-build](https://github.com/aspect-build/basic-aosp-build)
  contains the build script, the reclient configuration and the CI workflow.