Parallelize remote execution

With high parallelism on remote execution, Bazel can throttle execution on the client side while remote capacity sits idle: hundreds of actions ready to run, and a small fraction executing.

Symptoms

You’re likely hitting this if you see:

Many actions ready/queued, for example, 768 actions ready.
Only a small number actually running, for example, 72 actions running.
Abundant remote executor capacity, for example, 500+ container instances available.
Build profiles showing acquiringSemaphore traces.
Slower builds than expected despite high --jobs values.

Prerequisites

A working remote execution setup.
Bazel 7.0 or later.
High parallelism (for example, --jobs set to a multiple of host CPUs).

Diagnosing parallelization issues

To find out whether client-side throttling is limiting your build:

Step 1: Generate a build profile

bazel build //... --profile=profile.json

Step 2: Force a rebuild

To avoid cache hits when profiling:

bazel build //... \
  --profile=profile.json \
  --action_env=CACHE_BUST=$(date +%s)

Step 3: Inspect the profile

Look for acquiringSemaphore traces. They mean Bazel is waiting on internal semaphores rather than using the remote executors.

Configuration

These Bazel flags reduce client-side throttling and raise concurrency. Put them in your workspace .bazelrc, in the rbe group that registers your exec platforms, so they apply only when a build executes remotely. RBE platforms shows how to turn that group on from the --config group aspect setup bazelrc writes, or from an Aspect CLI task.

Bazel flags

# Disable client-side throttling for remote builds
build:rbe --noexperimental_throttle_remote_action_building
build:rbe --noexperimental_throttle_action_cache_check

# Reduce client memory for Merkle trees
build:rbe --experimental_remote_discard_merkle_trees

# Remove additional semaphore bottlenecks
build:rbe --noexperimental_use_semaphore_for_jobs

# Address Bazel 7 Skymeld regression
build:rbe --noexperimental_merged_skyframe_analysis_execution

What each flag does:

Flag	Description
`--noexperimental_throttle_remote_action_building`	Disables the `remoteActionBuildingSemaphore`, which can significantly slow down highly parallel remote builds. It keeps a client from overwhelming the remote executors, and becomes a bottleneck when you have substantial remote capacity. See bazelbuild/bazel#20478
`--noexperimental_throttle_action_cache_check`	Removes throttling when checking the action cache, so more cache lookups run in parallel.
`--experimental_remote_discard_merkle_trees`	Discards Merkle tree data structures once they’re no longer needed, reducing client-side memory.
`--noexperimental_use_semaphore_for_jobs`	Disables an internal semaphore that can limit job concurrency.
`--noexperimental_merged_skyframe_analysis_execution`	Disables Skymeld, which merges the analysis and execution phases. Skymeld has a known regression in Bazel 7 that slows remote execution down to actions running one at a time. See bazelbuild/bazel#22233

Memory flags that cost incrementality

If the Bazel client runs out of memory at high parallelism, these flags reduce what the Bazel server holds:

Flag	What it discards
`--heuristically_drop_nodes`	Skyframe nodes Bazel judges it won’t need again
`--notrack_incremental_state`	The dependency graph Bazel uses to rebuild incrementally
`--nokeep_state_after_build`	The analysis cache and build state, once the build completes

They cost warm incrementality: the next build on the same Bazel server re-analyzes from scratch. On Workflows runners, which keep the Bazel server and its analysis cache warm between jobs, that gives up most of what a warm runner saves. Use them only where the server isn’t reused, and fix memory with a larger client or --host_jvm_args first.

Job parallelism

A common pattern is a multiple of the host CPU count:

# Example: for a 24-core machine, 32 * 24 host CPUs
build:rbe --jobs=768

The exact multiplier depends on your workload characteristics and remote executor capacity.

Test and validation

To check the flags help:

Run a clean build with profiling enabled:

bazel clean
bazel build //... \
  --config=aspect-exec \
  --profile=profile-optimized.json \
  --action_env=CACHE_BUST=$(date +%s)

Check the build output to confirm more actions are running concurrently.
Analyze the profile to verify that acquiringSemaphore traces are fewer or gone.
Compare build times before and after applying these flags.

Before you keep a flag

Add flags one at a time. Profile after each and drop any that doesn’t help your workload, or that breaks the build.
Check your Bazel version. Flag behavior and support vary between Bazel releases; the release notes list known issues with experimental flags.
Check remote capacity. These flags remove client-side bottlenecks; the worker pool still needs the capacity for the extra actions in flight.

​Symptoms

​Prerequisites

​Diagnosing parallelization issues

​Step 1: Generate a build profile

​Step 2: Force a rebuild

​Step 3: Inspect the profile

​Configuration

​Bazel flags

​Memory flags that cost incrementality

​Job parallelism

​Test and validation

​Before you keep a flag