> ## Documentation Index > Fetch the complete documentation index at: https://site.aspect.build/llms.txt > Use this file to discover all available pages before exploring further. # aspect delivery > Deliver Bazel artifacts selectively with aspect delivery using content-hash change detection, deliveryd state tracking, dry-run, and CI examples. `aspect delivery` builds and dispatches Bazel targets (services, binaries, containers) to their destination. Its defining feature is **content-hash-based change detection**: a target is only re-delivered if its build outputs have actually changed, not just because a commit landed on main. On a monorepo with hundreds of services, this prevents cascade deploys. If a commit touches `//backend/api:server` but not `//frontend/web:app`, delivery re-deploys only the API service. The frontend deploy is a no-op — recorded as "already delivered" for this content, not skipped and not re-run. ## Selective delivery in depth Standard CI deploys trigger on commit SHA: every push to main re-deploys everything. At monorepo scale this means 200 services re-deploying when 1 changed. `aspect delivery` uses content hashes instead: **Phase 1: Hash extraction** A custom `hashsum` Bazel aspect runs alongside the normal build. It produces a per-target action digest by re-running the build with `--experimental_remote_require_cached` and extracting digest hashes from a gRPC execution log. This gives a stable, hermetic hash of each target's actual build outputs — not the source inputs, not the git SHA. **Phase 2: State query** `deliveryd` (a Unix-socket HTTP server started automatically on Aspect Workflows CI runners) stores delivery history as `(label, digest) → delivery event` tuples, keyed by commit SHA + task name. Before dispatching, the task queries deliveryd to check whether each `(label, hash)` pair has already been delivered. If yes, the target is skipped. If no, it's a delivery candidate. **Phase 3: Fused download and dispatch** Surviving candidates download their runfiles and dispatch as a single pipelined stream. Each target's `bazel run` starts the moment its runfiles land on disk, rather than waiting for every download to complete first. The task drains build events concurrently with the download, so a slow target no longer blocks earlier-ready ones. Up to `--max-parallelization` deliveries run in parallel (defaulting to hardware thread count), and after each dispatch deliveryd records the result. ### Why content hashes beat git SHAs Git SHAs capture "what committed?", not "what changed in the outputs?". Two commits that produce identical binaries (e.g., a docs-only change alongside a code change) get different SHAs. Content hashes capture the actual artifact identity, so identical outputs are never re-delivered. ### Uncacheable actions fail the task Change detection needs a cache entry for every action in a deliverable's dependency graph in order to compute its digest. If any action in that graph is uncacheable — typically a non-hermetic genrule (one that shells out to `git`, embeds a timestamp, etc.) or a stamped action — the digest can't be computed and the target would be silently skipped. By default `aspect delivery` treats this as a **hard failure**: the task exits non-zero and prints the offending target(s), the mnemonic(s) of the uncacheable action(s), and how to resolve it: ```text theme={null} Could not compute a change-detection digest for 2 targets because they — or an action they depend on — cannot be cached. Change detection needs a cache entry to tell whether a target changed, so these targets cannot be delivered. Uncacheable action(s): - //tools/stamp:version_stamp (Genrule) - //services/api:build_info (PyWriteBuildData) Resolve this by making the action cacheable, or by tagging it "no-remote" (or "local") so it runs locally instead of blocking delivery. ``` Two ways to fix each culprit: * Make the action deterministic and cacheable (remove the `git`/`date` call, drop `no-cache` tags, etc.). * Tag it `no-remote` or `local` so Bazel runs it locally on every invocation and never blocks the digest lookup on a missing remote cache entry. If your repo knowingly ships uncacheable actions on delivery targets and you'd rather keep the old lenient behavior, downgrade the failure to a warning with `--warn-unresolved-digest`: ```shell theme={null} aspect delivery --warn-unresolved-digest --task:name delivery ``` The task marks affected targets `WARN`, leaves them undelivered, and exits zero. Prefer fixing the underlying uncacheable action — the warning path silently drops targets from delivery. ## Configuration ### Delivery mode ```shell theme={null} aspect delivery --mode=selective --task:name delivery # default: skip unchanged aspect delivery --mode=always --task:name delivery # always deliver all resolved targets ``` `selective` is the correct mode for production pipelines. `always` is useful for initial setup, debugging, or forced rollouts. ### Resolving delivery targets ```shell theme={null} aspect delivery --query='kind(container_push, //services/...)' --task:name delivery ``` The `--query` flag is a Bazel query expression. Any Bazel query syntax works: `kind(...)`, `attr(...)`, unions, intersections. ### Forcing re-delivery ```shell theme={null} aspect delivery --force-target=//services/api:push --task:name delivery ``` Repeatable. Forces the listed targets to deliver even if their content hash matches a prior delivery. Useful for rollbacks or emergency re-deploys. ### Dry run ```shell theme={null} aspect delivery --dry-run --task:name delivery ``` Runs phases 1 and 2 (hash extraction, state query) without dispatching. Prints which targets would be delivered. Useful for validating your query and change-detection setup before the first real deploy. ### Parallelism ```shell theme={null} aspect delivery --max-parallelization=4 --task:name delivery ``` Default: all hardware threads. Set lower if your deploy targets have rate limits or serial ordering requirements. ### Stamping and Bazel flags Delivered binaries are **stamped by default** — `--release-bazel-flag` defaults to `--stamp`, so version-control info is embedded without any configuration. There are two separate flags for passing Bazel flags, and choosing the right one matters for change detection: | Flag | Applies to | Use for | | ---------------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | | `--release-bazel-flag` | the **final delivery build only** | `--stamp`, `--workspace_status_command=`, and any flag — including a `--config` — that adds non-determinism to release artifacts | | `--bazel-flag` | **every** build phase, including the change-detection digest | only flags you're certain don't change outputs run-to-run — e.g. `--jobs=100`, `--remote_cache=` | Put `--stamp`, `--workspace_status_command`, and any `--config` that enables them in `--release-bazel-flag`, **not** `--bazel-flag`. The change-detection digest (phases 1 and 2) is computed from the flags `--bazel-flag` feeds. Stamping injects volatile values — commit SHA, build time, workspace status — into those phases, so the digest shifts on **every commit**, change detection sees every target as new, and **everything re-delivers every time**. `--release-bazel-flag` applies only to the final delivery build, after change detection has decided what to deliver, so stamped artifacts stay stamped without polluting the digest. Watch for this via `--config`: a release config like `build:release --stamp --workspace_status_command=...` in `.bazelrc` pulls stamping in the moment it's activated, so `--config=release` belongs in `--release-bazel-flag` too. Only reach for `--bazel-flag` once you've confirmed a config adds no non-determinism. Set them once in `.aspect/config.axl`. A release `--config` typically enables stamping, so it goes in `release_bazel_flags`: ```python .aspect/config.axl theme={null} def config(ctx: ConfigContext): # Release-only flags — final delivery build only. Anything that adds # non-determinism to artifacts goes here, including a stamping --config. ctx.tasks["delivery"].args.release_bazel_flags = [ "--stamp", "--workspace_status_command=tools/bazel/workspace_status.sh", ] # Applied to every phase, including change detection — reserve for flags # confirmed not to change outputs (e.g. --jobs, --remote_cache). ctx.tasks["delivery"].args.bazel_flags = ["--jobs=100"] ``` Assigning release\_bazel\_flags **replaces** the default \["--stamp"] — include --stamp in your list if you still want stamping. To turn stamping off, set release\_bazel\_flags = \["--nostamp"] or pass --release-bazel-flag=--nostamp. Or pass either flag per invocation on the CLI. See [Stamp and release-only flags](/docs/cli/migration/delivery#stamp-and-release-only-flags) in the migration guide for CI examples. ### Isolate the delivery Bazel server (Aspect Workflows) On an [Aspect Workflows](/docs/aspect-workflows/overview) runner, every task shares one warm Bazel server (the runner-derived `--output_base`), keyed by its analysis cache. The delivery task's final build stamps and applies `--config=release`, reconfiguring the analysis — so on the shared server it discards the warm analysis cache the next `build`/`test` job on that runner would have reused (`Build options … have changed, discarding analysis cache`). `--workflows-runner-delivery-output-base-suffix=` moves that release build to its **own** persistent Bazel server by appending the suffix to the runner-derived `--output_base`, so its analysis stays warm across delivery runs without invalidating the cache the other jobs share. Only `--output_base` is made unique; `--output_user_root` stays shared, so the install base and the external-repository cache are reused rather than duplicated. (Delivery's plain change-detection build stays on the shared server; its checksum phase, which injects an aspect, automatically uses its own server regardless of this flag.) ```shell theme={null} aspect delivery --workflows-runner-delivery-output-base-suffix=delivery --task:name delivery ``` Set it once in `.aspect/config.axl`: ```python .aspect/config.axl theme={null} def config(ctx: ConfigContext): ctx.tasks["delivery"].args.workflows_runner_delivery_output_base_suffix = "delivery" ``` The flag only takes effect on Aspect Workflows runners (where the output base is runner-managed); it's a no-op on provider-hosted runners and local runs. See [Aspect Workflows runners](/docs/cli/tasks-ci#aspect-workflows-runners) for the runner-wide `--workflows:runner-output-base-suffix` that applies to every task. ## Delivery manifest Every invocation produces a structured **delivery manifest** — a JSON record of every target's outcome (`ok` / `skip` / `warn` / `fail` / `pending`), the resolved CI metadata, and any per-target enrichment your `config.axl` attached. It's uploaded as a CI artifact (one labeled download link per file on the GitHub Status Check / Buildkite annotation); pass `--manifest-file=` to also write it to disk. Four `DeliveryTrait` hooks let you enrich, render, upload, and act on the manifest. The on-disk file and the CI artifact have separate renderers so you can ship JSON for tooling AND YAML/CSV for humans (or any other split) from the same run: * `delivery_target(entry)` — per-target enrichment (e.g. attach the OCI image digest your push rule wrote alongside the binary). * `render_manifest_file(manifest)` — content of `--manifest-file`. Default: pretty-printed JSON. * `upload_manifest(manifest)` — list of CI artifacts to upload. Default: one JSON artifact. Return multiple entries to split the manifest across files / formats; return `[]` to disable uploads. * `delivery_manifest(manifest)` — end-of-task action (e.g. assemble an OCI layer in-task, post to an escrow registry, write an audit log). See the [Customize the delivery manifest](/docs/cli/guides/delivery-manifest) guide for the manifest schema, the hook signatures, and end-to-end examples. ## CI examples ```yaml GitHub Actions theme={null} on: push: branches: [main] jobs: delivery: runs-on: [self-hosted, aspect-workflows, aspect-default] permissions: id-token: write steps: - uses: actions/checkout@v6 - uses: aspect-build/setup-aspect@2306377a61c45954ab2df7c7311698b109364352 # v2026.26.9 with: aspect-api-token: ${{ secrets.ASPECT_API_TOKEN }} - run: > aspect delivery --task:name delivery --query='kind(container_push, //services/...)' ``` ```yaml Buildkite theme={null} steps: - label: ":rocket: Delivery" plugins: - aspect-build/setup-aspect#1d5768c5d28b72bf523b4722fc9177d2cc2d85c7: ~ # v2026.26.8 command: >- aspect delivery --task:name delivery --query='kind(container_push, //services/...)' agents: queue: aspect-default ``` ```yaml GitLab theme={null} include: - component: $CI_SERVER_FQDN/aspect-build/setup-aspect-gitlab-component/setup@2026.26.8 delivery: extends: .setup-aspect tags: [aspect-workflows, aspect-default] rules: - if: '$CI_COMMIT_BRANCH == "main"' script: > aspect delivery --task:name delivery --query='kind(container_push, //services/...)' # Set ASPECT_API_TOKEN as a masked CI/CD variable. ``` ```yaml CircleCI theme={null} version: 2.1 orbs: setup-aspect: aspect-build/setup-aspect@2026.26.10 jobs: delivery: machine: true resource_class: circleci-org/aspect-default steps: - checkout - setup-aspect/setup - run: command: > aspect delivery --task:name delivery --query='kind(container_push, //services/...)' ```