> ## Documentation Index
> Fetch the complete documentation index at: https://site.aspect.build/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom machine images

> Build a custom AWS AMI or GCP machine image for Aspect Workflows CI runners with Packer, and reference it from a resource_types entry.

Aspect Workflows runs Bazel directly on the host machine, not in a container, to avoid Docker-in-Docker complications. Runners therefore use a cloud machine image, not a container image. Aspect publishes open-source [starter machine images](/docs/aspect-workflows/enterprise/self-hosted/infrastructure/starter-images), but you may need to build a custom image with Packer when:

* You require tight control over security patches and versions of the OS and packages.
* The starter images don't include fixes for every vulnerability you track.
* Your non-hermetic build or tests require specific system-level packages.

## Step 1: Prepare the build machine

1. [Download Packer for your operating system](https://developer.hashicorp.com/packer/downloads). To have Bazel manage the Packer binary instead, see the starter images repository linked in step 5.
2. Authenticate with your cloud provider (Amazon Web Services or Google Cloud Platform). Packer needs permission to create EC2 or Compute Engine instances.

## Step 2: Choose a base image

### AWS

1. Access the AWS console.
2. Access the **EC2 Dashboard** and select **AMIs** from the left sidebar.

<Tip>
  Start with an **Amazon-supplied** image, for example Amazon Linux 2023 (<code>al2023-ami</code>).
  See Packer's [Getting Started with AWS](https://developer.hashicorp.com/packer/tutorials/aws-get-started) for building AMIs.
</Tip>

### GCP

1. Access the GCP console.
2. Navigate to **Compute Engine** > **Images**.
3. Search for the image that suits your needs, for example `debian-13` or `ubuntu-2404-lts`.

<Tip>
  See Packer's [Google Compute Builder](https://developer.hashicorp.com/packer/integrations/hashicorp/googlecompute/latest/components/builder/googlecompute) for building GCP images.
</Tip>

## Step 3: Create a Packer script

Packer scripts use the HashiCorp Configuration Language (HCL), like Terraform, with the `.pkr.hcl` file extension.

1. Add the Packer plugin for your cloud provider to your Packer script:
   * For AWS use [Amazon EBS](https://developer.hashicorp.com/packer/integrations/hashicorp/amazon/latest/components/builder/ebs).
   * For GCP use [Google Cloud Platform](https://developer.hashicorp.com/packer/integrations/hashicorp/googlecompute/latest/components/builder/googlecompute).
2. Create a `locals` block for the values you reuse:

```hcl theme={null}
locals {
  # This is a public Amazon Linux 2 image in us-east-2
  # We use this AMI because it already contains everything needed to interact with AWS
  # Name: amzn2-ami-kernel-5.10-hvm-2.0.20220719.0-x86_64-gp2
  source_ami = "ami-051dfed8f67f095f5"
  region = "us-east-2"
  platform = "linux/amd64"
}
```

3. Create a `source` block and a `build` block, following the Packer documentation.

## Step 4: Dependencies

### Minimal dependencies

Every custom machine image needs:

| Dependency | Purpose                                                       |
| :--------- | :------------------------------------------------------------ |
| `fuse`     | Required for the high-performance remote cache configuration. |
| `git`      | Used for fetching source code for testing.                    |
| `mdadm`    | Required when mounting NVMe drives with raid 0.               |
| `rsync`    | Used during bootstrap.                                        |
| `rsyslog`  | Used for system logging.                                      |

### AWS AMI requirements

AWS AMIs also need:

| Dependency                | Purpose                                       |
| :------------------------ | :-------------------------------------------- |
| `amazon-cloudwatch-agent` | Used for gathering logs.                      |
| `amazon-ssm-agent`        | Required for Amazon Web Services SSM support. |

<Note>
  <code>amazon-ssm-agent</code> comes pre-installed on Amazon Linux 2 and Amazon Linux 2023 base AMIs.
</Note>

### GCP image requirements

GCP images also need:

| Dependency              | Purpose                                                                              |
| :---------------------- | :----------------------------------------------------------------------------------- |
| `google-osconfig-agent` | Google operational monitoring tools used to collect and alarm on critical telemetry. |

### GitHub Actions requirements

Runners for GitHub Actions also need:

| Dependency | Purpose                                                                                                                                                           |
| :--------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `libicu`   | Needed by the GitHub Actions agent. See [Runner v2.303 not working on AmazonLinux2023 due to missing ICU package](https://github.com/actions/runner/issues/2511). |

### Recommended dependencies

| Dependency | Purpose                                                                           |
| :--------- | :-------------------------------------------------------------------------------- |
| `zip`      | Required by **Bazel** if any tests create archives of undeclared test outputs.    |
| `patch`    | May be used by certain rule sets and package managers during dependency fetching. |

## Step 5: Add dependencies for your build

Install the packages your build needs, such as Docker, with `apt-get` or `yum` depending on your Linux distribution:

```hcl theme={null}
build {

  ...

  provisioner "shell" {
    inline = [
      # Install additional dependencies
      "sudo apt-get update",
      "sudo apt-get --assume-yes install clang-13 libgdal-dev openjdk-17-jdk-headless libtinfo5"
    ]
  }
}
```

<Tip>
  The [Aspect Workflows starter images](https://github.com/aspect-build/workflows-images) repository has complete Packer files and a Bazel setup that runs a hermetic Packer binary.
</Tip>

## Step 6: Test a new image

Test image changes on a separate `canary` runner group before they reach the runners your developers depend on. In your Workflows Terraform configuration:

1. Add a `data "aws_ami"` or `data "google_compute_image"` block that selects the `canary` image.
2. Add an entry to `resource_types` with `image_id` set to the `canary` image ID.
3. Add a runner group under `runners` (for example in `runners.bk.groups`) that selects the new resource type.
4. Run `terraform apply`.
5. Point one or more jobs in your CI configuration at the canary runner group's queue.

To test the new image fully, force a complete, non-incremental build by adding cache-busting environment variables for actions and repository rules to `.bazelrc`:

```shell theme={null}
# cache-bust for testing canary, values are arbitrary
build --repo_env=CACHE_BUST=001 --action_env=CACHE_BUST=001
```