Operating the AWS agent

Procedures that apply to a formae agent deployed on AWS, regardless of whether you followed the Bootstrap, ECS Express, or Standard ECS walkthrough.

Updating the agent (Bootstrap)

If you installed with the Bootstrap installer, upgrade the agent the same way you created it: re-apply bootstrap.pkl with a newer image. --formae-image is the version knob. Find the target tag at GitHub Releases.

formae apply --mode reconcile aws/bootstrap.pkl <same flags as your apply> \
  --formae-image ghcr.io/platform-engineering-labs/formae:<new-version> \
  --watch --status-output-layout detailed

Pass the same flags you applied with (access mode, region, credentials, and the alb/tailnet options) so the reconcile changes only the image. formae plans a new task definition and a rolling service update. Then match your local CLI to the new agent version:

formae upgrade --version <new-version>

Keep the install you bootstrapped from. formae runs as a client and an agent. The local install you ran bootstrap.pkl from holds your agent's own infrastructure — the VPC, database, and ECS service that are the agent — in its state. That state lives only there: the deployed agent manages your workloads, not itself. Keep that local install and its datastore — re-applying bootstrap.pkl to upgrade or change the agent depends on it.

If you no longer have it, fall back to the manual Standard ECS update procedure — but note that rolling the task definition by hand leaves the agent's deployed state ahead of what bootstrap.pkl records.

Rolling back

Re-apply with the previous image tag:

formae apply --mode reconcile aws/bootstrap.pkl <same flags as your apply> \
  --formae-image ghcr.io/platform-engineering-labs/formae:<previous-version> --watch

Custom images with extra plugins

The stock formae image ships with the standard plugin metapackage — AWS, Azure, GCP, OCI, OVH, and the auth-basic schema. Most real deployments need plugins beyond this set: Grafana for observability, Kubernetes, your own custom plugins, etc.

To add plugins, build a custom image FROM the stock formae image and run the install script for each additional plugin:

FROM ghcr.io/platform-engineering-labs/formae:<version>

USER root
RUN apt-get update && apt-get install -y --no-install-recommends jq curl && \
    HOME=/home/pel /bin/bash -e -c "$(curl -fsSL https://hub.platform.engineering/get/setup.sh)" \
      -- install --yes --channel stable grafana@<plugin-version> && \
    HOME=/home/pel /bin/bash -e -c "$(curl -fsSL https://hub.platform.engineering/get/setup.sh)" \
      -- install --yes --channel stable k8s@<plugin-version> && \
    apt-get remove -y jq curl && apt-get autoremove -y --purge && \
    apt-get clean && rm -rf /var/lib/apt/lists/*
USER pel

jq and curl are required by the install script but aren't kept in the final image. All plugin installs run in the same RUN block so those tools are present when needed and removed at the end. Pin specific plugin versions for reproducibility.

If you're also baking in the RDS Global CA bundle for Standard ECS, combine both patterns in a single Dockerfile.

Build and push to your registry, then use that image when you update the agent — as --formae-image for a Bootstrap upgrade, or as the task definition image for Standard ECS / Express.

Aurora master credential rotation

If you're using Aurora as the agent's datastore (Data API or direct PostgreSQL), be aware of this: Aurora's managed master credential rotation (the default 30-day cycle) updates the master password on the cluster but does not update the agent's formae-config Secrets Manager secret, which holds the connection string. The agent will fail to connect after the next rotation.

The simplest mitigation is to disable managed rotation on the cluster's master credential and rotate manually when you choose to:

aws rds modify-db-cluster \
  --db-cluster-identifier formae-db \
  --manage-master-user-password false \
  --apply-immediately

After disabling, supply the master password directly with --master-user-password on cluster modifications.

In theory you could keep rotation enabled and wire a Lambda that re-renders the formae-config secret whenever the cluster's master credential rotates. We don't currently ship a reference implementation; disabling rotation is the practical recommendation.

Either way: address this before the 30-day mark — discovering it after the first rotation is no fun.

Backups

The agent itself is stateless; everything recoverable lives in the datastore. Aurora's default automated backups (1-day retention, point-in-time recovery within the retention window) cover the formae state. For longer retention, either bump the cluster's --backup-retention-period or run scheduled aws rds create-db-cluster-snapshot jobs.

Shell into the agent

The service runs with ECS Exec enabled, so you can open a shell inside the running container (Fargate has no host to docker exec into). Requires the AWS Session Manager plugin installed locally.

aws ecs execute-command --region <region> \
  --cluster <cluster-name> \
  --task "$(aws ecs list-tasks --region <region> --cluster <cluster-name> --query 'taskArns[0]' --output text)" \
  --container formae-agent --interactive --command /bin/sh

Teardown (Bootstrap)

For an agent installed with the Bootstrap installer, tear down in two steps — destroy removes the resources but not the target registration:

formae destroy --yes aws/bootstrap.pkl <same flags as your apply>
formae destroy --yes aws/destroy-target.pkl

Production tuning (Bootstrap)

The Bootstrap installer favours a lean default. For production, edit aws/bootstrap.pkl / aws/vars.pkl:

Multi-AZ database: the RDS instance is single-AZ by default — a single point of failure. Set multiAZ = true for automatic failover (roughly doubles RDS cost and deploy time).
Storage: maxAllocatedStorage caps storage autoscaling at 100 GB; raise it for larger inventories.
Ingress (alb mode): scope the ALB to your own network with --allow-cidr, or use tailnet mode for no public surface at all.
Sizing: choose a larger --size as your inventory grows (see the sizing table).