Incidents and recovery

Incidents come in two shapes. Sometimes your infrastructure is on fire and you need the smallest, fastest change that resolves it. Other times a formae command itself is the problem: one you triggered by mistake, or one that is stuck. This guide covers both.

Fix a live incident with a patch

Patch mode is built for firefighting. It only creates or updates the resources you name and never destroys anything, so the blast radius is exactly the resource or two you are touching. See Apply modes for the full contrast with reconcile.

CLI
AI assistants

Write a focused forma

Include only the resources you are adding or fixing. The stack and target must already exist:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@aws/aws.pkl"
import "@aws/s3/bucket.pkl"

forma {
  new formae.Stack {
    label = "production"
    description = "Production infrastructure"
  }

  new formae.Target {
    label = "prod-target"
    config = new aws.Config {
      region = "us-east-1"
    }
  }

  new bucket.Bucket {
    label = "incident-logs"
    bucketName = "acme-incident-logs"
  }
}

The file does not need to live in your repo. Create it anywhere, for example /tmp/hotfix.pkl, apply it now, and fold it into your code later.

Apply the fix

formae shows the plan and asks you to confirm, then streams the run in a live view:

formae apply --mode patch hotfix.pkl

  formae apply · patch                                                                                  hotfix.pkl
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  + 1 create

  ▌ Resources
  Operation ▲   Label                                                       Type
  + create      incident-logs                                               AWS::S3::Bucket

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓: select  space: expand  →←: column  s: sort  y: confirm  q: abort This operation will create 1 resource(s).  Do
you want to continue? (y/N)  ?: help

Add --yes to skip the confirmation prompt when you are certain and need speed.

With the formae MCP plugin connected, describe the fix and let the assistant patch it in:

Conversation

You

We need an S3 bucket for incident logs in the production stack right now, without touching anything else.

Assistant

I’ll patch it in so nothing else is affected, and simulate first: it creates one resource, the S3 bucket incident-logs in production, and changes nothing else. Apply it?

You

Yes.

Assistant

Applied. incident-logs is live in production, and the rest of the stack is untouched. This is drift now, so fold it into your forma once the incident is over.

Settle up after the fix

A patch leaves your codebase describing something slightly different from reality: it created drift. Once the fire is out, you settle up in one of two ways, depending on whether the change was worth keeping. Until you do, the next soft reconcile on that stack detects the drift and fails, which is the reminder that code and reality disagree.

Absorb the change, if you want to keep it

The bucket you patched in is worth keeping, so make your code describe it. Extract the live resource into a forma file, fold it into your formae project, and reconcile. Now the stack matches your code again, and the resource is a first-class part of it.

CLI
AI assistants

Extract the patched resource

Pull the resource formae created during the incident into a forma file:

formae extract --query "stack:production label:incident-logs" ./incident-logs.pkl

Extracted 1 resource to ./incident-logs.pkl

incident-logs (AWS::S3::Bucket) on stack production

Fold it into your formae project and reconcile

Add the extracted resource to your formae project, then reconcile so code and infrastructure agree:

formae apply --mode reconcile main.pkl

╭─ formae apply ──────────────────────────────────────────────────────────────╮
│ No changes needed                                                          │
│                                                                            │
│ The specified forma resources are up to date.                              │
╰────────────────────────────────────────────────────────────────────────────╯

The reconcile is now a no-op for the bucket, because your code already describes it. The drift is gone.

Conversation

You

The incident is over. I patched an S3 bucket called incident-logs into production and never put it in my code. Pull it into my forma so we’re back in sync.

Assistant

I extracted incident-logs from the production stack and added it to your forma. Reconciling now shows no changes for it, so your code and the live stack agree again. Want me to reconcile to confirm?

You

Yes.

Assistant

Done. The stack reconciles clean: incident-logs is now part of your committed infrastructure, not drift.

Discard the change, if it was a throwaway

If the patch was a temporary workaround you do not want to keep, do the opposite: reconcile your unchanged code back over it. Reconcile removes anything not in the forma, so the patched resource goes away and the stack returns to exactly what your code describes.

CLI
AI assistants

formae apply --mode reconcile --force main.pkl

  formae apply · reconcile                                                                                main.pkl
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  - 1 delete

  ▌ Resources
  Operation ▲   Label                                                       Type
  - delete      incident-logs                                               AWS::S3::Bucket

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓: select  space: expand  →←: column  s: sort  y: confirm  q: abort This operation will delete 1 resource(s).  Do
you want to continue? (y/N)  ?: help

--force overwrites every out-of-band change on the stack, not just your patch. Reach for it only when you know what you are discarding. Extract and compare the current state first if you are unsure.

Conversation

You

That incident-logs bucket was a throwaway. Drop it and put production back to what my code says.

Assistant

Reconciling your unchanged forma over production removes incident-logs, the one resource not in your code, and leaves everything else as is. This discards the patch. Go ahead?

You

Go ahead.

Assistant

Done. incident-logs is gone and production matches your code exactly.

Get started

Guides

Concepts

Reference

Incidents and recovery

Fix a live incident with a patch

Settle up after the fix

Absorb the change, if you want to keep it

Discard the change, if it was a throwaway

See also

​Fix a live incident with a patch

​Settle up after the fix

​Absorb the change, if you want to keep it

​Discard the change, if it was a throwaway

​See also

Fix a live incident with a patch

Settle up after the fix

Absorb the change, if you want to keep it

Discard the change, if it was a throwaway

See also