> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formae.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Incidents and recovery

> Fix a live infrastructure incident fast with a patch and settle up afterwards, and deal with a formae command that is itself the problem.

Incidents come in two shapes. Sometimes your infrastructure is on fire and you
need the smallest, fastest change that resolves it. Other times a formae command
itself is the problem: one you triggered by mistake, or one that is stuck. This
guide covers both.

## Fix a live incident with a patch

Patch mode is built for firefighting. It only creates or updates the resources you
name and never destroys anything, so the blast radius is exactly the resource or
two you are touching. See [Apply modes](/documentation/concepts/apply-modes) for
the full contrast with reconcile.

<Tabs>
  <Tab title="CLI">
    <Steps>
      <Step title="Write a focused forma">
        Include only the resources you are adding or fixing. The stack and target
        must already exist:

        ```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}}
        amends "@formae/forma.pkl"
        import "@formae/formae.pkl"
        import "@aws/aws.pkl"
        import "@aws/s3/bucket.pkl"

        forma {
          new formae.Stack {
            label = "production"
            description = "Production infrastructure"
          }

          new formae.Target {
            label = "prod-target"
            config = new aws.Config {
              region = "us-east-1"
            }
          }

          new bucket.Bucket {
            label = "incident-logs"
            bucketName = "acme-incident-logs"
          }
        }
        ```

        <Tip>
          The file does not need to live in your repo. Create it anywhere, for
          example `/tmp/hotfix.pkl`, apply it now, and fold it into your code later.
        </Tip>
      </Step>

      <Step title="Apply the fix">
        formae shows the plan and asks you to confirm, then streams the run in a
        live view:

        ```bash theme={"languages":{"custom":["/languages/pkl.json"]}}
        formae apply --mode patch hotfix.pkl
        ```

        <div className="fa-term">
          <div className="fa-term-body">  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(232,232,232)',fontWeight:'600'}}>formae</span> <span style={{fontWeight:'600'}} /><span style={{color:'rgb(255,133,51)',fontWeight:'600'}}>apply · patch</span>                                                                                  hotfix.pkl<br /><span style={{color:'rgb(85,85,102)'}}>────────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span><br /><span style={{color:'rgb(85,85,102)'}} />  <span style={{color:'rgb(232,232,232)'}}>+</span> 1 <span style={{color:'rgb(232,232,232)'}}>create</span><br /><br />  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(255,133,51)',fontWeight:'600'}}>▌ Resources</span><br />  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(232,232,232)',fontWeight:'600'}} /><span style={{color:'rgb(232,232,232)',fontWeight:'600'}}>Operation ▲   </span><span style={{color:'rgb(170,170,170)'}}>Label                                                       Type</span><br />  <span style={{color:'rgb(232,232,232)'}}>+ create      incident-logs                                               AWS::S3::Bucket</span><br /><br /><span style={{color:'rgb(85,85,102)'}}>────────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span><br /><span style={{color:'rgb(85,85,102)'}} />  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>↑↓</span><span style={{color:'rgb(153,153,153)'}}>: select</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>space</span><span style={{color:'rgb(153,153,153)'}}>: expand</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>→←</span><span style={{color:'rgb(153,153,153)'}}>: column</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>s</span><span style={{color:'rgb(153,153,153)'}}>: sort</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>y</span><span style={{color:'rgb(153,153,153)'}}>: confirm</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>q</span><span style={{color:'rgb(153,153,153)'}}>: abort</span> This operation will create 1 resource(s).  Do<br />you want to continue? (y/N)  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>?</span><span style={{color:'rgb(153,153,153)'}}>: help</span></div>
        </div>

        Add `--yes` to skip the confirmation prompt when you are certain and need
        speed.
      </Step>
    </Steps>
  </Tab>

  <Tab title="AI assistants">
    With the [formae MCP plugin](/documentation/guides/ai-coding-assistants)
    connected, describe the fix and let the assistant patch it in:

    <div className="fa-convo">
      <div className="fa-convo-head">Conversation</div>
      <div className="fa-turn fa-you"><div className="fa-tag">You</div><div className="fa-body">We need an S3 bucket for incident logs in the production stack right now, without touching anything else.</div></div>
      <div className="fa-turn fa-bot"><div className="fa-tag">Assistant</div><div className="fa-body">I'll patch it in so nothing else is affected, and simulate first: it creates one resource, the S3 bucket <code>incident-logs</code> in <code>production</code>, and changes nothing else. Apply it?</div></div>
      <div className="fa-turn fa-you"><div className="fa-tag">You</div><div className="fa-body">Yes.</div></div>
      <div className="fa-turn fa-bot"><div className="fa-tag">Assistant</div><div className="fa-body">Applied. <code>incident-logs</code> is live in <code>production</code>, and the rest of the stack is untouched. This is drift now, so fold it into your forma once the incident is over.</div></div>
    </div>
  </Tab>
</Tabs>

## Settle up after the fix

A patch leaves your codebase describing something slightly different from reality:
it created drift. Once the fire is out, you settle up in one of two ways, depending
on whether the change was worth keeping. Until you do, the next soft reconcile on
that stack detects the drift and fails, which is the reminder that code and reality
disagree.

### Absorb the change, if you want to keep it

The bucket you patched in is worth keeping, so make your code describe it. Extract
the live resource into a forma file, fold it into your formae project, and
reconcile. Now the stack matches your code again, and the resource is a first-class
part of it.

<Tabs>
  <Tab title="CLI">
    <Steps>
      <Step title="Extract the patched resource">
        Pull the resource formae created during the incident into a forma file:

        ```bash theme={"languages":{"custom":["/languages/pkl.json"]}}
        formae extract --query "stack:production label:incident-logs" ./incident-logs.pkl
        ```

        <div className="fa-term">
          <div className="fa-term-body"><span style={{color:'rgb(232,232,232)'}}>Extracted 1 resource to ./incident-logs.pkl</span><br /><br />  <span style={{color:'rgb(129,209,219)'}}>incident-logs</span> (<span style={{color:'rgb(170,170,170)'}}>AWS::S3::Bucket</span>) on stack <span style={{color:'rgb(232,232,232)'}}>production</span></div>
        </div>
      </Step>

      <Step title="Fold it into your formae project and reconcile">
        Add the extracted resource to your formae project, then reconcile so code
        and infrastructure agree:

        ```bash theme={"languages":{"custom":["/languages/pkl.json"]}}
        formae apply --mode reconcile main.pkl
        ```

        <div className="fa-term">
          <div className="fa-term-body"><span style={{color:'rgb(85,85,102)'}}>╭─</span> formae apply <span style={{color:'rgb(85,85,102)'}}>──────────────────────────────────────────────────────────────╮</span><br /><span style={{color:'rgb(85,85,102)'}}>│</span> No changes needed                                                          <span style={{color:'rgb(85,85,102)'}}>│</span><br /><span style={{color:'rgb(85,85,102)'}}>│</span>                                                                            <span style={{color:'rgb(85,85,102)'}}>│</span><br /><span style={{color:'rgb(85,85,102)'}}>│</span> The specified forma resources are up to date.                              <span style={{color:'rgb(85,85,102)'}}>│</span><br /><span style={{color:'rgb(85,85,102)'}}>╰────────────────────────────────────────────────────────────────────────────╯</span></div>
        </div>

        The reconcile is now a no-op for the bucket, because your code already
        describes it. The drift is gone.
      </Step>
    </Steps>
  </Tab>

  <Tab title="AI assistants">
    <div className="fa-convo">
      <div className="fa-convo-head">Conversation</div>
      <div className="fa-turn fa-you"><div className="fa-tag">You</div><div className="fa-body">The incident is over. I patched an S3 bucket called incident-logs into production and never put it in my code. Pull it into my forma so we're back in sync.</div></div>
      <div className="fa-turn fa-bot"><div className="fa-tag">Assistant</div><div className="fa-body">I extracted <code>incident-logs</code> from the <code>production</code> stack and added it to your forma. Reconciling now shows no changes for it, so your code and the live stack agree again. Want me to reconcile to confirm?</div></div>
      <div className="fa-turn fa-you"><div className="fa-tag">You</div><div className="fa-body">Yes.</div></div>
      <div className="fa-turn fa-bot"><div className="fa-tag">Assistant</div><div className="fa-body">Done. The stack reconciles clean: <code>incident-logs</code> is now part of your committed infrastructure, not drift.</div></div>
    </div>
  </Tab>
</Tabs>

### Discard the change, if it was a throwaway

If the patch was a temporary workaround you do not want to keep, do the opposite:
reconcile your unchanged code back over it. Reconcile removes anything not in the
forma, so the patched resource goes away and the stack returns to exactly what your
code describes.

<Tabs>
  <Tab title="CLI">
    ```bash theme={"languages":{"custom":["/languages/pkl.json"]}}
    formae apply --mode reconcile --force main.pkl
    ```

    <div className="fa-term">
      <div className="fa-term-body">  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(232,232,232)',fontWeight:'600'}}>formae</span> <span style={{fontWeight:'600'}} /><span style={{color:'rgb(255,133,51)',fontWeight:'600'}}>apply · reconcile</span>                                                                                main.pkl<br /><span style={{color:'rgb(85,85,102)'}}>────────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span><br /><span style={{color:'rgb(85,85,102)'}} />  <span style={{color:'rgb(232,232,232)'}}>-</span> 1 <span style={{color:'rgb(232,232,232)'}}>delete</span><br /><br />  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(255,133,51)',fontWeight:'600'}}>▌ Resources</span><br />  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(232,232,232)',fontWeight:'600'}} /><span style={{color:'rgb(232,232,232)',fontWeight:'600'}}>Operation ▲   </span><span style={{color:'rgb(170,170,170)'}}>Label                                                       Type</span><br />  <span style={{color:'rgb(232,232,232)'}}>- delete      incident-logs                                               AWS::S3::Bucket</span><br /><br /><span style={{color:'rgb(85,85,102)'}}>────────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span><br /><span style={{color:'rgb(85,85,102)'}} />  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>↑↓</span><span style={{color:'rgb(153,153,153)'}}>: select</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>space</span><span style={{color:'rgb(153,153,153)'}}>: expand</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>→←</span><span style={{color:'rgb(153,153,153)'}}>: column</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>s</span><span style={{color:'rgb(153,153,153)'}}>: sort</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>y</span><span style={{color:'rgb(153,153,153)'}}>: confirm</span>  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>q</span><span style={{color:'rgb(153,153,153)'}}>: abort</span> This operation will delete 1 resource(s).  Do<br />you want to continue? (y/N)  <span style={{fontWeight:'600'}} /><span style={{color:'rgb(129,209,219)',fontWeight:'600'}}>?</span><span style={{color:'rgb(153,153,153)'}}>: help</span></div>
    </div>

    <Warning>
      `--force` overwrites every out-of-band change on the stack, not just your patch.
      Reach for it only when you know what you are discarding. Extract and compare the
      current state first if you are unsure.
    </Warning>
  </Tab>

  <Tab title="AI assistants">
    <div className="fa-convo">
      <div className="fa-convo-head">Conversation</div>
      <div className="fa-turn fa-you"><div className="fa-tag">You</div><div className="fa-body">That incident-logs bucket was a throwaway. Drop it and put production back to what my code says.</div></div>
      <div className="fa-turn fa-bot"><div className="fa-tag">Assistant</div><div className="fa-body">Reconciling your unchanged forma over <code>production</code> removes <code>incident-logs</code>, the one resource not in your code, and leaves everything else as is. This discards the patch. Go ahead?</div></div>
      <div className="fa-turn fa-you"><div className="fa-tag">You</div><div className="fa-body">Go ahead.</div></div>
      <div className="fa-turn fa-bot"><div className="fa-tag">Assistant</div><div className="fa-body">Done. <code>incident-logs</code> is gone and <code>production</code> matches your code exactly.</div></div>
    </div>
  </Tab>
</Tabs>

## See also

* [Apply modes](/documentation/concepts/apply-modes): patch versus reconcile, and how drift is detected and resolved.
* [Synchronization](/documentation/concepts/synchronization): how formae detects the out-of-band changes a patch introduces.
* [Everyday changes](/documentation/guides/everyday-changes): the routine inspect, change, and tear-down operations.
* [Troubleshoot a failed command](/documentation/guides/troubleshoot-a-failed-command): read the detailed status layout to see which resource broke and why.
* [Cancel a running command](/documentation/guides/cancel-a-running-command): stop an in-progress command without leaving orphaned resources.
* [CLI reference](/documentation/reference/cli): full flags for `apply`, `cancel`, `status`, `inventory`, and `extract`.
