> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formae.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Observe the agent

> Export the formae agent's metrics, logs, and traces through OpenTelemetry, or scrape them from a Prometheus endpoint.

The formae agent exposes its own telemetry through OpenTelemetry: metrics, logs,
and traces pushed over OTLP, plus a Prometheus-compatible `/metrics` endpoint for
pull-based collection. This guide turns it on and shows what you get. The full
list of configuration knobs lives in the
[configuration reference](/documentation/reference/configuration#opentelemetry).

## What gets exported

With OpenTelemetry enabled, the agent exports three signals over OTLP:

| Signal  | Transport | What it covers                                              |
| ------- | --------- | ----------------------------------------------------------- |
| Metrics | OTLP push | Agent stats, Go runtime, host metrics, database performance |
| Logs    | OTLP push | Structured application logs, correlated with traces         |
| Traces  | OTLP push | API requests, resource operations, and database queries     |

A Prometheus-compatible endpoint is also available at
`http://localhost:49684/api/v1/metrics` for pull-based scraping.

## Enable OpenTelemetry

Add an `oTel` block to your [configuration file](/documentation/reference/configuration):

```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}}
agent {
  oTel {
    enabled = true
    serviceName = "formae-agent"
    otlp {
      enabled = true
      endpoint = "localhost:4317"
      protocol = "grpc"
      insecure = true
    }
  }
}
```

Restart the agent and it begins pushing telemetry to the OTLP endpoint. For the
complete property table (including `otlp.temporality` and the `prometheus`
settings), see the
[configuration reference](/documentation/reference/configuration#opentelemetry).

<Note>
  **Metric temporality.** Keep the default `delta` for OpenTelemetry-native backends
  such as Grafana Cloud, or a collector running the `deltatocumulative` processor.
  Switch to `cumulative` for Prometheus or Mimir backends that do not support delta
  temporality.
</Note>

### Scrape with Prometheus instead

To scrape metrics rather than push them, keep the Prometheus endpoint on and turn
OTLP push off:

```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}}
agent {
  oTel {
    enabled = true
    serviceName = "formae-agent"
    otlp {
      enabled = false
    }
    prometheus {
      enabled = true
    }
  }
}
```

## Key metrics

Alongside Go runtime, host, Ergo actor-system, and database metrics, the agent
exports formae-specific stats:

| Metric                       | Labels                    | Description                                              |
| ---------------------------- | ------------------------- | -------------------------------------------------------- |
| `formae_stacks_total`        |                           | Number of stacks                                         |
| `formae_targets_total`       | `plugin`                  | Targets by plugin                                        |
| `formae_resources_managed`   | `plugin`                  | Managed resources by plugin                              |
| `formae_resources_unmanaged` | `plugin`                  | Unmanaged resources by plugin                            |
| `formae_commands_total`      | `command_type`            | Commands by type (apply, destroy, sync, eval)            |
| `formae_commands_by_state`   | `state`                   | Commands by state (Pending, InProgress, Success, Failed) |
| `formae_resources_by_type`   | `plugin`, `resource_type` | Resource count by type                                   |
| `formae_resource_errors`     | `plugin`, `resource_type` | Resources with errors by type                            |

## Logs and traces

Structured logs are pushed to the OTLP endpoint with trace and span IDs attached,
so you can jump from a log line to the operation that produced it. Local file
logging stays available whatever the OpenTelemetry setting:

```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}}
agent {
  logging {
    filePath = "~/.pel/formae/log/formae.log"
    fileLogLevel = "debug"
    consoleLogLevel = "info"
  }
}
```

Traces cover API request handling, resource create/read/update/delete operations,
database queries (with query text and latency), and plugin interactions, all
correlated with the matching logs.

## Grafana dashboards

Pre-built dashboards live in the
[formae-grafana-dashboards](https://github.com/platform-engineering-labs/formae-grafana-dashboards)
repository. To import one by hand, open **Dashboards > Import** in Grafana and
upload the dashboard JSON, then select your Prometheus and Loki datasources. To
provision them automatically, point a Grafana dashboard provider at the cloned
repository:

```yaml theme={"languages":{"custom":["/languages/pkl.json"]}}
apiVersion: 1
providers:
  - name: 'formae'
    orgId: 1
    folder: 'Formae'
    type: file
    options:
      path: /path/to/formae-grafana-dashboards/dashboards
```

## See also

* [Configuration reference](/documentation/reference/configuration#opentelemetry): every `oTel` and `prometheus` property.
* [Deploy the LGTM observability stack](/documentation/reference/providers/kubernetes/patterns/deploy-the-lgtm-observability-stack): stand up Grafana, Loki, Tempo, and Mimir with formae.