Observability

The formae agent provides comprehensive observability through OpenTelemetry, exposing metrics, logs, and traces to your observability platform.

What gets exported

When you enable OpenTelemetry, formae exports three types of telemetry via OTLP:

Telemetry	Transport	Description
Metrics	OTLP push	Agent stats, Go runtime, host metrics, database performance
Logs	OTLP push	Structured application logs with trace correlation
Traces	OTLP push	Distributed traces for operations and database queries

Additionally, a Prometheus-compatible /metrics endpoint is available for pull-based collection.

Enabling OpenTelemetry

Configure the oTel section in your configuration file:

agent {
    oTel {
        enabled = true
        serviceName = "formae-agent"
        otlp {
            enabled = true
            endpoint = "localhost:4317"
            protocol = "grpc"
            insecure = true
        }
    }
}

Property	Default	Description
`enabled`	`false`	Enable OpenTelemetry integration
`serviceName`	`"formae-agent"`	Service name for telemetry identification
`otlp.enabled`	`true`	Enable OTLP push export for traces, metrics, and logs
`otlp.endpoint`	`"localhost:4317"`	OTLP receiver endpoint
`otlp.protocol`	`"grpc"`	Transport protocol: `"grpc"` or `"http"`
`otlp.insecure`	`true`	Disable TLS for the OTLP connection
`otlp.temporality`	`"delta"`	Metric temporality: `"delta"` (OTel-native) or `"cumulative"`
`prometheus.enabled`	`true`	Enable Prometheus `/metrics` endpoint

!!! note "Metric Temporality" Use "delta" (default) for OpenTelemetry-native backends like Grafana Cloud or collectors with the deltatocumulative processor. Use "cumulative" for Prometheus/Mimir backends that don't support delta temporality.

Prometheus-Only Setup

You can disable OTLP push while keeping the Prometheus /metrics endpoint enabled for pull-based collection:

agent {
    oTel {
        enabled = true
        serviceName = "formae-agent"
        otlp {
            enabled = false  // Disable OTLP push
        }
        prometheus {
            enabled = true   // Keep /metrics endpoint
        }
    }
}

This configuration is useful when you want to use only Prometheus scraping without pushing telemetry to an OTLP collector.

Metrics

The agent exports metrics covering multiple aspects of the system:

Formae Stats

Metric	Labels	Description
`formae_stacks_total`	-	Number of stacks
`formae_targets_total`	`plugin`	Targets by plugin (AWS, Azure, etc.)
`formae_resources_managed`	`plugin`	Managed resources by plugin
`formae_resources_unmanaged`	`plugin`	Unmanaged resources by plugin
`formae_clients_connected`	-	Connected CLI clients
`formae_commands_total`	`command_type`	Commands by type (apply, destroy, sync, eval)
`formae_commands_by_state`	`state`	Commands by state (Pending, InProgress, Success, Failed, etc.)
`formae_resources_by_type`	`plugin`, `resource_type`	Resource count by type
`formae_resource_errors`	`plugin`, `resource_type`	Resources with errors by type

Infrastructure Metrics

Category	Metrics
Host	CPU time, memory usage, network I/O, disk I/O
Process	CPU usage (user/system)
Go Runtime	Goroutines, memory allocation, GC statistics
Ergo Actor System	Node uptime, processes, applications, network messages
Database	Connection pool stats, query latencies

Prometheus Endpoint

When enabled, metrics are also available at http://localhost:49684/api/v1/metrics for Prometheus scraping.

Logs

Structured logs are pushed to your OTLP endpoint with automatic trace correlation. Log entries include:

Timestamp and log level
Service name and instance
Trace and span IDs (when within a traced operation)
Structured fields for context

Local file logging remains available regardless of OpenTelemetry status:

agent {
    logging {
        filePath = "~/.pel/formae/log/formae.log"
        fileLogLevel = "debug"
        consoleLogLevel = "info"
    }
}

Traces

Distributed traces cover agent operations and provide visibility into:

API request handling
Resource operations (create, read, update, delete)
Database queries (with query text and latency)
Plugin interactions

Traces are correlated with logs, allowing you to jump from a log entry to its parent trace.

Grafana Dashboards

Pre-built Grafana dashboards are available in the formae-grafana-dashboards repository.

Manual Import

To import dashboards into your Grafana instance:

Download the dashboard JSON from the formae-grafana-dashboards repository
In Grafana, navigate to Dashboards > Import
Upload the JSON file or paste its contents
Select your Prometheus and Loki datasources

Provisioning

For automated provisioning, clone the repository and add to your Grafana configuration:

apiVersion: 1
providers:
  - name: 'formae'
    orgId: 1
    folder: 'Formae'
    type: file
    options:
      path: /path/to/formae-grafana-dashboards/dashboards

Example: Complete Setup

Here's a complete configuration for local development:

amends "formae:/Config.pkl"

agent {
    oTel {
        enabled = true
        serviceName = "formae-agent"
        otlp {
            endpoint = "localhost:4317"
            protocol = "grpc"
            insecure = true
        }
    }

    logging {
        filePath = "~/.pel/formae/log/formae.log"
        fileLogLevel = "debug"
        consoleLogLevel = "info"
    }
}

With your observability stack running, start the agent and view metrics in Grafana.