> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formae.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy the LGTM observability stack

> A Kubernetes infrastructure pattern, deployed with formae.

A full observability stack in one namespace, wired across two plugins. The
Kubernetes plugin deploys Loki, Tempo, and Mimir (each in Simple Scalable mode
with write, read, and backend tiers), MinIO as their shared object storage, an
OpenTelemetry Collector (gateway plus per-node agent), and Grafana behind a
LoadBalancer. The grafana plugin then configures Grafana over its HTTP API,
provisioning a dashboard folder, Loki, Tempo, and Mimir datasources, and the
shipped dashboards. Target chaining wires the two together: the grafana target's
URL is a reference on the Grafana Service's LoadBalancer ingress, so nothing is
hardcoded. Optional telemetrygen workloads produce synthetic traffic when you
pass `--enable-demo-traffic`.

```mermaid theme={"languages":{"custom":["/languages/pkl.json"]}}
flowchart LR
  target["k8s-target-local-lgtm"]:::tgt

  subgraph stack["k8s-lgtm-observability-local"]
    direction LR
    ns["Namespace (observability)"]:::res
    minioSecret["MinIO credentials Secret"]:::res
    minio["MinIO (object storage)"]:::res
    loki["Loki (logs)"]:::res
    tempo["Tempo (traces)"]:::res
    mimir["Mimir (metrics)"]:::res
    otel["OpenTelemetry Collector"]:::res
    grafana["Grafana"]:::res
    demo["telemetrygen (demo traffic)"]:::res

    grafanaTarget["grafana-target-local"]:::tgt
    folder["Dashboard folder"]:::res
    datasources["Datasources (Loki, Tempo, Mimir)"]:::res
    dashboards["Dashboards (Agent, Plugins)"]:::res

    ns --> minioSecret
    ns --> minio
    ns --> loki
    ns --> tempo
    ns --> mimir
    ns --> otel
    ns --> grafana
    ns --> demo
    minioSecret --> minio
    minioSecret --> loki
    minioSecret --> tempo
    minioSecret --> mimir
    loki --> minio
    tempo --> minio
    mimir --> minio
    demo --> otel
    otel --> loki
    otel --> tempo
    otel --> mimir
    loki --> datasources
    tempo --> datasources
    mimir --> datasources
    grafana --> grafanaTarget
    grafanaTarget --> folder
    grafanaTarget --> datasources
    grafanaTarget --> dashboards
    folder --> dashboards
  end

  target --> stack

  classDef tgt fill:#FF8201,stroke:#B25900,color:#ffffff
  classDef res fill:#FFF3E6,stroke:#FF8201,color:#02024B
  style stack fill:#ffffff,stroke:#02024B,stroke-width:2px,stroke-dasharray:6 4,color:#02024B
```

**Deploy.** Apply the example forma. formae orders the resources by their
dependencies automatically, so MinIO and its credentials come up before the
backends that store to it, and the Grafana Service before the grafana target that
derives its URL from it. Add `--enable-demo-traffic` to run synthetic telemetry
through telemetrygen:

```bash theme={"languages":{"custom":["/languages/pkl.json"]}}
formae apply --mode reconcile \
  /opt/pel/formae/examples/formae-plugin-kubernetes/lgtm-observability/local.pkl
```

**Verify.** Once the command completes, the stack's resources are under
management:

```bash theme={"languages":{"custom":["/languages/pkl.json"]}}
formae inventory resources --query="type:K8S::Apps::Deployment"
```

**Tear down.** Remove everything the forma created:

```bash theme={"languages":{"custom":["/languages/pkl.json"]}}
formae destroy \
  /opt/pel/formae/examples/formae-plugin-kubernetes/lgtm-observability/local.pkl
```

The full forma is in the
[lgtm-observability example](https://github.com/platform-engineering-labs/formae-plugin-kubernetes/tree/main/examples/lgtm-observability).