> ## Documentation Index > Fetch the complete documentation index at: https://docs.formae.io/llms.txt > Use this file to discover all available pages before exploring further. # 11 - Conformance tests > Verify your plugin behaves correctly when orchestrated by the formae agent by running the conformance test suite against the full resource lifecycle. Now that we've built our plugin and verified it works with unit and integration tests, we're ready to run **conformance tests**. These tests verify that your plugin correctly integrates with the formae agent, the component that orchestrates resource lifecycle operations in production. ## Why conformance tests? Unit tests verify your plugin logic in isolation. Integration tests verify it works with real infrastructure. Conformance tests go further: they verify your plugin behaves correctly when orchestrated by the formae agent, which is how it will actually be used. Conformance tests exercise the full resource lifecycle through the same code paths used in production: ``` formae apply --> Create --> Read/Verify | formae apply --> Update --> Read/Verify (optional) | formae apply --> Replace --> Read/Verify (optional) | formae destroy --> Delete --> Verify Gone | Discovery Test ``` ## What you need Conformance tests require only **PKL files**, no Go test code needed. The test framework discovers your PKL files and runs the appropriate lifecycle operations. ``` testdata/ PklProject # PKL project configuration sftp-file.pkl # Base resource definition (Create test) sftp-file-update.pkl # Modified properties (Update test, optional) sftp-file-replace.pkl # Changed immutable properties (Replace test, optional) config/ # Ignored - for shared configs and variables vars.pkl # Shared variables (e.g., test prefixes) target.pkl # Reusable target configuration ``` The `PklProject.deps.json` file is auto-generated when you run the tests, so you don't need to create it manually. ### The config/ folder The `testdata/config/` folder is **ignored** by the conformance test framework. Use it for: * **Shared variables** (`vars.pkl`): common values like test prefixes, stack names * **Target configurations** (`target.pkl`): reusable target definitions * **Helper modules**: any PKL files that support your tests but aren't test cases This keeps your testdata organized without helper files being picked up as failed tests. ## Test lifecycle steps ### Step 1: Create The test framework runs `formae apply` with your base PKL file (e.g., `sftp-file.pkl`). This creates the resource through your plugin's `Create` method. ### Step 2: Read and verify After creation, the framework uses `formae inventory` to read the resource back and compares it to the expected state from `formae eval`. This verifies your `Read` method returns consistent data. ### Step 3: Update (optional) If you provide an update file (e.g., `sftp-file-update.pkl`), the framework runs `formae apply` again with the modified properties. This exercises your plugin's `Update` method. Skip this step if your resource type has only immutable (`createOnly`) properties. ### Step 4: Replace (optional) If you provide a replace file (e.g., `sftp-file-replace.pkl`) that changes a `createOnly` property, the framework verifies that formae correctly deletes and recreates the resource. ### Step 5: Delete The framework runs `formae destroy` to remove the resource, exercising your plugin's `Delete` method. ### Step 6: Discovery Finally, the framework tests resource discovery: 1. Creates a resource directly via your plugin (bypassing formae) 2. Runs discovery to verify formae detects the unmanaged resource 3. Cleans up the resource This verifies your `List` method works correctly for discovery scenarios. ## Test data files ### Base file (`sftp-file.pkl`) The base file defines the initial resource state: ```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}} amends "@formae/forma.pkl" import "@formae/formae.pkl" import "@sftp/sftp.pkl" // Use test run ID from environment for unique resource naming local testRunID = read("env:FORMAE_TEST_RUN_ID") local stackName = "conformance-test-\(testRunID)" forma { new formae.Stack { label = stackName } new formae.Target { label = "sftp-target" config = new sftp.Config { url = "sftp://localhost:2222" } } new sftp.File { label = "test-file" path = "/upload/conformance-test-\(testRunID).txt" content = "Hello from conformance test" permissions = "0644" } } ``` The `FORMAE_TEST_RUN_ID` environment variable is set by the test framework to ensure unique resource names across test runs. Include the test run ID in resource names and use a consistent prefix (like `conformance-test-`). This makes it easy to identify and clean up test resources, especially after interrupted test runs. ### Update file (`sftp-file-update.pkl`) Changes mutable properties only: ```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}} amends "@formae/forma.pkl" import "@formae/formae.pkl" import "@sftp/sftp.pkl" local testRunID = read("env:FORMAE_TEST_RUN_ID") local stackName = "conformance-test-\(testRunID)" forma { new formae.Stack { label = stackName } new formae.Target { label = "sftp-target" config = new sftp.Config { url = "sftp://localhost:2222" } } new sftp.File { label = "test-file" path = "/upload/conformance-test-\(testRunID).txt" // Same (createOnly) content = "Updated content" // Changed (mutable) permissions = "0644" // Same } } ``` ### Replace file (`sftp-file-replace.pkl`) Changes a `createOnly` property, triggering delete + create: ```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}} amends "@formae/forma.pkl" import "@formae/formae.pkl" import "@sftp/sftp.pkl" local testRunID = read("env:FORMAE_TEST_RUN_ID") local stackName = "conformance-test-\(testRunID)" forma { new formae.Stack { label = stackName } new formae.Target { label = "sftp-target" config = new sftp.Config { url = "sftp://localhost:2222" } } new sftp.File { label = "test-file" path = "/upload/conformance-test-\(testRunID)-replaced.txt" // Changed! content = "Content after replacement" permissions = "0644" } } ``` ### PklProject The `testdata/PklProject` file configures PKL dependencies: ```kotlin theme={"languages":{"custom":["/languages/pkl.json"]}} amends "pkl:Project" dependencies { // Reference local plugin schema ["sftp"] = import("../schema/pkl/PklProject") // Formae schema from public registry ["formae"] { uri = "package://hub.platform.engineering/plugins/pkl/schema/pkl/formae/formae@0.77.2-internal" } } ``` The version in the URI is automatically updated when you run conformance tests. ## Running conformance tests Conformance tests consist of two types: **CRUD tests** that verify the resource lifecycle, and **discovery tests** that verify your plugin can find existing resources. By default, both are run together: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} make conformance-test ``` By default, this downloads and tests against the **latest stable release** of formae. ### Filtering tests To run a specific test by name: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} make conformance-test TEST=sftp-file ``` The `TEST` parameter filters tests by name pattern. This is useful when debugging a specific resource type. #### Regex filters Wrap a filter segment in `/…/` delimiters to use a regular expression: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} # Select all chart tests make conformance-test TEST='/.*-chart/' # Select all cluster-scoped resources make conformance-test TEST='/^cluster.*/' # Mix regex and literal filters make conformance-test TEST='/.*-chart/,namespace' ``` Regex patterns match against the **test case name** and **resource type**. An invalid regex causes the test run to fail immediately with a clear error message. Comma separation continues to work: each segment is independently evaluated as either a regex (if wrapped in `/…/`) or a literal (exact match first, then substring fallback). ### Running CRUD or discovery only **CRUD tests** exercise the full resource lifecycle: create a resource, read it back, optionally update it, and delete it. These verify your plugin's Create, Read, Update, and Delete operations work correctly with the formae agent. **Discovery tests** verify that your plugin can find existing resources that weren't created through formae. The test creates a resource directly via your plugin, then runs discovery to confirm formae detects it. To run only CRUD lifecycle tests: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} make conformance-test-crud ``` To run only discovery tests: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} make conformance-test-discovery ``` Both targets support the `TEST` and `VERSION` parameters: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} make conformance-test-crud TEST=sftp-file VERSION=0.80.0 ``` ### Targeting a specific version To test against a specific formae version: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} make conformance-test VERSION=0.80.0 ``` This is useful for: * Testing compatibility with older formae versions * Reproducing issues reported by users on specific versions * Validating your plugin before upgrading formae ### What happens When you run `make conformance-test`: 1. Downloads the specified formae version (or latest) 2. Updates PKL dependencies to match that version 3. Builds and installs your plugin locally 4. Runs `clean-environment.sh` to remove leftover test resources 5. Executes the conformance test suite 6. Cleans up test resources ## Environment variables The conformance test framework reads several environment variables to control behavior: | Variable | Default | Description | | ----------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `FORMAE_TEST_RUN_ID` | *(set by framework)* | Unique ID for test isolation. Use in resource names to avoid collisions. | | `FORMAE_BINARY` | *(unset, auto-download)* | Path to an existing `formae` binary. When unset (or pointing at a missing path), the harness downloads formae via orbital into a temp directory. Set this to skip the download in local dev or pin to a specific build. | | `FORMAE_VERSION` | *(set by framework after install)* | formae version used to pin PKL dependencies (`pkl project resolve`). Auto-set from the downloaded binary; override only when running against a custom build whose `--version` output you need to spoof. | | `FORMAE_TEST_FILTER` | *(unset, run all)* | Substring match against test case names (e.g., `AWS::s3-bucket`) or file paths. Comma-separated for multiple filters (e.g., `s3-bucket,ec2-instance`). | | `FORMAE_TEST_TYPE` | `all` | Which suite to run: `all`, `crud`, or `discovery`. | | `FORMAE_TEST_PARALLEL` | `1` | When set to `>1`, marks subtests as parallel-eligible (must combine with `go test -parallel N`). Leave at default unless you've verified your plugin / target backend tolerates concurrent CRUD. | | `FORMAE_TEST_EXTRA_DISCOVERY_TYPES` | *(unset)* | Comma-separated list of additional resource types to include in the discovery scan, on top of those derived from the test cases. Use for static parent resources that aren't created OOB (e.g., OCI compartments). | | `FORMAE_TEST_TIMEOUT` | `5` | Timeout in minutes for CRUD operations (apply, update, replace, destroy). Increase for slow resources (e.g., `15` for Cloud SQL). | | `FORMAE_TEST_DISCOVERY_TIMEOUT` | `2` | Timeout in minutes for discovery inventory polling. Increase for plugins with many resource types or chart-based tests (e.g., `5`). | | `FORMAE_TEST_OOB_TIMEOUT` | `30` | Timeout in minutes for the OOB Create / OOB Delete plugin RPC waits (used by the discovery test's CreateOOB step and the CRUD test's OOB-delete setup). Raise only if a single Create/Delete RPC to your plugin can legitimately take longer than 30 minutes. *Added in `pkg/plugin-conformance-tests` v0.2.3.* | | `FORMAE_TEST_OOB_DELETE_TIMEOUT` | `2` | Timeout in minutes for Step 24 of the CRUD test, the post-sync inventory-tombstone wait, *after* the plugin's Delete RPC has already returned. Raise if sync to tombstone propagation in your agent setup is slow. *Added in `pkg/plugin-conformance-tests` v0.2.3.* | **`FORMAE_TEST_OOB_TIMEOUT` vs `FORMAE_TEST_OOB_DELETE_TIMEOUT`.** These bound different waits in different phases and are independent: * `FORMAE_TEST_OOB_TIMEOUT` covers the slow part: your plugin's Create/Delete RPC for the OOB resource finishing on the cloud side (e.g. EKS or OVH Kube cluster reaching a terminal state). * `FORMAE_TEST_OOB_DELETE_TIMEOUT` covers the much shorter wait *after* that RPC returns: the agent's sync detecting the OOB deletion and tombstoning the inventory entry. Most plugins won't need to override either. Defaults cover everything we currently exercise on AWS, OVH, and GCP. Example with custom timeouts: ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} make conformance-test TIMEOUT=10 ``` Or set timeouts separately in CI (e.g. for a slow managed-Kubernetes resource): ```yaml theme={"languages":{"custom":["/languages/pkl.json"]}} - name: Run conformance tests env: FORMAE_TEST_DISCOVERY_TIMEOUT: "5" FORMAE_TEST_OOB_TIMEOUT: "45" run: make conformance-test ``` ## Environment cleanup The `scripts/ci/clean-environment.sh` script removes test resources before and after tests. Customize it for your infrastructure. The script should be idempotent (safe to run multiple times) and delete all resources matching the test prefix. For our SFTP plugin, we list files matching the prefix, then delete them individually (wildcards don't work reliably in sftp batch mode): ```bash theme={"languages":{"custom":["/languages/pkl.json"]}} #!/bin/bash set -euo pipefail TEST_PREFIX="${TEST_PREFIX:-formae-plugin-sdk-test-}" SFTP_HOST="${SFTP_HOST:-localhost}" SFTP_PORT="${SFTP_PORT:-2222}" SFTP_DIRECTORY="${SFTP_DIRECTORY:-/upload}" if [[ -z "${SFTP_USERNAME:-}" ]] || [[ -z "${SFTP_PASSWORD:-}" ]]; then echo "Warning: SFTP credentials not set, skipping cleanup" exit 0 fi if ! command -v sshpass &> /dev/null; then echo "Warning: sshpass not found, skipping cleanup" exit 0 fi # List files matching prefix FILES=$(sshpass -p "$SFTP_PASSWORD" sftp -P "$SFTP_PORT" \ -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ "${SFTP_USERNAME}@${SFTP_HOST}" 2>/dev/null </dev/null <