11 - Conformance tests

Now that we’ve built our plugin and verified it works with unit and integration tests, we’re ready to run conformance tests. These tests verify that your plugin correctly integrates with the formae agent, the component that orchestrates resource lifecycle operations in production.

Why conformance tests?

Unit tests verify your plugin logic in isolation. Integration tests verify it works with real infrastructure. Conformance tests go further: they verify your plugin behaves correctly when orchestrated by the formae agent, which is how it will actually be used. Conformance tests exercise the full resource lifecycle through the same code paths used in production:

formae apply  -->  Create  -->  Read/Verify
                      |
formae apply  -->  Update  -->  Read/Verify  (optional)
                      |
formae apply  -->  Replace -->  Read/Verify  (optional)
                      |
formae destroy --> Delete  -->  Verify Gone
                      |
               Discovery Test

What you need

Conformance tests require only PKL files, no Go test code needed. The test framework discovers your PKL files and runs the appropriate lifecycle operations.

testdata/
  PklProject                # PKL project configuration
  sftp-file.pkl             # Base resource definition (Create test)
  sftp-file-update.pkl      # Modified properties (Update test, optional)
  sftp-file-replace.pkl     # Changed immutable properties (Replace test, optional)
  config/                   # Ignored - for shared configs and variables
    vars.pkl                # Shared variables (e.g., test prefixes)
    target.pkl              # Reusable target configuration

The PklProject.deps.json file is auto-generated when you run the tests, so you don’t need to create it manually.

The config/ folder

The testdata/config/ folder is ignored by the conformance test framework. Use it for:

Shared variables (vars.pkl): common values like test prefixes, stack names
Target configurations (target.pkl): reusable target definitions
Helper modules: any PKL files that support your tests but aren’t test cases

This keeps your testdata organized without helper files being picked up as failed tests.

Test lifecycle steps

Step 1: Create

The test framework runs formae apply with your base PKL file (e.g., sftp-file.pkl). This creates the resource through your plugin’s Create method.

Step 2: Read and verify

After creation, the framework uses formae inventory to read the resource back and compares it to the expected state from formae eval. This verifies your Read method returns consistent data.

Step 3: Update (optional)

If you provide an update file (e.g., sftp-file-update.pkl), the framework runs formae apply again with the modified properties. This exercises your plugin’s Update method. Skip this step if your resource type has only immutable (createOnly) properties.

Step 4: Replace (optional)

If you provide a replace file (e.g., sftp-file-replace.pkl) that changes a createOnly property, the framework verifies that formae correctly deletes and recreates the resource.

Step 5: Delete

The framework runs formae destroy to remove the resource, exercising your plugin’s Delete method.

Step 6: Discovery

Finally, the framework tests resource discovery:

Creates a resource directly via your plugin (bypassing formae)
Runs discovery to verify formae detects the unmanaged resource
Cleans up the resource

This verifies your List method works correctly for discovery scenarios.

Test data files

Base file (`sftp-file.pkl`)

The base file defines the initial resource state:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@sftp/sftp.pkl"

// Use test run ID from environment for unique resource naming
local testRunID = read("env:FORMAE_TEST_RUN_ID")
local stackName = "conformance-test-\(testRunID)"

forma {
  new formae.Stack {
    label = stackName
  }

  new formae.Target {
    label = "sftp-target"
    config = new sftp.Config {
      url = "sftp://localhost:2222"
    }
  }

  new sftp.File {
    label = "test-file"
    path = "/upload/conformance-test-\(testRunID).txt"
    content = "Hello from conformance test"
    permissions = "0644"
  }
}

The FORMAE_TEST_RUN_ID environment variable is set by the test framework to ensure unique resource names across test runs.

Include the test run ID in resource names and use a consistent prefix (like conformance-test-). This makes it easy to identify and clean up test resources, especially after interrupted test runs.

Update file (`sftp-file-update.pkl`)

Changes mutable properties only:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@sftp/sftp.pkl"

local testRunID = read("env:FORMAE_TEST_RUN_ID")
local stackName = "conformance-test-\(testRunID)"

forma {
  new formae.Stack {
    label = stackName
  }

  new formae.Target {
    label = "sftp-target"
    config = new sftp.Config {
      url = "sftp://localhost:2222"
    }
  }

  new sftp.File {
    label = "test-file"
    path = "/upload/conformance-test-\(testRunID).txt"  // Same (createOnly)
    content = "Updated content"                          // Changed (mutable)
    permissions = "0644"                                 // Same
  }
}

Replace file (`sftp-file-replace.pkl`)

Changes a createOnly property, triggering delete + create:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@sftp/sftp.pkl"

local testRunID = read("env:FORMAE_TEST_RUN_ID")
local stackName = "conformance-test-\(testRunID)"

forma {
  new formae.Stack {
    label = stackName
  }

  new formae.Target {
    label = "sftp-target"
    config = new sftp.Config {
      url = "sftp://localhost:2222"
    }
  }

  new sftp.File {
    label = "test-file"
    path = "/upload/conformance-test-\(testRunID)-replaced.txt"  // Changed!
    content = "Content after replacement"
    permissions = "0644"
  }
}

PklProject

The testdata/PklProject file configures PKL dependencies:

amends "pkl:Project"

dependencies {
  // Reference local plugin schema
  ["sftp"] = import("../schema/pkl/PklProject")

  // Formae schema from public registry
  ["formae"] {
    uri = "package://hub.platform.engineering/plugins/pkl/schema/pkl/formae/formae@0.77.2-internal"
  }
}

The version in the URI is automatically updated when you run conformance tests.

Running conformance tests

Conformance tests consist of two types: CRUD tests that verify the resource lifecycle, and discovery tests that verify your plugin can find existing resources. By default, both are run together:

make conformance-test

By default, this downloads and tests against the latest stable release of formae.

Filtering tests

To run a specific test by name:

make conformance-test TEST=sftp-file

The TEST parameter filters tests by name pattern. This is useful when debugging a specific resource type.

Regex filters

Wrap a filter segment in /…/ delimiters to use a regular expression:

# Select all chart tests
make conformance-test TEST='/.*-chart/'

# Select all cluster-scoped resources
make conformance-test TEST='/^cluster.*/'

# Mix regex and literal filters
make conformance-test TEST='/.*-chart/,namespace'

Regex patterns match against the test case name and resource type. An invalid regex causes the test run to fail immediately with a clear error message. Comma separation continues to work: each segment is independently evaluated as either a regex (if wrapped in /…/) or a literal (exact match first, then substring fallback).

Running CRUD or discovery only

CRUD tests exercise the full resource lifecycle: create a resource, read it back, optionally update it, and delete it. These verify your plugin’s Create, Read, Update, and Delete operations work correctly with the formae agent. Discovery tests verify that your plugin can find existing resources that weren’t created through formae. The test creates a resource directly via your plugin, then runs discovery to confirm formae detects it. To run only CRUD lifecycle tests:

make conformance-test-crud

To run only discovery tests:

make conformance-test-discovery

Both targets support the TEST and VERSION parameters:

make conformance-test-crud TEST=sftp-file VERSION=0.80.0

Targeting a specific version

To test against a specific formae version:

make conformance-test VERSION=0.80.0

This is useful for:

Testing compatibility with older formae versions
Reproducing issues reported by users on specific versions
Validating your plugin before upgrading formae

What happens

When you run make conformance-test:

Downloads the specified formae version (or latest)
Updates PKL dependencies to match that version
Builds and installs your plugin locally
Runs clean-environment.sh to remove leftover test resources
Executes the conformance test suite
Cleans up test resources

Environment variables

The conformance test framework reads several environment variables to control behavior:

Variable	Default	Description
`FORMAE_TEST_RUN_ID`	(set by framework)	Unique ID for test isolation. Use in resource names to avoid collisions.
`FORMAE_BINARY`	(unset, auto-download)	Path to an existing `formae` binary. When unset (or pointing at a missing path), the harness downloads formae via orbital into a temp directory. Set this to skip the download in local dev or pin to a specific build.
`FORMAE_VERSION`	(set by framework after install)	formae version used to pin PKL dependencies (`pkl project resolve`). Auto-set from the downloaded binary; override only when running against a custom build whose `--version` output you need to spoof.
`FORMAE_TEST_FILTER`	(unset, run all)	Substring match against test case names (e.g., `AWS::s3-bucket`) or file paths. Comma-separated for multiple filters (e.g., `s3-bucket,ec2-instance`).
`FORMAE_TEST_TYPE`	`all`	Which suite to run: `all`, `crud`, or `discovery`.
`FORMAE_TEST_PARALLEL`	`1`	When set to `>1`, marks subtests as parallel-eligible (must combine with `go test -parallel N`). Leave at default unless you’ve verified your plugin / target backend tolerates concurrent CRUD.
`FORMAE_TEST_EXTRA_DISCOVERY_TYPES`	(unset)	Comma-separated list of additional resource types to include in the discovery scan, on top of those derived from the test cases. Use for static parent resources that aren’t created OOB (e.g., OCI compartments).
`FORMAE_TEST_TIMEOUT`	`5`	Timeout in minutes for CRUD operations (apply, update, replace, destroy). Increase for slow resources (e.g., `15` for Cloud SQL).
`FORMAE_TEST_DISCOVERY_TIMEOUT`	`2`	Timeout in minutes for discovery inventory polling. Increase for plugins with many resource types or chart-based tests (e.g., `5`).
`FORMAE_TEST_OOB_TIMEOUT`	`30`	Timeout in minutes for the OOB Create / OOB Delete plugin RPC waits (used by the discovery test’s CreateOOB step and the CRUD test’s OOB-delete setup). Raise only if a single Create/Delete RPC to your plugin can legitimately take longer than 30 minutes. Added in `pkg/plugin-conformance-tests` v0.2.3.
`FORMAE_TEST_OOB_DELETE_TIMEOUT`	`2`	Timeout in minutes for Step 24 of the CRUD test, the post-sync inventory-tombstone wait, after the plugin’s Delete RPC has already returned. Raise if sync to tombstone propagation in your agent setup is slow. Added in `pkg/plugin-conformance-tests` v0.2.3.

FORMAE_TEST_OOB_TIMEOUT vs FORMAE_TEST_OOB_DELETE_TIMEOUT. These bound different waits in different phases and are independent:

FORMAE_TEST_OOB_TIMEOUT covers the slow part: your plugin’s Create/Delete RPC for the OOB resource finishing on the cloud side (e.g. EKS or OVH Kube cluster reaching a terminal state).
FORMAE_TEST_OOB_DELETE_TIMEOUT covers the much shorter wait after that RPC returns: the agent’s sync detecting the OOB deletion and tombstoning the inventory entry.

Most plugins won’t need to override either. Defaults cover everything we currently exercise on AWS, OVH, and GCP. Example with custom timeouts:

make conformance-test TIMEOUT=10

Or set timeouts separately in CI (e.g. for a slow managed-Kubernetes resource):

- name: Run conformance tests
  env:
    FORMAE_TEST_DISCOVERY_TIMEOUT: "5"
    FORMAE_TEST_OOB_TIMEOUT: "45"
  run: make conformance-test

Environment cleanup

The scripts/ci/clean-environment.sh script removes test resources before and after tests. Customize it for your infrastructure. The script should be idempotent (safe to run multiple times) and delete all resources matching the test prefix. For our SFTP plugin, we list files matching the prefix, then delete them individually (wildcards don’t work reliably in sftp batch mode):

#!/bin/bash
set -euo pipefail

TEST_PREFIX="${TEST_PREFIX:-formae-plugin-sdk-test-}"
SFTP_HOST="${SFTP_HOST:-localhost}"
SFTP_PORT="${SFTP_PORT:-2222}"
SFTP_DIRECTORY="${SFTP_DIRECTORY:-/upload}"

if [[ -z "${SFTP_USERNAME:-}" ]] || [[ -z "${SFTP_PASSWORD:-}" ]]; then
    echo "Warning: SFTP credentials not set, skipping cleanup"
    exit 0
fi

if ! command -v sshpass &> /dev/null; then
    echo "Warning: sshpass not found, skipping cleanup"
    exit 0
fi

# List files matching prefix
FILES=$(sshpass -p "$SFTP_PASSWORD" sftp -P "$SFTP_PORT" \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    "${SFTP_USERNAME}@${SFTP_HOST}" 2>/dev/null <<EOF | grep "^${TEST_PREFIX}" || true
cd ${SFTP_DIRECTORY}
ls -1
EOF
)

if [[ -z "$FILES" ]]; then
    echo "No test files found"
    exit 0
fi

# Build and execute delete commands
DELETE_COMMANDS="cd ${SFTP_DIRECTORY}"
while IFS= read -r file; do
    DELETE_COMMANDS="${DELETE_COMMANDS}
rm \"$file\""
done <<< "$FILES"

sshpass -p "$SFTP_PASSWORD" sftp -P "$SFTP_PORT" \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    "${SFTP_USERNAME}@${SFTP_HOST}" 2>/dev/null <<EOF || true
${DELETE_COMMANDS}
EOF

echo "Cleanup complete"

Verify

With the test data files in place, run the conformance tests:

SFTP_USERNAME=testuser SFTP_PASSWORD=testpass make conformance-test

You should see output showing each test phase:

Using formae binary: /tmp/formae-conformance-xxx/formae/bin/formae
formae version 0.77.0

Updating PKL dependencies for formae version 0.77.0...
PKL dependencies resolved successfully

Running conformance tests...
=== RUN   TestConformance
=== RUN   TestConformance/CRUD/sftp-file
    runner.go:xxx: Step 1: Creating resource...
    runner.go:xxx: Step 2: Reading and verifying resource...
    runner.go:xxx: Step 3: Updating resource...
    runner.go:xxx: Step 4: Verifying update...
    runner.go:xxx: Step 5: Replacing resource...
    runner.go:xxx: Step 6: Verifying replacement...
    runner.go:xxx: Step 7: Deleting resource...
    runner.go:xxx: Step 8: Verifying deletion...
--- PASS: TestConformance/CRUD/sftp-file (45.23s)
=== RUN   TestConformance/Discovery/sftp-file
    runner.go:xxx: Step 1: Creating resource out-of-band via plugin...
    runner.go:xxx: Step 2: Running discovery...
    runner.go:xxx: Step 3: Verifying resource was discovered...
--- PASS: TestConformance/Discovery/sftp-file (12.34s)
--- PASS: TestConformance (57.57s)
PASS

Next: 12 - CI Setup - Set up continuous integration for your plugin

Get started

Guides

Concepts

Reference

11 - Conformance tests

Why conformance tests?

What you need

The config/ folder

Test lifecycle steps

Step 1: Create

Step 2: Read and verify

Step 3: Update (optional)

Step 4: Replace (optional)

Step 5: Delete

Step 6: Discovery

Test data files

Base file (`sftp-file.pkl`)

Update file (`sftp-file-update.pkl`)

Replace file (`sftp-file-replace.pkl`)

PklProject

Running conformance tests

Filtering tests

Regex filters

Running CRUD or discovery only

Targeting a specific version

What happens

Environment variables

Environment cleanup

Verify

​Why conformance tests?

​What you need

​The config/ folder

​Test lifecycle steps

​Step 1: Create

​Step 2: Read and verify

​Step 3: Update (optional)

​Step 4: Replace (optional)

​Step 5: Delete

​Step 6: Discovery

​Test data files

​Base file (sftp-file.pkl)

​Update file (sftp-file-update.pkl)

​Replace file (sftp-file-replace.pkl)

​PklProject

​Running conformance tests

​Filtering tests

​Regex filters

​Running CRUD or discovery only

​Targeting a specific version

​What happens

​Environment variables

​Environment cleanup

​Verify

Why conformance tests?

What you need

The config/ folder

Test lifecycle steps

Step 1: Create

Step 2: Read and verify

Step 3: Update (optional)

Step 4: Replace (optional)

Step 5: Delete

Step 6: Discovery

Test data files

Base file (`sftp-file.pkl`)

Update file (`sftp-file-update.pkl`)

Replace file (`sftp-file-replace.pkl`)

PklProject

Running conformance tests

Filtering tests

Regex filters

Running CRUD or discovery only

Targeting a specific version

What happens

Environment variables

Environment cleanup

Verify