Skip to content

11 - Conformance Tests

Now that we've built our plugin and verified it works with unit and integration tests, we're ready to run conformance tests. These tests verify that your plugin correctly integrates with the formae agent - the component that orchestrates resource lifecycle operations in production.

Why Conformance Tests?

Unit tests verify your plugin logic in isolation. Integration tests verify it works with real infrastructure. Conformance tests go further: they verify your plugin behaves correctly when orchestrated by the formae agent, which is how it will actually be used.

Conformance tests exercise the full resource lifecycle through the same code paths used in production:

formae apply  -->  Create  -->  Read/Verify
                      |
formae apply  -->  Update  -->  Read/Verify  (optional)
                      |
formae apply  -->  Replace -->  Read/Verify  (optional)
                      |
formae destroy --> Delete  -->  Verify Gone
                      |
               Discovery Test

What You Need

Conformance tests require only PKL files - no Go test code needed. The test framework discovers your PKL files and runs the appropriate lifecycle operations.

testdata/
  PklProject                # PKL project configuration
  sftp-file.pkl             # Base resource definition (Create test)
  sftp-file-update.pkl      # Modified properties (Update test, optional)
  sftp-file-replace.pkl     # Changed immutable properties (Replace test, optional)

The PklProject.deps.json file is auto-generated when you run the tests - you don't need to create it manually.

Test Lifecycle Steps

Step 1: Create

The test framework runs formae apply with your base PKL file (e.g., sftp-file.pkl). This creates the resource through your plugin's Create method.

Step 2: Read and Verify

After creation, the framework uses formae inventory to read the resource back and compares it to the expected state from formae eval. This verifies your Read method returns consistent data.

Step 3: Update (Optional)

If you provide an update file (e.g., sftp-file-update.pkl), the framework runs formae apply again with the modified properties. This exercises your plugin's Update method.

Skip this step if your resource type has only immutable (createOnly) properties.

Step 4: Replace (Optional)

If you provide a replace file (e.g., sftp-file-replace.pkl) that changes a createOnly property, the framework verifies that formae correctly deletes and recreates the resource.

Step 5: Delete

The framework runs formae destroy to remove the resource, exercising your plugin's Delete method.

Step 6: Discovery

Finally, the framework tests resource discovery:

  1. Creates a resource directly via your plugin (bypassing formae)
  2. Runs discovery to verify formae detects the unmanaged resource
  3. Cleans up the resource

This verifies your List method works correctly for discovery scenarios.

Test Data Files

Base File (sftp-file.pkl)

The base file defines the initial resource state:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@sftp/sftp.pkl"

// Use test run ID from environment for unique resource naming
local testRunID = read("env:FORMAE_TEST_RUN_ID")
local stackName = "conformance-test-\(testRunID)"

forma {
  new formae.Stack {
    label = stackName
  }

  new formae.Target {
    label = "sftp-target"
    config = new sftp.Config {
      url = "sftp://localhost:2222"
    }
  }

  new sftp.File {
    label = "test-file"
    path = "/upload/conformance-test-\(testRunID).txt"
    content = "Hello from conformance test"
    permissions = "0644"
  }
}

The FORMAE_TEST_RUN_ID environment variable is set by the test framework to ensure unique resource names across test runs.

Good practice: Include the test run ID in resource names and use a consistent prefix (like conformance-test-). This makes it easy to identify and clean up test resources, especially after interrupted test runs.

Update File (sftp-file-update.pkl)

Changes mutable properties only:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@sftp/sftp.pkl"

local testRunID = read("env:FORMAE_TEST_RUN_ID")
local stackName = "conformance-test-\(testRunID)"

forma {
  new formae.Stack {
    label = stackName
  }

  new formae.Target {
    label = "sftp-target"
    config = new sftp.Config {
      url = "sftp://localhost:2222"
    }
  }

  new sftp.File {
    label = "test-file"
    path = "/upload/conformance-test-\(testRunID).txt"  // Same (createOnly)
    content = "Updated content"                          // Changed (mutable)
    permissions = "0644"                                 // Same
  }
}

Replace File (sftp-file-replace.pkl)

Changes a createOnly property, triggering delete + create:

amends "@formae/forma.pkl"
import "@formae/formae.pkl"
import "@sftp/sftp.pkl"

local testRunID = read("env:FORMAE_TEST_RUN_ID")
local stackName = "conformance-test-\(testRunID)"

forma {
  new formae.Stack {
    label = stackName
  }

  new formae.Target {
    label = "sftp-target"
    config = new sftp.Config {
      url = "sftp://localhost:2222"
    }
  }

  new sftp.File {
    label = "test-file"
    path = "/upload/conformance-test-\(testRunID)-replaced.txt"  // Changed!
    content = "Content after replacement"
    permissions = "0644"
  }
}

PklProject

The testdata/PklProject file configures PKL dependencies:

amends "pkl:Project"

dependencies {
  // Reference local plugin schema
  ["sftp"] = import("../schema/pkl/PklProject")

  // Formae schema from public registry
  ["formae"] {
    uri = "package://hub.platform.engineering/plugins/pkl/schema/pkl/formae/formae@0.77.2-internal"
  }
}

The version in the URI is automatically updated when you run conformance tests.

Running Conformance Tests

Conformance tests consist of two types: CRUD tests that verify the resource lifecycle, and discovery tests that verify your plugin can find existing resources. By default, both are run together:

make conformance-test

By default, this downloads and tests against the latest stable release of formae.

Filtering Tests

To run a specific test by name:

make conformance-test TEST=sftp-file

The TEST parameter filters tests by name pattern. This is useful when debugging a specific resource type.

Running CRUD or Discovery Only

CRUD tests exercise the full resource lifecycle: create a resource, read it back, optionally update it, and delete it. These verify your plugin's Create, Read, Update, and Delete operations work correctly with the formae agent.

Discovery tests verify that your plugin can find existing resources that weren't created through formae. The test creates a resource directly via your plugin, then runs discovery to confirm formae detects it.

To run only CRUD lifecycle tests:

make conformance-test-crud

To run only discovery tests:

make conformance-test-discovery

Both targets support the TEST and VERSION parameters:

make conformance-test-crud TEST=sftp-file VERSION=0.80.0

Targeting a Specific Version

To test against a specific formae version:

make conformance-test VERSION=0.80.0

This is useful for:

  • Testing compatibility with older formae versions
  • Reproducing issues reported by users on specific versions
  • Validating your plugin before a formae upgrade

What Happens

When you run make conformance-test:

  1. Downloads the specified formae version (or latest)
  2. Updates PKL dependencies to match that version
  3. Builds and installs your plugin locally
  4. Runs clean-environment.sh to remove leftover test resources
  5. Executes the conformance test suite
  6. Cleans up test resources

Environment Cleanup

The scripts/ci/clean-environment.sh script removes test resources before and after tests. Customize it for your infrastructure. The script should be idempotent (safe to run multiple times) and delete all resources matching the test prefix.

For our SFTP plugin, we list files matching the prefix, then delete them individually (wildcards don't work reliably in sftp batch mode):

#!/bin/bash
set -euo pipefail

TEST_PREFIX="${TEST_PREFIX:-formae-plugin-sdk-test-}"
SFTP_HOST="${SFTP_HOST:-localhost}"
SFTP_PORT="${SFTP_PORT:-2222}"
SFTP_DIRECTORY="${SFTP_DIRECTORY:-/upload}"

if [[ -z "${SFTP_USERNAME:-}" ]] || [[ -z "${SFTP_PASSWORD:-}" ]]; then
    echo "Warning: SFTP credentials not set, skipping cleanup"
    exit 0
fi

if ! command -v sshpass &> /dev/null; then
    echo "Warning: sshpass not found, skipping cleanup"
    exit 0
fi

# List files matching prefix
FILES=$(sshpass -p "$SFTP_PASSWORD" sftp -P "$SFTP_PORT" \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    "${SFTP_USERNAME}@${SFTP_HOST}" 2>/dev/null <<EOF | grep "^${TEST_PREFIX}" || true
cd ${SFTP_DIRECTORY}
ls -1
EOF
)

if [[ -z "$FILES" ]]; then
    echo "No test files found"
    exit 0
fi

# Build and execute delete commands
DELETE_COMMANDS="cd ${SFTP_DIRECTORY}"
while IFS= read -r file; do
    DELETE_COMMANDS="${DELETE_COMMANDS}
rm \"$file\""
done <<< "$FILES"

sshpass -p "$SFTP_PASSWORD" sftp -P "$SFTP_PORT" \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    "${SFTP_USERNAME}@${SFTP_HOST}" 2>/dev/null <<EOF || true
${DELETE_COMMANDS}
EOF

echo "Cleanup complete"

Verify

With the test data files in place, run the conformance tests:

SFTP_USERNAME=testuser SFTP_PASSWORD=testpass make conformance-test

You should see output showing each test phase:

Using formae binary: /tmp/formae-conformance-xxx/formae/bin/formae
formae version 0.77.0

Updating PKL dependencies for formae version 0.77.0...
PKL dependencies resolved successfully

Running conformance tests...
=== RUN   TestConformance
=== RUN   TestConformance/CRUD/sftp-file
    runner.go:xxx: Step 1: Creating resource...
    runner.go:xxx: Step 2: Reading and verifying resource...
    runner.go:xxx: Step 3: Updating resource...
    runner.go:xxx: Step 4: Verifying update...
    runner.go:xxx: Step 5: Replacing resource...
    runner.go:xxx: Step 6: Verifying replacement...
    runner.go:xxx: Step 7: Deleting resource...
    runner.go:xxx: Step 8: Verifying deletion...
--- PASS: TestConformance/CRUD/sftp-file (45.23s)
=== RUN   TestConformance/Discovery/sftp-file
    runner.go:xxx: Step 1: Creating resource out-of-band via plugin...
    runner.go:xxx: Step 2: Running discovery...
    runner.go:xxx: Step 3: Verifying resource was discovered...
--- PASS: TestConformance/Discovery/sftp-file (12.34s)
--- PASS: TestConformance (57.57s)
PASS

Next: 12 - CI Setup - Setting up continuous integration