Skip to main content
Kubernetes annotations and labels routinely contain dots: app.kubernetes.io/name, objectset.rio.cattle.io/applied. Agents before 0.89.0 read those dots as structure when writing a resource back to the datastore, so a key like that ended up stored twice: once as the literal key the provider reported, and once as a nested tree spelling it out. Upgrading stops it happening. Rows already written that way do not fix themselves, because nothing in the stored data says which of the two copies was meant. This guide shows how to recognise the shape, what the agent repairs on its own, and what to do about the rest.

​
Recognise the shape

Run formae extract on an affected resource, or look at the stored properties. The tell is a key rendered twice:
Both copies carry the same value. The nested one stops at io/applied rather than splitting further, because only the dots were read as structure and the slash was not. Extracted Pkl from such a resource does not evaluate: the same field is rendered twice.

​
What the agent repairs on its own

On its first start after the upgrade, the agent repairs discovered resources that are not yet under management. It does this once, before it begins any other work, and it records what it did so it never runs twice over the same target. The repair is not an edit. The agent forgets the affected rows and lets the next discovery cycle read them fresh from the cloud, which is why it is safe: nothing in your infrastructure is touched, and nothing formae knows is rewritten from a guess. Discovered resources hold no declarations of yours, so there is nothing to lose by re-reading them. Two consequences are worth knowing about. Repaired resources come back with new identities. A re-read resource is a new row, and its label is derived again from its cloud tags the way any freshly discovered resource’s is. If you had come to rely on a particular discovered label, check it afterwards. Repair happens per target, not per resource. Discovered resources reference their parents, so the agent re-reads a target’s discovered resources together rather than leaving some pointing at rows that no longer exist.

​
What the agent deliberately skips

The agent logs each case it declines, so a search of the startup log for dotted-key tells you what is left. Resources you have referenced from a forma. If one of your declared resources, or a target’s configuration, points at a discovered resource on the affected target, the agent leaves the whole target alone. Re-reading would give those resources new identities and leave your reference pointing at nothing. Choosing what to do instead is yours: bring the resource under management, or repoint the reference, and then re-read the target yourself. Targets with a command still in flight. If a command against the target had not finished when the agent stopped, the agent defers that target and repairs it on a later start, once the command has completed or been cancelled. Nothing is needed from you. Managed resources. Resources under management carry your declarations, so the agent never rewrites them on its own; see below. Datastores the agent cannot lock. The repair needs a lock so two agents sharing a datastore cannot run it at once. The Aurora Data API backend cannot hold one, so the agent logs that the repair is unsupported and skips it. Use the manual route below.

​
Repair a managed resource

A managed resource’s properties are yours, so formae will not guess which copy you meant. Narrow the field with a query against your datastore, then confirm by reading each candidate.
Read <> FALSE as <> 0 on SQLite and SQL Server, where the column is an integer rather than a boolean. The pattern looks for the Kubernetes annotation and label shape, a key holding both a dot and a slash, which is where this turns up in practice. It matches on values as well as keys, so treat the result as a candidate list rather than a verdict. Drop the / from the pattern to widen it to any dotted key, at the cost of many more false matches. Confirm each candidate by reading its properties, and look for the shape above: a dotted key beside a nested tree carrying the same value. A candidate with only one of the two is fine and needs nothing. To repair one, remove the nested copy from the forma that declares the resource, keeping the literal dotted key, and apply. Simulate first and confirm the plan touches only the field you meant:
If the resource is one formae discovered rather than one you wrote, you can also stop managing it and let discovery read it again, which produces a clean row the same way the automatic repair does.

​
Before you upgrade

A few conditions have to hold for the repair to be sound, and all of them are already how formae expects to be run. One version at a time against one datastore. The agent records that a target has been repaired. An older agent writing to the same datastore afterwards would mint the duplication again, under a marker saying it had been dealt with. Stop the old agent before starting the new one. Restore whole snapshots, not individual tables. A full snapshot restore is safe: the repair’s records restore alongside the data, and the repair simply runs again. Restoring the resource tables on their own, without the record of what was repaired, leaves the agent believing work is done that has been rolled back. On SQLite, take that snapshot with the backup API or a copy made while nothing is writing. Copying the database file alone while the agent is running does not capture a consistent state. Downgrading re-opens the problem. An older agent running against a repaired datastore starts writing duplicated keys again. Those rows show up in the query above, and are repaired the same way.