Reconcile, Don't Command

How your control plane changes the data plane is a trust decision on par with how it connects. A control plane that can run arbitrary commands inside customer accounts holds unbounded capability: every remote code execution bug in your stack becomes remote code execution in every customer account you operate. Narrow the channel to declarative intent. The control plane transmits desired state (this Helm release, this node pool, this secret reference), and an agent inside the customer account reconciles reality toward that state using its own bounded, local permissions.

Define the interaction as a typed contract with three actions: Apply, Describe, and Delete. The control plane submits an entity's desired state, then polls its sync status until the agent reports convergence. No action runs a script, so the worst thing the control plane can request is a state the agent already knows how to reach, and the agent's IAM ceiling (Commandment V) bounds even that. A compromised control plane can submit bad desired state, and your reviews should assume it will; it cannot open a shell, run exfiltration commands, or grant itself new capability inside the customer account.

Build the orchestration above this contract for failure as the normal case. Run provisioning, upgrades, and teardown as durable workflows with explicit retry policies, backoff, timeouts, compensation steps, and a defined class of non-retryable errors that halt the workflow for inspection. Engineer idempotency from the start, so a retried step converges on the resource that already exists and creates no duplicate. Add concurrency guards that reject conflicting mutations on the same entity, so two workflows can never race each other into a broken state.

Track every resource the platform creates in a per-instance inventory, and treat that inventory as the authority for cleanup. Deprovisioning then deletes exactly the set of resources that were created, in the right order, and leaves nothing orphaned in an account you do not own. Best-effort deletion by tag scan or naming convention will eventually miss something, and what it misses keeps accruing charges in your customer's account.

Reconciliation also makes the data plane self-healing without new privileges. Dead processes restart, failed nodes are replaced, and a cell that drifts from desired state converges back to it, all executed locally by the agent against its local view. The control plane learns outcomes through status reports. Repair therefore requires no emergency access path, no break-glass credential, and no human shelling into a customer account, which means those paths never need to exist and never need to be audited, disclosed, or defended. This architecture has the added advantage of reconciling existing state without any control plane intervention.