# Runtime Safety

Every workflow node runs as an Action Scheduler job. That gives Flowcrafter durability — a run picks up where it left off after a crash or a deploy — but it also means your handler can be called **more than once** for the same node. If your node charges a card, sends an email, or writes to a remote system, running twice can't quietly turn into two charges.

![Workflow activation controls that validate changes before production](/documentation/flowcrafter/screenshots/workflow-activation-unsaved-changes.png)

This page is the contract for third-party node handlers: what Flowcrafter guarantees, what you have to guarantee, and the concrete APIs that get you there. If you haven't read [Extending Flowcrafter](/documentation/flowcrafter/extending/overview/) yet, start there — this page assumes you know how a node handler is registered and what `execute()` returns.

:::note[At a glance]
- A node can execute more than once, so remote writes need a stable identity and a recoverable result.
- Flowcrafter retries run-level failures (with backoff); loop-body failures follow the loop's Continue or Stop policy instead.
- Keep one handler call short — it has to finish inside its Action Scheduler callback. Split big jobs into resumable work.
- Use WordPress's safe HTTP functions, validate resolved JSON before you send it, and decide whether builder tests may perform live side effects.
:::

## Know the boundary

Flowcrafter and your integration each own a side of the safety line. Flowcrafter protects the *workflow*; you protect the *side effect*.

| Flowcrafter protects | Your integration protects |
|---|---|
| Workflow state and downstream dispatch intent survive queue failures. | A remote charge, email, order, or update can safely replay. |
| A completed node cannot be claimed as fresh work again. | A crash after the remote service succeeds but before local completion does not create a duplicate. |
| Run-level retry timing and retry limits. | Which failures are temporary, which are permanent, and what a retry should return. |
| Loop progress between completed items. | Each loop-body side effect and any long-running producer request. |

Flowcrafter gives you **at-least-once execution** around failure windows: work may run again, but it shouldn't be lost silently. It can't make a third-party API exactly-once for you — that part is yours.

Run inspection exposes the node and dispatch identities behind a real execution. This completed sandbox run's report is actual command output, rendered as text.

![The run inspector reports each node and its delivered dispatch](/documentation/flowcrafter/screenshots/guide-cli-inspect.png)

## Make side effects replayable

The safe pattern for any external write is the same every time. Build a stable identity for the operation, check whether that identity already finished, and only then call the remote service:

1. Build a stable operation identity.
2. If that identity already completed, return the saved result instead of calling again.
3. Claim the operation before calling the remote service.
4. Send the same identity as the provider's idempotency key when it supports one.
5. Save the successful result *before* you return node success.
6. Release or expire incomplete claims so a retry can recover.

### Where the identity comes from

At run scope, the identity usually includes the run ID and node ID — both stable across retries of the same node. The run ID is on `$run->id`; `{{run.id}}` and `{{run.key}}` are also exposed through the runtime context.

Inside a loop, add the loop node ID and the iteration index so each item gets its own identity. Those live in the runtime context under `loop`, keyed by the loop node's ID:

```php
$ctx  = $run->get_runtime_context();
$loop = $ctx['loop'][ $loop_node_id ] ?? null;
// $loop === [ 'index' => 0, 'number' => 1, 'count' => 5, 'remaining' => 4 ]

$identity = sprintf( 'run:%d|node:%s', $run->id, $node['id'] );
if ( $loop !== null ) {
    $identity .= sprintf( '|loop:%s|i:%d', $loop_node_id, $loop['index'] );
}
```

For builder tests that perform real work, base the identity on a fingerprint of the *fully resolved* operation — destination, method, behavior-changing headers, payload, and any side-effect settings — rather than the run/node IDs. A Quick Test re-run reuses the same run, so run/node IDs alone won't tell two distinct test payloads apart. (More on tests below.)

:::caution
**A local "pending" row is not enough to guarantee safety.** If saving the result fails *after* the provider already accepted the request, the next attempt has to be able to recover. Use a provider idempotency key, make claims expire or release them on local failure, and stop an expired worker from overwriting a newer worker's result. If the provider offers no idempotency or reconciliation API, document that duplicate side effects remain possible — don't pretend they can't happen.
:::

<details>
<summary>Durable claim details</summary>

- Give each claim an owner token. Completion, failure, and release writes should all require that token *and* a pending status, so a stale worker can't clobber a fresh one.
- Check every persistence result. A failed save is a *retryable storage error*, not a successful remote result and not a permanent failure.
- Keep saved output small, and define retention for anything that might hold personal or sensitive data.
- Don't reject literal text just because it contains `{{`. Resolve documented flow tags, then validate the *resolved* value for the target field.

</details>

## Choose the right identity for the guarantee

The identity you build decides *what* you're protected against. Run and node IDs are born fresh on every run — a retry keeps them, but a full workflow rerun (the trigger fires again, or someone reopens the run over the same data) gets a brand-new run ID. So run/node identity guards duplicates *inside one run*; it does nothing across reruns. If the same business object must never be created twice no matter how many times the workflow runs, the identity has to come from the *data*, not the run.

| Requirement | Build identity from |
|---|---|
| Repeat the action intentionally on every run | Run ID + node ID (plus loop node ID and index inside a loop) |
| Never create the same business object twice across reruns | A provider/business key such as an order ID, plus the operation kind |
| Allow settings edits after a failure | A stable business key, plus a separately stored fingerprint of the resolved operation |

The **fingerprint** covers the resolved destination, method, behavior-changing headers, and payload — store just a hash when those values hold secrets. If the same business key ever shows up with a *different* fingerprint, that's a settings edit after a failure, not a safe replay: fail with an idempotency conflict instead of sending changed work under an old key.

Inside a loop, derive the business identity from stable item data — not the run and iteration index alone — whenever a full rerun must not duplicate the remote effect. If the item has no stable key you can trust, block the node inside loops or clearly warn about loop placement in the UI and docs.

## Resolve the request that actually parked

A custom parking node must carry both `scope_key` and `park_id` back to `Engine::resolve_parked_node( $run_id, $node_id, $event_id, $next_nodes, $scope_key, $park_id, $output )`. The executor stores them in `park_data`; a modern callback without them is rejected rather than guessing which visit it meant.

If you send a signed callback link **inside the handler**, create the identity first with `Park_Occurrence::create( $run, $node['id'] )`, sign both fields into the link, and include the same fields in the returned `park_data`. Otherwise the executor creates the identity when it parks. Reuse it for reminders, not a new UUID. The shared resolver logs output in the correct scope before publishing downstream work.

Legacy callbacks without identity work only for pre-existing, empty-scope requests without a UUID. They never resolve a new Go To pass or a retried modern request. Parking nodes still implement `Pauses_Run` and remain forbidden inside a For Each body.

## Reconcile after an ambiguous attempt

There's one window a local claim can't close on its own: the worker dies *after* the provider accepts the request but *before* you save the success result. On the next attempt you can't tell "never sent" apart from "sent, lost the receipt." Guessing wrong either drops the work or duplicates it.

- Release a claim as fresh-and-unsent **only** when the provider definitely couldn't have accepted it — say, a connection refused before any bytes left.
- After a timeout, a disconnect, or any other ambiguous outcome, persist a *needs-reconciliation* marker on the claim instead of releasing it.
- When a later attempt sees that marker, reconcile first: look the object up by business key, or hit the provider's idempotency/lookup API, *before* sending anything. If you can't prove the outcome either way, stop for manual recovery rather than risk a duplicate.

If the provider supports idempotency keys, resending with the same key does this reconciliation for you. If it doesn't, that lookup is the only thing standing between an ambiguous crash and a double charge.

## Classify failures

How you fail decides whether Flowcrafter tries again. Return a `WP_Error` and Flowcrafter treats it as **retryable by default** — it schedules a backoff retry until the node's retry budget runs out. To fail immediately and permanently, put `[ 'retryable' => false ]` in the error data:

```php
// Temporary — let Flowcrafter retry with backoff.
return new \WP_Error( 'gateway_timeout', 'Payment gateway timed out.' );

// Permanent — no point retrying a missing credential.
return new \WP_Error(
    'missing_api_key',
    __( 'No API key configured for this node.', 'flowcrafter' ),
    [ 'retryable' => false ]
);
```

Run-scope retries use exponential backoff with jitter. A node's `max_retries` defaults to **3** and is capped at **8**; the total execution ceiling is `1 + max_retries`. A zero or exhausted budget fails right away, even for a retryable error.

**Loop-body failures work differently.** A node running inside a [For Each](/documentation/flowcrafter/nodes/flow/for-each/) iteration does *not* get automatic backoff retries — the loop follows its own Continue or Stop policy instead. Continue records the failed item and moves on; Stop fails the run. Design loop-body side effects so that "run this item again later" is safe on its own, because a retry there means re-running the whole item, not resuming mid-handler.

| Failure | Retry? |
|---|---|
| Timeout, connection failure, rate limit, temporary 5xx | Yes |
| Local result-storage failure | Yes — after making the operation claim recoverable |
| Invalid settings, missing credentials, malformed resolved JSON | No (`retryable => false`) |
| Provider rejects the request as invalid | Usually no |

## Keep work inside the callback budget

One handler call has to fit comfortably inside its Action Scheduler callback. The trap is pagination where every page can wait out a full HTTP timeout — bound the *worst-case time*, not just the page count. Your `maximum pages × per-request timeout` has to leave room for parsing and persistence inside the callback.

Ordinary nodes have no "continue later" result. They must finish within that bound, fail clearly, or report explicit truncation when a page ceiling leaves more data behind. For durable multi-callback work, don't schedule Action Scheduler yourself from a handler — model the items through the core **Loop** node, which is built to span callbacks.

The For Each node checks elapsed time between completed items, saves its next index and counters, then continues in a later callback. That protects *loop progress* — it does not rescue a single slow body action or a producer that fetches every page before saving anything. Keep each iteration's body short, and have producers save incrementally.

## Send HTTP safely

Flowcrafter's own [HTTP Request](/documentation/flowcrafter/nodes/actions/http-request/) node routes requests through `wp_safe_remote_request()` — WordPress validates redirect targets against SSRF-prone hosts, not just the first URL. Follow the same rules in your own nodes:

- **Use `wp_safe_remote_request()`** (or its safe convenience wrappers) so redirects are validated too, not only the initial host.
- **"Safe" is not "HTTPS only."** If credentials or payloads require TLS, check that the resolved *initial* URL uses `https`, then either set `redirection => 0` and handle redirects yourself, or reject every non-HTTPS destination before resending.
- **Resolve flow tag JSON templates before decoding, and validate the decoded shape** before any external call. Don't send a body you haven't confirmed is well-formed.
- **Set a timeout below the callback budget.** The core node defaults to 30s and clamps the requested timeout to a queue-safe ceiling — the `flowcrafter_node_http_request_node_max_timeout` filter tunes that ceiling.
- **Set `limit_response_size`** when a response has a size ceiling. Checking `strlen()` after the download has already happened does nothing to protect worker memory.
- **Classify retryable statuses (429, temporary 5xx) before you require a JSON body.** Error pages are often HTML or empty — don't let a temporary outage decode to a permanent JSON error.
- **Treat credentials and unneeded response data as secrets.** Keep them out of node output, logs, and long-lived idempotency storage.

If you're extending the core HTTP Request node rather than writing your own, the `flowcrafter_node_http_request_node_request` filter lets you rewrite the URL, headers, method, or body before the request goes out (return `null` to keep the node's own request, an array to override it, or a `WP_Error` to fail the node). See [Hooks](/documentation/flowcrafter/hooks/) for the full list.

## Protect builder tests

Quick Test runs a node through its normal `execute()` — webhooks fire, notifications send, conditions evaluate. That's usually what you want. But if testing a node should *not* send an email, charge a card, or create a remote record, implement the `Has_Test_Mode_Override` contract. When a handler implements it, Quick Test calls `execute_in_test_mode()` instead of `execute()`:

```php
use Flowcrafter\Nodes\Contracts\Has_Test_Mode_Override;
use Flowcrafter\Workflow_Run;

class My_Charge_Node extends Abstract_Node_Handler implements Has_Test_Mode_Override {

    public function execute( array $node, Workflow_Run $run, array $form ) {
        // …real charge…
    }

    public function execute_in_test_mode( array $node, Workflow_Run $run, array $form ) {
        // Same return shape as execute(): next_nodes plus optional output.
        return [
            'next_nodes' => $this->get_next_nodes( $node ),
            'output'     => [ 'charge_id' => 'test_stub' ],
            'test_note'  => 'Charge skipped — test mode.',
        ];
    }
}
```

`execute_in_test_mode()` returns the same shape as `execute()`: `next_nodes` (the IDs to run next), an optional `output` array, and an optional `test_note` string shown in the builder. Use the override for nodes whose real `execute()` would block indefinitely or leave side effects that stop you from re-testing (the core [Wait for Payment](/documentation/flowcrafter/nodes/flow/wait-for-payment/), [Wait for Approval](/documentation/flowcrafter/nodes/flow/wait-for-approval/), and [Gravity Forms Feed](/documentation/flowcrafter/nodes/actions/gf-feed/) nodes all do this).

If your override needs a choice from the tester first — the way Wait for Approval needs an approve-or-reject before it can pick a branch — read it from the run's runtime context under `test_input`: `$run->get_runtime_context()['test_input'] ?? []`. The builder gathers it and sends it on `POST /runs/{run_id}/step`; the test runner forwards it untouched, so its shape is entirely yours. See 11 Test Mode.

If tests are *meant* to hit the live service, say so in the node's UI and docs, and use the full resolved-operation fingerprint (see [Where the identity comes from](#where-the-identity-comes-from)) as the idempotency key so a re-run doesn't duplicate the side effect.

The builder lets a user select a real entry for a node test. Treat that as real execution context, not permission to send production mail or charge a card.

![The builder asks for a real entry before testing a form trigger](/documentation/flowcrafter/screenshots/guide-test-entry-picker.png)

## Testing checklist

Before you ship a node that touches an external system, walk this list:

- [ ] Retrying after remote success but before local result storage does **not** duplicate the side effect.
- [ ] A failed result save leaves the operation recoverable.
- [ ] An expired worker can't overwrite a newer worker's result.
- [ ] Temporary failures retry; permanent validation and credential failures return `retryable => false`.
- [ ] Changed saved settings produce a fingerprint conflict or a deliberate new operation — never silent reuse of an old business key.
- [ ] A timeout or disconnect after possible provider acceptance persists a needs-reconciliation marker, and the next attempt reconciles before creating again.
- [ ] Loop iterations have distinct operation identities; Loop Continue, Stop, and full-rerun behavior all match the documented guarantee.
- [ ] Large or slow pagination stops, fails, or reports truncation at a documented safe limit.
- [ ] Redirects to private, loopback, or metadata hosts are rejected.
- [ ] HTTPS-only requests can't redirect to plaintext HTTP.
- [ ] HTTPS-only requests reject an initial plaintext HTTP URL.
- [ ] Resolved JSON is decoded and validated before the request.
- [ ] Temporary non-JSON 429/5xx responses stay retryable.
- [ ] Response byte limits apply during download (`limit_response_size`), not only afterward.
- [ ] Builder tests either avoid live side effects or clearly document and deduplicate them.
- [ ] Stored remote responses have a size limit, a retention policy, and no unnecessary secrets.