After the Timeout: How to Reconcile Uncertain API Writes Safely

A timeout is not a failed write. It is a missing answer.

That distinction is the foundation of a reliable recovery design. A client may stop waiting after five seconds while the server continues processing. The server may have committed the mutation and then lost the response. It is also possible that the request never reached the server.

A blind retry can create a duplicate. Recording the operation as failed without evidence creates a different problem: your database may now disagree with the remote system.

The operator’s job is not to infer an outcome from a broken connection. It is to preserve the original operation’s identity, obtain an authoritative result, and avoid issuing a second logical write while the first remains unresolved.

The timeout is not the outcome

Consider a request that creates a payment, provisions an account, submits an order, or starts a long-running workflow.

The client sends the write. The provider commits it. Then the connection resets before the success response reaches the client.

The client sees a timeout. The provider sees a successful operation.

This is where a conventional retry loop becomes dangerous:

attempt 1 -> remote write succeeds -> response is lost
attempt 2 -> remote write succeeds again

The retry is necessarily a second network request. It must not become a second business operation.

Treating the timeout as a definitive failure is no safer. The client might release inventory, tell the user to try again, or launch a compensating process even though the remote operation has already succeeded.

AWS’s guidance on durable execution describes the underlying problem: a step interrupted before completion from the workflow’s perspective may run again even if part of its work already occurred. Unprotected, non-idempotent side effects are therefore unsafe to replay. A checkpoint can restore completed orchestration state, but it cannot retroactively make an external mutation idempotent.

A resilient pipeline needs a third outcome alongside success and failure:

SUCCEEDED
FAILED
UNKNOWN

UNKNOWN is not a miscellaneous error bucket. It is a legitimate operational state that requires reconciliation.

Model delivery uncertainty explicitly

A definitive business rejection and a transport failure are not the same event.

If the provider returns a validated response saying an account is closed, the operation failed. If the socket closes before an authoritative response arrives, the outcome is unknown. No confirmation does not mean confirmed failure.

After a timeout, retain an operation record with at least:

• An internal operation ID
• The original idempotency key
• A canonical payload hash
• The current state
• First-seen and last-attempt timestamps
• Attempt history
• The timeout or transport-error category
• Any remote identifier received
• The final response or a durable reference to it

Do not replace that record with a generic failure. Safe reconciliation depends on retaining the operation’s original identity.

A useful state model might look like this:

PENDING -> INPROGRESS -> SUCCEEDED
                      -> FAILED
                      -> UNKNOWN
UNKNOWN -> INPROGRESS
        -> SUCCEEDED
        -> FAILED
        -> MANUAL_REVIEW

The labels are less important than the distinction they preserve. FAILED should mean the system has authoritative evidence of failure. UNKNOWN should mean the system cannot yet determine whether the side effect occurred.

That distinction also governs downstream behavior. An unknown payment should not automatically trigger a new payment request. An uncertain provisioning operation should not immediately allocate a second resource. The pipeline should pause, query the remote system, replay safely where that is supported, or send the operation for review.

Give the logical write one stable identity

An idempotency key identifies a business operation, not one HTTP attempt.

Generate or derive the key once, before the first request, and preserve it across every replay. AWS explicitly advises keeping the same idempotency token across attempts for a side-effecting operation. Generating a fresh token during recovery defeats deduplication.

This is correct:

operation: create-order-8472
attempt 1 key: 4f1d...
attempt 2 key: 4f1d...
attempt 3 key: 4f1d...

This is not:

operation: create-order-8472
attempt 1 key: 4f1d...
attempt 2 key: a92b...
attempt 3 key: 73c8...

In the second pattern, the receiver has no durable evidence that all three requests represent the same logical write.

Keys may be random values generated by the client or values derived from stable business fields. Payload-derived keys are convenient only when the selected fields capture the operation’s real identity. Retry counters, request timestamps, trace IDs, and other changing metadata should stay out unless they genuinely change the business meaning.

I would also bind each key to a canonical payload hash. The key answers, “Which logical operation is this?” The hash answers, “Does this replay still describe that same operation?”

If a caller reuses a key with different parameters, return a conflict. Do not execute the changed payload. Do not return an earlier success as if it applied to the new request.

AWS Powertools’ idempotency documentation describes records containing the key, execution status, expiration data, payload hash, and stored response. That is a practical minimum because it supports replay, conflict detection, concurrent-request control, and eventual cleanup.

Make the receiver’s decision atomic

A client-provided key offers no protection unless the receiver claims it atomically.

Before performing the side effect, the receiver should create an idempotency record in an INPROGRESS state. The claim needs a uniqueness constraint, conditional write, or equivalent atomic operation.

The handling sequence should be:

1. Receive key and payload.
2. Canonicalize the payload and calculate its hash.
3. Atomically create an INPROGRESS record for the key.
4. Perform the side effect.
5. Save COMPLETE with the result or a durable result reference.
6. Return the result.

An existing record changes the path:

existing COMPLETE + matching hash
    -> return the stored result

existing INPROGRESS + matching hash
    -> report, wait for, or query the existing operation

existing key + different hash
    -> reject as a conflict

A loose “check, then insert” sequence is not enough. Two requests can run the check simultaneously, both observe that the key is absent, and both execute the mutation.

The critical boundary is the business side effect, not merely the HTTP handler. Marking the record INPROGRESS before calling the protected function helps prevent concurrent requests from crossing that boundary together, as documented by AWS Powertools for TypeScript.

The completion step matters as well. When practical, a repeated request should receive the original result rather than a vague “duplicate” error. A stored response or durable result reference allows the retry to converge on the same observable outcome as the first successful request.

Reconcile before creating a new operation

When a request times out, recovery should begin by preserving the original operation identity.

If the provider supports idempotent requests, resend the original payload with the original key. A correct implementation will continue the existing operation, return its stored result, or reject a payload mismatch.

If the provider offers a status endpoint, query it with the strongest available reference:

• Idempotency key
• Execution name
• Client-generated operation ID
• Provider-generated operation ID
• Stable business reference

For example, AWS Lambda durable executions use a supplied execution name as an idempotency key. Starting the execution again with the same name and identical payload returns information about the existing execution instead of creating another one. Reusing the name with a different payload is rejected.

Reconciliation should place the remote operation into one of four useful outcomes:

• Succeeded: Record the authoritative result locally.
• Definitively failed: Record the rejection and stop retrying.
• Still processing: Leave the original operation open and check again later.
• Unresolved: Quarantine the operation instead of guessing.

Create a new logical operation—with a new key—only after the original has been authoritatively ruled out or intentionally compensated.

This is a common point of failure. A retry button that quietly generates a new key is not a recovery mechanism. It is a duplicate-operation button with friendly labeling.

Sometimes the business may decide to submit a replacement even though the remote state remains unresolved. That choice should be explicit. Link the replacement to the original operation, record who or what authorized it, and retain the reason. High-consequence side effects may require a human to remain in that decision loop.

Treat stale INPROGRESS records as ambiguous

An INPROGRESS record cannot block work indefinitely. Workers crash, functions time out, and execution environments disappear.

Give each in-progress record a lease or execution deadline. Base that deadline on the operation’s maximum credible execution time, not on the shorter period the client is willing to wait.

For example, a client may time out after five seconds while the server can legitimately continue for a minute. Expiring the receiver’s lock after those same five seconds reopens the duplication window while the first worker may still be active.

AWS Powertools accounts for Lambda timeouts by attaching an in-progress expiration related to the invocation’s remaining execution time. Once that expiration has elapsed, a later attempt can treat the record as expired rather than remain blocked forever.

Expiration, however, grants permission to investigate. It does not prove that the write never happened.

This sequence is still possible:

1. Worker claims the idempotency key.
2. Worker sends a non-idempotent external request.
3. External provider commits the request.
4. Worker crashes before saving COMPLETE.
5. The INPROGRESS lease expires.

Rerunning the external write after step five can create a duplicate unless the external provider also honors the same idempotency identity.

Once a lease is stale, reconcile the external state before replaying an unprotected side effect. If the provider offers neither an authoritative lookup nor a deduplication facility, route the operation to manual review. Do not turn uncertainty into duplication automatically.

Keep two expiration policies separate:

• In-progress expiration determines when an abandoned execution may be investigated or reclaimed.
• Completed-record retention determines how long successful operations remain deduplicated.

After a completed record is removed, the same key may be accepted as new. The retention period therefore needs to cover the realistic retry and reconciliation horizon. Saving storage is a poor trade if cleanup reopens the duplicate window the mechanism was meant to close.

Match retry semantics to the side effect

Retry policy should follow the side effect. Not every operation deserves the same behavior.

At-least-once execution works when repeating the operation is harmless or when the receiver deduplicates it. Reads, upserts with stable semantics, and external calls that accept idempotency keys are examples.

A non-idempotent external action with no deduplication support calls for a more conservative policy. AWS recommends at-most-once behavior, with retries disabled, when a side effect cannot safely execute more than once. Unknown outcomes still occur under that policy, but they go to reconciliation rather than automatic replay.

The tradeoff is real:

• Automatic retries improve completion rates but can duplicate unsafe writes.
• Disabling retries prevents automated duplication but creates more unresolved operations.
• Reconciliation requires additional state, status queries, operator tooling, and retention.
• Human review takes time, but it can be the right boundary when the provider supplies no safe, machine-readable answer.

I would reject a design that claims “exactly once” based only on a workflow engine’s checkpointing or one retry setting. AWS’s durable execution guidance cautions that at-most-once behavior may apply per attempt rather than across the full workflow. An interrupted step may still execute again.

The defensible guarantee is narrower: one logical operation is durably deduplicated within a defined identity scope and retention window.

Build reconciliation as an operating capability

Reconciliation belongs in the write pipeline. It should not be an improvised database query someone runs during an incident.

Persist enough evidence to answer:

• What operation did we intend to perform?
• Which payload was attached to it?
• Which key was sent?
• How many attempts occurred?
• What did each attempt return?
• Did the provider issue a reference?
• When did the operation become uncertain?
• What evidence established its final disposition?

Alert when operations remain unresolved beyond an explicit service threshold. Track their age as well as their count. One old, ambiguous operation can matter more than many recent retries that are still within the expected processing window.

Useful operational signals include:

• Duplicate requests suppressed
• Idempotency-key payload conflicts
• Stale INPROGRESS records
• Reconciliation latency
• Unresolved-operation age
• Operations escalated for manual review

The repair interface should expose a controlled set of actions: query the provider again, accept an authoritative success, record a definitive failure, continue waiting, or escalate.

It should not make “send again” the default.

Auditability matters here. If an operator deliberately creates a replacement operation, the system should link it to the unresolved original. Otherwise, later reconciliation may discover both operations with no record of why the second one was authorized.

Test the gaps between state transitions

A few happy-path idempotency tests will not establish reliability. The dangerous failures occur between durable state changes.

At minimum, test these sequences:

1. The remote write commits, but its response never reaches the client.
2. Two matching requests with the same key arrive concurrently.
3. A key is reused with a different payload.
4. A worker fails after claiming the key but before performing the write.
5. A worker fails after performing the side effect but before saving COMPLETE.
6. A retry arrives after the in-progress lease expires.
7. A retry arrives after the completed record has been deleted.
8. A workflow replay generates a new key and bypasses deduplication.
9. The provider reports “still processing” longer than the client’s timeout.
10. A replacement operation is requested while the original remains unresolved.

The fifth case is the one I would insist on testing before release. It exposes the gap between local idempotency state and an external side effect. If the external system cannot deduplicate the request or report an authoritative status, a local wrapper cannot manufacture certainty after the fact.

A practical decision sequence after a timeout

Once a write times out, keep the recovery path narrow:

1. Mark the local operation UNKNOWN.
2. Preserve the original key, payload hash, and attempt evidence.
3. Determine whether the target supports idempotent replay.
4. If it does, replay with the same key and same payload.
5. Otherwise, query status using the original operation identity.
6. Accept stored success, active processing, or definitive failure as authoritative.
7. If no authoritative result exists, quarantine for reconciliation.
8. Create a replacement only through an explicit business decision.

Do not generate a new idempotency key merely because one transport attempt ended.

Do not treat an expired INPROGRESS lease as evidence that an external write never occurred.

Do not collapse UNKNOWN into FAILED because the data model offers only a Boolean success field.

Reliable write pipelines preserve ambiguity until evidence resolves it. That requires more state than a generic retry loop and more operational discipline than returning a 500. It is still cheaper than letting a timeout choose, by accident, between duplicate execution and inconsistent records.

Sources and further reading

• Idempotency — Powertools for AWS Lambda (Python)
• Idempotency — Powertools for AWS Lambda (TypeScript)
• Idempotency and retries — AWS Durable Execution SDK Developer Guide
• Idempotency — AWS Lambda
• Understanding the Idempotency Key in Bitcoin and Fintech — Lightspark