Idempotency in Distributed Systems: Prevent Duplicate Transactions
Learn how idempotency keys, database transactions, and safe retry strategies prevent duplicate payments and API operations in distributed systems.
Introduction: When a Successful Request Looks Like a Failure
Imagine a user makes a payment through your application.
The frontend sends a request to your backend. The backend processes the payment successfully, but the network connection times out before the response reaches the client.
From the user's perspective, the payment failed.
What happens next?
The user clicks the payment button again.
Your backend receives another request and processes the same payment a second time.
Now the customer has been charged twice, even though they intended to pay only once.
This is a common challenge in distributed systems. Network failures, timeouts, retries, and unreliable connections can cause the same logical operation to be submitted multiple times.
The solution isn't simply to disable the button or prevent users from clicking twice. Those techniques improve the user experience, but they cannot guarantee correctness across network failures and concurrent requests.
This is where idempotency becomes essential.
In this article, we'll explore what idempotency means, why it matters, how to implement it in a Node.js backend, and what production systems need to handle beyond a basic implementation.
1. What Is Idempotency?
An operation is idempotent when repeating it produces the same intended effect as performing it once.
In mathematical notation:
f(f(x)) = f(x)
In API design, this means that repeating the same logical request should not repeat its business effect.
Consider an API that creates a payment.
Without idempotency:
Request 1 creates payment #101.
Request 2 creates payment #102.
Both payments charge the customer.
With idempotency:
Request 1 creates payment #101.
Request 2 uses the same idempotency key.
The server returns the original result instead of creating another payment.
The client can safely retry the request without unintentionally duplicating the operation.
An important distinction: idempotency does not mean every repeated HTTP request must return the same status code or response under all circumstances. It means the same logical operation must not produce the same business effect multiple times.
For example, an API may return the original successful response for a repeated payment request, while a repeated request with a different key represents a new payment.
2. Why Distributed Systems Need Idempotency
Distributed systems cannot assume that every request will receive exactly one response.
A request may reach the server successfully while the response is lost. A load balancer may retry a request. A background worker may process a message again after a crash.
Let's examine a typical failure scenario.
A client sends
POST /api/payments.The backend receives the request.
The payment operation succeeds.
The response gets lost because of a timeout.
The client retries the request.
The backend processes it again.
The client cannot reliably determine whether the first operation succeeded simply because it did not receive a response.
This creates a fundamental challenge: a timeout does not necessarily mean an operation failed.
Idempotency helps the backend recognize repeated attempts and return the appropriate result.
Common situations include:
Payment processing and refunds.
Order creation and checkout.
Account creation and registration.
Booking and reservation systems.
Webhook processing.
Message queues and background jobs.
Retried API calls between microservices.
Whenever an operation has side effects, duplicate execution deserves careful consideration.
3. Idempotency Keys: The Core Mechanism
A common solution is an idempotency key: a unique identifier assigned to one logical operation.
For example, a client might send:
Idempotency-Key: 7f3e6d20-9c4b-4e8f-8a1c-2e6d9f3b7c1e
The client must reuse this key when retrying the same operation.
A different logical operation should use a different key.
A typical request looks like this:
POST /api/payments
Content-Type: application/json
Idempotency-Key: 7f3e6d20-9c4b-4e8f-8a1c-2e6d9f3b7c1e
{
"amount": 100,
"currency": "USD"
}The server uses the key to determine whether it has already processed the logical request.
Conceptually, the flow is:
Receive the request and idempotency key.
Check whether the key already exists.
If a completed operation exists, return its stored result.
If the operation is new, execute it safely.
Store the result against the key.
Return the response to the client.
However, a production implementation must also handle concurrent requests, failures during processing, key expiration, and conflicting payloads.
4. Implementing Idempotency in Node.js and Express
Let's build a simplified example.
Assume we have a payments API and a database table that stores idempotency records.
Step 1: Create an idempotency table
A PostgreSQL table could look like this:
CREATE TABLE idempotency_records (
id BIGSERIAL PRIMARY KEY,
scope_id TEXT NOT NULL,
idempotency_key TEXT NOT NULL,
request_hash TEXT NOT NULL,
status TEXT NOT NULL,
response_status INTEGER,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ,
UNIQUE (scope_id, idempotency_key)
);Here:
scope_ididentifies the customer, account, or operation scope.idempotency_keyidentifies the logical request.request_hashhelps detect reuse of a key with a different payload.statusrepresents the operation's processing state.response_statusandresponse_bodystore the completed response.expires_atsupports a defined retention policy.
The unique constraint is critical. It prevents two concurrent transactions from successfully creating the same idempotency record within the same scope.
The exact schema should reflect your application's authentication model, business requirements, and data-retention policies.
Step 2: Require a valid key
The API should reject requests that don't contain a valid idempotency key.
function getIdempotencyKey(req, res, next) {
const key = req.get("Idempotency-Key");
if (!key || key.length > 255) {
return res.status(400).json({
error: "A valid Idempotency-Key is required",
});
}
req.idempotencyKey = key;
next();
}In a production application, validate the permitted key format and enforce authentication and request-size limits as well.
Step 3: Check for an existing operation
The following simplified example illustrates how the server can return a previously stored response:
app.post(
"/api/payments",
getIdempotencyKey,
async (req, res, next) => {
try {
const scopeId = req.user.id;
const key = req.idempotencyKey;
const existing = await db.idempotencyRecords.findUnique({
where: {
scopeId_idempotencyKey: {
scopeId,
idempotencyKey: key,
},
},
});
if (existing?.status === "COMPLETED") {
return res
.status(existing.responseStatus)
.json(existing.responseBody);
}
// A new operation must be claimed and processed
// using an atomic, concurrency-safe workflow.
// See the transaction discussion below.
} catch (error) {
next(error);
}
}
);This example uses illustrative database-client methods. The exact query depends on your ORM and schema.
Important: checking for an existing record and then inserting a new record in separate, unprotected operations is not enough. Two requests can both observe that no record exists and then both perform the operation.
The database must enforce uniqueness, and the processing workflow must handle concurrent requests correctly.
Step 4: Make processing and persistence atomic
For business operations that can be completed within a single database transaction, claim the key, perform the business change, and store the response in the same transaction.
A simplified PostgreSQL-style workflow is:
BEGIN;
INSERT INTO idempotency_records (
scope_id,
idempotency_key,
request_hash,
status
)
VALUES (
$1,
$2,
$3,
'PROCESSING'
)
ON CONFLICT (scope_id, idempotency_key)
DO NOTHING
RETURNING id;If the insert succeeds, the transaction owns the new operation. It can apply the corresponding business change and store the completed response before committing.
UPDATE idempotency_records
SET
status = 'COMPLETED',
response_status = $4,
response_body = $5::jsonb
WHERE
scope_id = $1
AND idempotency_key = $2;
COMMIT;The business write and the completed idempotency record must be part of the same transaction.
If the transaction rolls back, both changes roll back.
If the insert conflicts with another transaction, the application must load the existing record and decide whether to return its completed response, wait for completion, or return a documented in-progress response.
These SQL fragments illustrate the approach rather than a complete drop-in payment implementation. Real code must handle the result of the insert, request-hash validation, transaction isolation, error handling, and database-client transaction semantics.
Step 5: Handle external payment providers carefully
What if your operation calls an external payment gateway?
A PostgreSQL transaction cannot atomically commit a change in your database and a charge performed by an independent payment provider.
For example:
Your API starts a payment operation.
The payment provider charges the customer.
Your application crashes before storing the result.
A retry arrives.
A database-only idempotency check cannot, by itself, determine whether the external charge succeeded.
For payment processing, use the provider's idempotency mechanism when available. Persist the logical operation and its provider reference, reconcile uncertain outcomes, and design recovery procedures for interrupted workflows.
For longer-running operations, a durable state machine or workflow with explicit states such as PENDING, PROCESSING, COMPLETED, and FAILED may be appropriate.
Never assume that an external request can safely be repeated just because the previous response was lost.
5. Idempotency vs. Duplicate Prevention
These concepts are related, but they are not interchangeable.
Frontend duplicate prevention
Disables a button or displays a loading state to reduce accidental repeated submissions.
Idempotency keys
Identify retries of the same logical request, allowing the backend to recognize repeated attempts.
Database constraints
Enforce invariants such as unique order references or one registration per account.
Transactions
Keep related database changes consistent and atomic.
Provider-level idempotency
Helps prevent duplicate effects when calling external services that support idempotent operations.
Reliable systems often use several of these techniques together.
A disabled button improves the user experience. A database constraint protects data integrity. An idempotency key helps the server recognize retries. None should be treated as a universal replacement for the others.
6. Common Idempotency Mistakes
Mistake 1: Generating a new key on every retry
If the client generates a new key each time it retries, the server sees each attempt as a different operation.
Better approach: create the key once per logical operation and reuse it for retries.
Mistake 2: Using only an in-memory cache
An in-memory map may work in a local demonstration, but it fails when the process restarts or another server instance receives the retry.
Better approach: use durable shared storage with database-enforced uniqueness or an equivalent coordination mechanism.
Mistake 3: Checking before inserting without atomic protection
Two requests can check simultaneously, both find no record, and both proceed.
Better approach: use a unique constraint and a concurrency-safe claim-and-process workflow.
Mistake 4: Reusing a key with a different payload
A client might accidentally submit a different amount with an existing key.
Better approach: store a canonical request hash and reject incompatible reuse according to a clearly documented API contract.
Mistake 5: Storing only the key
Knowing that a key exists isn't enough to reproduce a completed result.
Better approach: persist the necessary status and response data, subject to privacy and retention requirements.
Mistake 6: Ignoring incomplete operations
A process can fail after claiming a key but before finishing the business operation.
Better approach: define recovery rules for in-progress operations, including leases, timeouts, reconciliation, and safe retries where appropriate.
Mistake 7: Assuming idempotency guarantees exactly-once execution everywhere
Idempotency can prevent repeated business effects within a correctly designed boundary. It does not make a distributed workflow globally atomic.
Better approach: combine idempotency with transactional boundaries, durable messaging, reconciliation, and downstream deduplication as required.
7. Best Practices for Production Systems
Use the following checklist when implementing idempotency:
Generate unique, unpredictable keys for logical operations.
Scope keys to the appropriate customer, account, or endpoint.
Reuse keys for retries of the same operation.
Validate that repeated requests have compatible payloads.
Enforce uniqueness at the storage layer.
Atomically persist database changes and their idempotency results when possible.
Define what happens when the original request is still processing.
Set a documented retention period for idempotency records.
Use external providers' idempotency capabilities where available.
Monitor duplicate attempts, conflicting keys, failed operations, and reconciliation outcomes.
Test concurrent requests, timeouts, crashes, and retries.
Document which errors are safe to retry and how clients should behave.
Remember that the correct retention period depends on the operation. A short-lived read-like operation and a financial transaction may require very different policies.
8. How to Test Idempotency
A successful response in a single-request test doesn't prove your implementation is safe.
Test at least these scenarios:
Repeated request: Send the same payload and key twice. Verify that the business operation occurs only once.
Concurrent requests: Send multiple requests with the same key simultaneously. Verify that only one logical operation is committed.
Timeout and retry: Simulate a lost response after the operation succeeds, then retry with the original key.
Payload mismatch: Reuse the key with a different amount or other meaningful input. Verify that the request is rejected according to the API contract.
Process interruption: Simulate a crash or database failure during processing and verify that recovery doesn't create duplicate effects.
Multi-instance behavior: Send retries to different backend instances. Verify that shared persistence, rather than process-local state, coordinates the requests.
External provider uncertainty: Simulate a provider timeout where the external operation may have succeeded. Verify that the application reconciles the outcome instead of blindly issuing a second operation.
These tests are especially important for payment systems, order processing, and asynchronous workflows.
Conclusion: Build APIs That Handle Retries Safely
In distributed systems, retries are unavoidable. Networks fail, clients time out, processes restart, and messages may be delivered more than once.
The objective isn't to eliminate every retry. It's to ensure that retrying a logical operation doesn't unintentionally repeat its business effect.
Idempotency keys provide a practical foundation, but production reliability also requires atomic database operations, concurrency control, durable state, external-provider coordination, and thoughtful recovery behavior.
When designed correctly, idempotency makes APIs safer to retry and helps protect the consistency of your business data.
Build systems that handle retries — gracefully.
About the Author
Asad Saeed is a Senior Software Engineer and Full Stack Developer focused on building scalable, maintainable web applications and exploring software architecture, backend engineering, cloud technologies, and modern development practices.
Portfolio: https://asad-saeed.vercel.app/
LinkedIn: https://linkedin.com/in/asad-saeed-dev
GitHub: https://github.com/Asad-Saeed
Medium: https://asad-saeed.medium.com/
Follow me for practical insights on full-stack development, system design, backend engineering, and software architecture.
#DistributedSystems #BackendDevelopment #SystemDesign #API #AsadSaeed
