Reliable SQS Workers: Retries, Partial Failures, and Dead-Letter Queues

An SQS-triggered Lambda receives batches, and delivery is at least once. If one record fails and the function throws for the entire batch, successful records may return to the queue too. Without idempotency, that retry can repeat their effects.

Process records independently#

Enable partial batch responses and return only failed message identifiers:

export const handler = async (event: SQSEvent) => {
  const failures: { itemIdentifier: string }[] = [];

  for (const record of event.Records) {
    try {
      const message = messageSchema.parse(JSON.parse(record.body));
      await processMessage(message);
    } catch (error) {
      logger.error("orders.message.failed", {
        messageId: record.messageId,
        error,
      });
      failures.push({ itemIdentifier: record.messageId });
    }
  }

  return { batchItemFailures: failures };
};

This lets SQS retry failed records while removing successful ones. If records must be processed in order, especially with FIFO queues, stop after the first failure and report the remaining records too.

Make processing idempotent#

Partial failure reduces duplicate work; it does not remove it. Lambda can receive a message again after processing succeeds. Use the event ID or a business key with a unique index, conditional write, or idempotency store.

Do not mark a message complete before its effect commits. A status such as processing also needs an expiry or recovery policy so a crashed worker does not block the event forever.

Separate temporary and permanent failures#

A database timeout may succeed on retry. A payload missing a required field never will. Both can enter a dead-letter queue after the configured receive count, but recording a stable error category makes remediation much faster.

Set the visibility timeout longer than the Lambda timeout, with room for retries. Configure a redrive policy and alarm on dead-letter queue depth. A DLQ without ownership and an alarm is only delayed data loss.

Control concurrency deliberately#

More concurrency increases throughput and pressure on databases and third-party APIs. Use reserved or event-source maximum concurrency to protect dependencies. Batch size should match work duration and failure isolation, not simply the largest allowed value.

A reliable worker assumes duplicates, validates every message, retries narrowly, and makes failed work visible. The queue transports records; your processing design provides the delivery semantics the business actually needs.