My first response to a difficult production issue was to add more logs. Soon the useful events were buried among request bodies, successful reads, and repeated debug lines. The system produced more data but fewer answers.
Good observability starts with questions: Which operation failed? For whom? At what stage? How often? How long did it take?
Log events, not sentences#
Use stable event names and structured context:
logger.info("invitations.accepted", {
correlationId,
invitationId,
organizationId,
durationMs,
});
invitations.accepted is searchable and can remain stable even if the human message changes. A practical naming pattern is feature.entity.action.
Record lifecycle boundaries and unusual decisions: operation started, completed, failed, skipped as an idempotent duplicate, or sent to a fallback. Routine database reads rarely need their own information log.
Protect sensitive data#
Never log authorization headers, tokens, passwords, raw request bodies, or personal content by default. Allow-list fields rather than trying to redact an unlimited set afterward.
For payload diagnostics, byte size, schema version, record count, and a one-way hash often answer the operational question without preserving the content.
Use metrics for rates and alarms#
Logs explain a specific execution. Metrics answer aggregate questions such as “Are invitation failures increasing?” Emit low-cardinality dimensions like environment, operation, and outcome.
Do not use user IDs, request IDs, or arbitrary error messages as metric dimensions. Their unbounded values increase cost and make aggregation useless. CloudWatch Embedded Metric Format can extract metrics from structured log events, but the same cardinality rule applies.
Useful serverless metrics include latency, failure count, throttles, queue age, DLQ depth, and business outcomes that require operational action.
Sample noise, retain failures#
High-volume success events may be sampled. Errors and rare state transitions should normally be retained. The policy should be explicit so an engineer knows what evidence exists during an incident.
Observability is a product surface for operators. Define stable names, attach correlation context, keep secrets out, and choose metrics based on decisions and alarms. Logging every step feels safe; recording the few events that explain the system is far more useful.