Logging is easy to add and surprisingly difficult to design. Most applications have plenty of logs. They print startup messages, exceptions, HTTP paths, SQL errors, and occasionally large objects that nobody intended to store forever. Then an incident happens and the logs still fail to answer the basic question: What happened to this request? The...
Configuration systems tend to grow accidentally. A service begins with a few environment variables: DATABASE_URL PORT LOG_LEVEL Six months later it has sixty. Some values come from environment variables, some from YAML, some from command-line arguments, several are hard-coded defaults, and a few are fetched dynamically from a remote configuration service....
“Out of memory” sounds like a simple failure mode. A program asks the operating system for more memory than the machine has available, the allocation fails, and the program crashes. On Linux, the real behavior is considerably more complicated. Memory allocation is affected by virtual memory, overcommit, page cache, swap, cgroups, container limits, and the...
One of the more uncomfortable engineering situations is being asked to investigate a service you did not build. The report is usually vague: “Requests are sometimes slow.” There may be no obvious error. CPU usage looks normal. Memory is not exhausted. The service eventually returns successful responses. Restarting it may even appear to fix the...
Database performance debugging often begins with an unfortunate sentence: “The query looks simple.” A query may indeed be simple to read while still requiring PostgreSQL to scan millions of rows, sort a large intermediate result, perform an expensive nested loop, or repeatedly access data that cannot remain in memory. The SQL text tells us what...
Background jobs look deceptively simple. A request arrives, we put a message into a queue, a worker picks it up, performs some work, and marks the job as completed. This model works well enough until the system encounters one of the conditions distributed systems are particularly good at producing: the worker crashes after completing the...
