Practice: Debugging Across Layers
Realistic production symptoms with evidence — top output, socket tables, packet captures, DNS answers. Choose what to investigate; each area reveals a finding, and some are red herrings. Name the layer the fault lives in, find the root cause, then compare your fix.
Debugging challenges
DNSBeginner
Half the users still hit the old server, two hours after the migration
The API was moved to a new host at 14:00; the old host was shut down at 14:30.
SecurityBeginner
The service runs, `curl` from the bastion hangs for two minutes
A new internal service on port 8443 was deployed and its own health check passes locally.
TCPBeginner
Works with `curl localhost`, refused from every other machine
A Node.js service in a container answers `curl http://localhost:3000` inside the container.
TLSBeginner
Every client broke at midnight and nobody deployed anything
Starting at 00:00 UTC, mobile apps and browsers show "connection is not private" / `ERR_CERT_DATE_INVALID`.
TLSIntermediate
Works in the browser, fails from the backend with a certificate error
A new subdomain `reports.example.com` was added behind the shared reverse proxy.
EdgeIntermediate
502s during every deploy, for exactly eight seconds
Each rolling deploy produces a burst of `502 Bad Gateway` for ~8 s, then recovers on its own.
EdgeIntermediate
The report endpoint returns 504 — and the server logs a 200 later
`GET /reports/annual` returns `504 Gateway Timeout` after exactly 60 s.
TCPAdvanced
A 1 Gbit/s link, and the backup crawls at 9 Mbit/s
A nightly 40 GB backup between two data centres used to take 25 minutes and now takes over 9 hours.
IPAdvanced
Over the VPN, small responses work and anything large hangs forever
Through the new site-to-site VPN, `curl https://api.internal/health` (a 40-byte body) returns instantly.
IPAdvanced
Outbound calls fail from the whole cluster whenever traffic peaks
During peak hours, services across all nodes see sporadic `connect: connection timed out` to external APIs.
PerformanceIntermediate
Fast from Frankfurt, 2.4 seconds from Sydney — and the server is idle
Users in Australia report the dashboard takes 2–3 s to load; European users see ~300 ms.
TogetherIntermediate
One slow client makes the whole API slow
Every few minutes the API’s p99 jumps from 40 ms to 8–12 s for all clients, then recovers.
TogetherAdvanced
Connection errors to a healthy upstream at 400 requests per second
A proxy service starts failing calls to its upstream with `connect: cannot assign requested address` (`EADDRNOTAVAIL`) once traffic passes ~400 req/s.
TogetherAdvanced
The producer hangs, the consumer is idle, and nothing is on the wire
An event-ingestion pipeline stops: the producer service’s `send()` calls block and its internal queue fills; the consumer service shows 2% CPU.
TogetherIntermediate
"Too many open files" — and then the health check passes
After 6–8 hours of traffic, the service starts returning 500s; logs show `accept: too many open files` and `EMFILE` from an HTTP client.