API response increased from 300ms to 8 seconds. How do you debug?

When latency jumps 25x, don't guess — find the one slow checkpoint, like a traffic jam at a single toll booth.

Question: API response increased from 300ms to 8 seconds. How do you debug?

Answer: I first confirm the slowdown is real by comparing the same endpoint, same payload, and the same environment, then I split the request into pieces: app code, database, and downstream services. In Spring Boot, I use Actuator metrics, logs with a trace id, thread dumps, and SQL timings to see whether the delay is caused by waiting, retries, pool exhaustion, or a slow query. Once I know the slow layer, I fix that layer and re-test under load.

Interview-Ready Answer: I’d start by confirming the regression with p95 and p99 latency for the same endpoint and payload, not just averages. Then I’d break the request path into app, database, and external calls using Spring Boot Actuator metrics, request logs, and a trace id. If CPU is low but latency is high, I’d suspect waiting — for example Hikari pool exhaustion, Tomcat thread starvation, or a downstream retry loop. After that I’d inspect thread dumps, query plans, and client timeouts, fix the bottleneck, and verify the improvement with a canary or load test.

1) Confirm the regression before changing anything

First, make sure the 8 seconds is not a measurement bug. Compare the same API, same input, same release, and same environment. Look at p50 (typical), p95 (slow tail), and p99 (worst tail), because averages can hide spikes. A jump from 300ms to 8s often means only some requests are getting stuck, not all of them.

1. Check whether the issue is in prod only or also in staging.
2. Compare one slow request to one normal request with the same payload.
3. Look for a recent change: new release, config change, DB migration, cache change, or new downstream dependency.

2) Split the request into layers

Think of a Spring Boot API as a chain: ingress, controller, service, database, remote calls, and response serialization. The goal is to find which link got slow. In practice, the first big question is: is the app doing more work, or is it waiting?

1. Use request logs with a correlation id, often called a trace id (a unique id that follows one request through many logs).
2. Add timing around service methods and external calls with StopWatch or Micrometer Timer.
3. Check whether the time is spent in your code, in JDBC, in HTTP clients, or before the request even reaches the app.

3) Check the three most common bottlenecks

In Spring Boot systems, the usual culprits are database waiting, thread pool saturation, and downstream retries. A very common pattern is: one slow dependency blocks threads, blocked threads fill the pool, and then even fast requests start waiting. That is how 300ms becomes 8 seconds very quickly.

4) Use Spring Boot observability the right way

Enable spring-boot-starter-actuator and expose the metrics you need. Useful signals include http.server.requests, GC pause time, JVM memory, Hikari pool usage, and thread pool queue depth. If you use Spring Boot with embedded Tomcat, remember the default request thread limit is often around 200; with HikariCP, the default connection pool size is usually 10. If all 10 DB connections are busy, requests can wait even when the database itself is not broken.

1. Check request latency metrics first: which endpoint is slow, and is it all traffic or only one route?
2. Check Hikari metrics: active, idle, and pending connections.
3. Check thread dumps: are threads BLOCKED or waiting on socketRead, locks, or the DB pool?
4. Check GC pauses: a long stop-the-world pause can freeze the JVM and make one request look like an 8-second hang.

5) Look for hidden retry storms and timeout math

Retries are sneaky. A client with 3 retries and a 2-second timeout can already add 6 seconds, and that is before backoff, queueing, or serialization. So if a downstream service becomes flaky, your API can become much slower even though your own code did not change. This is why I always ask: did we add retries, increase timeout values, or remove a circuit breaker?

1. Inspect HTTP client timeouts and retry counts.
2. Check whether the code waits for the full timeout on each attempt.
3. Confirm whether the remote service is slow, erroring, or rate-limiting you.

6) Compare the main tools

7) Fix, then prove the fix

After you find the cause, do not trust one happy-path request. Re-run with load, watch p95/p99, and compare before-and-after numbers. If the issue was in production, do a canary release or a small rollout so you can confirm the fix without risking the whole traffic path. The goal is not just to make one request fast again, but to keep it fast under real traffic.

Memory model: a request is a relay race. You time each lap, and the slowest lap decides the finish. If you do not time the laps, you will blame the wrong runner.

Performance note: Adding request timing and trace ids is usually tiny overhead, roughly O(1) per request. Full tracing and deep profiling should be sampled in production, because collecting everything can add noise and cost.

Real-World Example: Imagine an e-commerce checkout service built with Spring Boot. One morning, the checkout API jumps from 300ms to 8 seconds after a release that added a call to a payment-risk service. At first the team blames the database, but thread dumps show many request threads waiting on HTTP calls, and Hikari metrics show the app is nearly out of DB connections because slow requests are holding transactions open too long. The logs show repeated Read timed out messages and three retries per request. Users see spinning checkout buttons, delayed order confirmations, and sometimes failed payments even though the code path looks small on paper.

What went wrong is usually not one giant bug; it is a chain reaction. A slow downstream call triggers retries, retries hold threads, threads hold connections, and the whole API becomes slow. That is why production debugging is about following the request path step by step, not guessing from the last symptom you saw.

Follow-up & Tricky Questions:

• How do you tell whether the bottleneck is the database, a remote service, or the JVM? I compare request timing with DB metrics, downstream call timings, and JVM signals. If CPU is low and threads are waiting, I look for I/O or pool exhaustion; if CPU is high, I suspect hot code, JSON work, or GC.
• Which Spring Boot metrics do you check first? I start with http.server.requests for endpoint latency, then Hikari pool metrics, GC pauses, and thread pool usage. That gives me the fastest map of where the time is going.
• Why do p95 and p99 matter more than the average? The average can look fine while a small set of requests are horribly slow. Tail latency tells you whether a subset of users is hitting retries, contention, or a bad code path.
• What does a thread dump tell you that logs do not? It shows what threads are doing right now: running, blocked, waiting on locks, or waiting on I/O. That is the quickest way to prove whether the app is stuck or just slow.
• When would you add a circuit breaker or cache? If a downstream service is flaky or expensive and the API cannot wait forever, a circuit breaker prevents retry storms and a cache can remove repeated work. I would still fix the root cause, but these protect the user while I do it.
• Is 8 seconds always a backend problem? No. Client retries, network issues, TLS handshake delays, DNS slowness, and even oversized response serialization can all produce the same symptom.

Tricky / gotchas:

• If I add more logging, will the API become faster? No. Logging helps you observe the problem, but heavy debug logging can add overhead and hide timing issues, so use it deliberately and remove it after the fix.
• If CPU is only 20%, can the service still be saturated? Yes. A service can be saturated on threads, connections, locks, or a remote dependency even when CPU is low.
• If only one endpoint is slow, is the database always the culprit? No. One endpoint may do extra serialization, call a flaky downstream service, or hit a cache miss path that the others do not.

Common Mistakes:

• Guessing too early. Correction: measure first, then change one thing at a time.
• Only checking the average latency. Correction: compare p50, p95, and p99 so you see the slow tail.
• Blaming the database without proof. Correction: inspect query plans, connection pool usage, and downstream call timings before changing indexes.
• Forgetting about retries and thread pools. Correction: a few retries or a blocked thread pool can turn a small slowdown into an 8-second outage.

Memory Hook: Think of the request like airport security: one slow lane can hold up the whole terminal. Don’t stare at the crowd — find the slow lane, then time each checkpoint.

Cheat Sheet:

• Confirm the regression with the same endpoint, same payload, and p95/p99.
• Use Actuator metrics, trace ids, and request timings to break the request into layers.
• If CPU is low, suspect waiting: DB pool, thread pool, locks, or downstream I/O.
• If CPU is high, use a profiler and check GC pauses or hot code.
• Check retry counts and timeout values; 3 retries with 2s timeout can explain 8s.
• Verify the fix with canary traffic and a load test, not one lucky request.

Practice Tasks:

• Add a request-timing filter to one endpoint and print a trace id plus elapsed time.
• Expose Spring Boot Actuator metrics and compare p95 latency before and after a simulated 2-second downstream delay.
• Take a thread dump while the endpoint is slow and label every waiting thread with its cause.