Application throwing OutOfMemoryError. How do you investigate?

An OutOfMemoryError is like a warehouse that keeps filling up; the real trick is finding which room overflowed: heap, metaspace, direct memory, or thread stacks.

Question: Application throwing OutOfMemoryError. How do you investigate?

Answer: First I read the exact error text, because Java heap space, Metaspace, and unable to create new native thread point to different causes. Then I capture evidence with GC logs and a heap dump, reproduce the issue with the same JVM flags, and inspect the dump to find the biggest retainers and their GC roots. If it is not heap-related, I check direct buffers, class loading, or thread count instead of blindly raising -Xmx.

Interview-Ready Answer: I start by checking the exact OOME message and JVM flags, because that tells me whether it is heap, metaspace, direct memory, or native threads. Next I enable heap-dump-on-OOME and GC logging, reproduce the issue, and analyze the dump in a tool like MAT to find the dominant objects and reference chain. If the message points to native memory, I switch to thread dumps, class-loader analysis, or Native Memory Tracking, because the fix depends on the memory area that is actually exhausted.

What you are really investigating

An OutOfMemoryError means the JVM could not satisfy an allocation request. That does not always mean the Java heap is full; the failure may be in class metadata, off-heap buffers, or native thread memory. So the first job is to identify the exact flavor of the error and collect evidence before the process dies.

1. Read the message first. Java heap space usually means heap retention or a leak. GC overhead limit exceeded means the JVM is spending too much time collecting and reclaiming too little. Metaspace means class metadata is growing. unable to create new native thread means the OS or process cannot create more threads.
2. Turn on the right evidence. In production, add -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=/path. For GC logs, use -Xlog:gc* on JDK 9+; on JDK 8 the common flags are -XX:+PrintGCDetails and -XX:+PrintGCDateStamps. Also capture the full JVM flags with -XX:+PrintFlagsFinal or your startup config, because a bad -Xmx or too many threads can be the real cause.
3. Use the right tool for the symptom. A heap dump helps when the heap is exhausted. A thread dump helps when threads are blocked or you hit native thread limits. Native Memory Tracking helps when the heap looks fine but the process still dies.
4. Analyze for retention, not just size. In Eclipse MAT or VisualVM, look at the Dominator Tree, then follow Path to GC Roots. The question is not only “what is big?” but “what is keeping it alive?”

Comparison of common OOME messages:

How to investigate under the hood

1. Reproduce with the same workload. If possible, lower -Xmx in staging to make the failure happen sooner. Keep the same JVM version and GC, because behavior changes between collectors and JDK releases.
2. Look at the live set after GC. The live set is the memory still in use after a collection. If the live set keeps growing across multiple full GCs, that is a classic leak pattern. If it jumps up and then falls back, it may be a burst or a temporary spike instead.
3. Open the heap dump. MAT can show the biggest retained objects, dominators, and suspicious collections. Common culprits are unbounded caches, static maps, session stores, listener lists, ThreadLocal misuse, and huge byte[] or char[] buffers.
4. Check non-heap memory too. Metaspace problems often come from classloader leaks, hot reloading, bytecode generation, or many proxies. Direct buffer issues often show up when NIO code allocates ByteBuffer.allocateDirect without releasing pressure. For threads, remember that each thread consumes native memory plus stack; a common stack size is about 1 MB, so thousands of threads can fail even if heap usage looks fine.
5. Prove the fix. After the change, rerun the same load and verify that memory plateaus, full GC frequency drops, and the process survives longer under the same pressure. Do not call the job done just because the error disappeared after increasing -Xmx; that often only hides the leak.

Memory hook: ask yourself, “Which shelf is empty, and what keeps refilling it?” That is the JVM memory mindset.

Performance note: heap-dump analysis is roughly proportional to object count, and a multi-GB dump can take minutes to open. Tools like MAT may need several gigabytes of workstation RAM just to inspect a big production dump comfortably.

Real-World Story: A checkout service in a flash sale started returning 500s after a new release. The pods were not immediately dead, but latency climbed, CPU rose, and logs showed repeated full GCs followed by Java heap space. A heap dump in MAT showed one static ConcurrentHashMap inside a promotion cache retaining millions of SKU objects with no TTL. The fix was to cap the cache, add expiration, and move the hot catalog to Redis. The misunderstanding caused a slow-motion outage: requests queued, GC paused the app, and the autoscaler kept adding more pods that all hit the same leak.

The key lesson is that OOME often looks like a performance issue first: timeouts, retries, and GC churn appear before the final crash. If you wait until the last minute to collect a heap dump, you may only see a dying process and miss the real retainer graph.

Follow-up & Tricky Questions:

• How do you tell a leak from a spike? A leak keeps the live set growing after GC; a spike rises during load and then returns close to baseline after collections.
• Why do GC logs matter if you already have a heap dump? Logs show the timeline: allocation rate, pause frequency, promotion failures, and whether the JVM was already struggling before the crash.
• What is a dominator tree? It is a view of which objects retain the most memory. The dominator is the object whose removal would free the largest subgraph.
• How do you investigate Metaspace OOME? Check classloader leaks, dynamic proxies, generated classes, and hot reload behavior; a heap dump plus classloader stats usually helps more than just raising heap size.
• How do you investigate unable to create new native thread? Count live threads, inspect thread dumps, and verify OS limits and stack size, because the heap may be healthy while native memory is exhausted.
• What is Native Memory Tracking? It is a JVM feature that reports native memory usage by category, which is very useful when the heap is not the problem.
• Can I catch OutOfMemoryError and keep going? Technically you can catch it, but the JVM may already be in a fragile state. In production the safer move is to log evidence and let the process restart.
• If I increase -Xmx, did I fix it? Not necessarily. You may have only delayed the crash; if the retained set keeps growing, the real bug is still there.
• Is a heap dump enough for every OOME? No. It is great for heap leaks, but not for most native-memory failures, thread exhaustion, or some direct-buffer problems.

• Trick: If the heap is only 40% used, can I still get OOME? Yes. The failure may be metaspace, direct memory, or native threads, none of which show up as normal heap usage.
• Trick: Does GC overhead limit exceeded always mean a leak? No. It means GC is buying very little memory back; the root cause could be a leak, but also an allocation burst or too-small heap.
• Trick: Should I always start with MAT? No. First read the exact error and collect logs. MAT is powerful, but the wrong tool for a thread or native-memory failure.

Common Mistakes:

• Only increasing -Xmx. That may postpone the crash but hide the leak; first find the retaining object or memory area.
• Ignoring the exact message. Java heap space and Metaspace need different tools and fixes.
• Forgetting non-heap memory. Direct buffers, threads, and class metadata can kill the process even when heap usage looks normal.
• Not collecting evidence early. If you wait, the process may restart and erase the clue trail; enable heap-dump and GC logging before the next incident.

Memory Hook: picture the JVM as a city: heap is apartment space, metaspace is the city registry, direct memory is the warehouse, and threads are the roads. The OOME message tells you which district flooded.

Cheat Sheet:

• Read the exact OOME text first.
• Turn on heap dumps and GC logs.
• Heap dump + MAT = find retainers and GC roots.
• Metaspace/direct/thread failures need different evidence.
• Do not trust a fix until the same load stays stable.

Practice Tasks:

• Run the demo with no args and watch memory stay flatter.
• Run it with leak and a small heap such as -Xmx64m; note how the snapshot changes before failure.
• Use jcmd <pid> GC.heap_info and jcmd <pid> Thread.print on a test app to practice collecting evidence quickly.