Remote debugging a Spring Boot service on Kubernetes with JDWP

When a bug appears only inside the cluster, the usual approach is to add log lines and redeploy. You rebuild the image, push it, redeploy the pod, wait for the rollout and reproduce the bug. On a local application this takes seconds. In Kubernetes it takes several minutes for each attempt.

The JVM supports remote debugging with JDWP (Java Debug Wire Protocol). With JDWP enabled, IntelliJ or VS Code attaches to the JVM inside the pod. You can then use breakpoints, inspect variables, evaluate expressions and step through the code while it runs in the cluster.

How JDWP works

JDWP is part of the JVM. One startup flag enables it:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

The options mean:

• transport=dt_socket: use a TCP socket.
• server=y: the JVM opens the debug port.
• suspend=n: the application starts without waiting for a debugger.
• address=*:5005: listen on port 5005 on all interfaces.

The application starts as usual, and the JVM accepts a debugger connection on port 5005 at any time. The IDE debugger then works the same way as with a local application.

JDWP has no authentication and no encryption. Anyone who can reach the port can control the JVM, so the debug configuration must never reach production.

Keeping the debug configuration out of production

The base deployment has no JDWP configuration, so the production manifests never contain it. A Kustomize overlay adds it only in the environments that need it.

The folder structure:

k8s/
  base/
    deployment.yaml
    service.yaml
    kustomization.yaml

  overlays/
    dev/
      kustomization.yaml
      debug-patch.yaml

    stg/
      kustomization.yaml
      debug-patch.yaml

    prod/
      kustomization.yaml

The dev and staging overlays have a debug-patch.yaml. The prod overlay does not.

overlays/dev/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

patches:
  - path: debug-patch.yaml

overlays/dev/debug-patch.yaml

apiVersion: apps/v1
kind: Deployment

metadata:
  name: debug-demo

spec:
  template:
    spec:
      containers:
        - name: app
          env:
            - name: JAVA_TOOL_OPTIONS
              value: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"

          ports:
            - containerPort: 5005
              name: debug

Deploy the development environment:

kubectl apply -k k8s/overlays/dev

Deploy production:

kubectl apply -k k8s/overlays/prod

The prod overlay uses only the base configuration, so the production pods have no debug port.

Use JAVA_TOOL_OPTIONS and not JAVA_OPTS. The JVM itself reads JAVA_TOOL_OPTIONS, so it works with any image and any start script. JAVA_OPTS works only if the start script passes it to java.

The demo project

The demo project is on GitHub at github.com/valeriomc/remote-debug-kubernetes. It is a Spring Boot order service with a pricing engine. The pricing engine has a simple bug that I added on purpose, so there is something to find with the debugger.

Requirements:

• minikube
• kubectl (v1.14+ for built-in kustomize support)
• Docker
• k9s (optional, the steps below use it)

Clone the project and start it:

git clone https://github.com/valeriomc/remote-debug-kubernetes
cd remote-debug-kubernetes

make all

make all starts a Minikube profile named remote-debug and builds the image inside the Docker daemon of Minikube. The deployment uses imagePullPolicy: Never, so no registry is needed. Then it deploys the dev overlay. When the command ends, the pod is running and JDWP listens on port 5005 inside the container.

Check the deployment:

make k9s
# or: k9s --context remote-debug

Opening the debug tunnel

The debug port is inside the pod. kubectl port-forward opens a tunnel from the local machine to that port:

make debug

The target runs this command:

kubectl --context=remote-debug port-forward deploy/debug-demo 5005:5005

Keep that terminal open. You can also start the same port-forward from k9s.

localhost:5005 on the local machine now reaches the JVM inside the pod.

Connecting IntelliJ IDEA

Inside IntelliJ:

1. Open Run > Edit Configurations, click + and choose Remote JVM Debug.
2. Set Host to localhost and Port to 5005.
3. Set Debugger mode to Attach to remote JVM.
4. In Use module classpath, select the project module.
5. Click OK, then start the configuration with the debug button.

When the connection works, the console shows:

Connected to the target VM

From now on, the breakpoints stop the code that runs inside the pod.

Connecting VS Code

Add this to .vscode/launch.json:

{
  "type": "java",
  "name": "Attach to Kubernetes JVM",
  "request": "attach",
  "hostName": "localhost",
  "port": 5005
}

Start the configuration from the Run and Debug panel. VS Code attaches through JDWP like IntelliJ.

Finding the bug

The pricing engine has a boundary bug. Gold customers must get a 10% discount when the order is $500 or more. The code checks the threshold like this:

if (subtotal.compareTo(GOLD_THRESHOLD) > 0) {

With this code:

• 500.01 gets the discount.
• 700.00 gets the discount.
• 500.00 does not get the discount.

The service does not fail and the response looks valid, so the bug is easy to miss.

Set a breakpoint on the comparison line in PricingEngine.goldDiscount() and send a request:

curl -X POST http://localhost:8080/orders \
  -H "Content-Type: application/json" \
  -d '{"customerId":"c1","tier":"gold","subtotal":500.00}'

The breakpoint stops the code in the pod. The variables panel shows:

• subtotal = 500.00
• GOLD_THRESHOLD = 500.00
• compareTo() returns 0

The condition is > 0, so the discount branch does not run for exactly 500.00.

The fix adds one character:

>= 0

Rebuild and redeploy the service:

make redeploy

After the rollout, an order of exactly 500.00 gets the discount.

Python with debugpy

The demo project also has a Python service with the same setup. Python uses debugpy, which implements the Debug Adapter Protocol (DAP). The dev overlay exposes the debug port, kubectl port-forward forwards it, and the IDE attaches to it.

Only the environment variable and the port change. debugpy listens on port 5678 by default. It can also wait for the IDE before the application starts, so you can debug the startup code.

When remote debugging helps

The bug in the demo is simple on purpose. You can find an off-by-one error in compareTo locally, without Kubernetes.

Remote debugging is useful when the bug depends on the cluster environment. Examples are a service that uses an internal message broker you cannot run locally, configuration that comes from a secrets manager or a sidecar, network policies or DNS that work differently inside the cluster, and bugs that appear only under staging load.

In these cases you cannot reproduce the bug locally. With the JDWP flag in the dev and staging overlays, you can attach the debugger to the running pod at any time without a rebuild.