Application developers often use debuggers to assist with application development and fix problems in their code. Typically, developers run and debug their applications locally on their workstation, however one of the challenges with developing big data applications is that they’re designed to be run on a multi-node cluster. The presents a challenge for debugging because you have to attach a debugger to a process that’s running on a remote machine. Fortunately, it’s actually not that hard.
My favorite IDE is IntelliJ. I use it to develop Spark applications that run on remote multi-node clusters. I configure maven to compile my application and all its dependencies into a single jar, then after I build my jar file I upload it to my remote cluster and run it like this:
In order to attach the IntelliJ debugger to my spark application I’ll define the SPARK_SUBMIT_OPTS environment variable and run it like this:
After running that command, it will wait until you connect your debugger, as shown below:
Now you can configure the IntelliJ debugger like this, where 10.200.1.101 is the IP address of the remote machine where I’m running my Spark job:
Start the debugger by clicking Debug under IntelliJ’s Run menu. Once it connects to your remote Spark process you’ll be off and running. Now you can set breakpoints, pause the Spark runtime, and do everything else you can normally do in a debugger. Here’s an example of what IntelliJ shows when pausing a Spark job with a breakpoint:
So, that’s how you attach IntelliJ’s debugger to a Spark application running on a remote cluster. Isn’t that nice!