Main Content

Configure Environment for Interactive Debugging

Supported Platform: Linux® only.

Before you can interactively debug your applications using the MATLAB® API for Spark™, you must configure your MATLAB environment by adding the location of the Spark assembly jar to the front of MATLAB's static Java class path.

A Spark assembly JAR file includes all the Spark dependencies, including some internal Hadoop® dependencies. This JAR file is created during Spark installation and is specific to a particular combination of Spark and Hadoop versions. For example, using Spark version 1.4.0 and Hadoop version 2.4.0 creates the file spark-assembly-1.4.0-hadoop2.4.0.jar.

To add the location of the Spark assembly jar to the front of MATLAB’s static Java® class path, use the following procedure:

  1. Open a Linux terminal.

  2. Navigate to the directory <matlabroot>/bin, where <matlabroot> is the location where MATLAB is installed.

  3. Set the environment variable SPARK_HOME to point to the Spark installation directory. For example:

    ShellCommand
    zsh / csh / tcsh 

    % setenv SPARK_HOME /share/spark

    bash 

    $ export SPARK_HOME=/share/spark

  4. From the directory <matlabroot>/bin, execute the shell script generatesparkjcpath. For example:

    $ ./generatesparkjcpath

    This will create a file called javaclasspath.txt in your <matlabroot>/bin directory. The contents of a sample javaclasspath.txt file look like this:

    <before>
/share/spark/conf/
/share/spark/lib/spark-assembly-1.4.0-hadoop2.4.0.jar
/share/spark/lib/datanucleus-rdbms-3.2.9.jar
/share/spark/lib/datanucleus-api-jdo-3.2.6.jar
/share/spark/lib/datanucleus-core-3.2.10.jar

    The <before> tag ensures that spark-assembly-1.4.0-hadoop2.4.0.jar is added to the front of the Java class path.

  5. Start MATLAB from the <matlabroot>/bin directory.

Note

You can create the javaclasspath.txt file in a directory other than <matlabroot>/bin by specifying the -o option. The file must be named javaclasspath.txt. For example:

$ ./generatesparkjcpath -o /home/user/javaclasspath.txt
Start MATLAB from the directory that contains the javaclasspath.txt file.

You can verify whether spark-assembly-1.4.0-hadoop2.4.0.jar was added to the front of the Java class path by typing the following at the MATLAB command prompt:

>> spath = javaclasspath('-static') ;
>> spath(1:5)
You should see /share/spark/lib/spark-assembly-1.4.0-hadoop2.4.0.jar on the top of the list.

You need to complete this configuration just once. Subsequent usage only requires that you start MATLAB from the folder containing the javaclasspath.txt file.

Note

You may instead place the generated javaclasspath.txt in your prefdir. If you do so, MATLAB adds the necessary libraries to the static path at launch.

Related Topics