PyCharm 2020.2 Help

Spark monitoring

With the Big Data Tools plugin you can monitor your Spark jobs.

Typical workflow:

  1. Establish connection to a Spark server

  2. Adjust the preview layout

  3. Filter out jobs parameters

Create a connection to a Spark server

  1. In the Big Data Tools window, click Add a connection and select Spark under the Monitoring section. The Big Data Tools Connection dialog opens.

  2. The Big Data Tools Connection dialog opens.

    Configure Spark connection

    Mandatory parameters:

    • URL: the path to the target server.

    Optionally, you can set up:

    • Name: the name of the connection to distinguish it between the other connections.

    • Per project: select to enable these connection settings only for the current project. Deselect it if you want this connection to be visible in other projects.

    • Enable connection: deselect if, for some reasons, you want to restrict using this connection. By default, the newly created connections are enabled.

    • Enable HTTP basic authentication: connection with the HTTP authentication using the specified username and password.

    • Enable HTTP proxy: connection with the HTTP proxy using the specified host, port, username, and password.

    • HTTP Proxy: connection with the HTTP or SOCKS Proxy authentication. Select if you want to use IDEA HTTP Proxy settings or use custom settings with the specified host name, port, login, and password.

    • Kerberos authentication settings: opens the Kerberos authentication settings.
      Kerberos settings

      Specify the following options:

      • Enable Kerberos auth: select to use the Kerberos authentication protocol.

      • Krb5 config file: a file that contains Kerberos configuration information.

      • JAAS login config file: a file that consists of one or more entries, each specifying which underlying authentication technology should be used for a particular application or applications.

      • Use subject credentials only: allows the mechanism to obtain credentials from some vendor-specific location. Select this checkbox and provide the username and password.

      • To include additional login information into PyCharm log, select the Kerberos debug logging and JGSS debug logging.

        Note that the Kerberos settings are effective for all you Spark connections.

  3. Once you fill in the settings, click Test connection to ensure that all configuration parameters are correct. Then click OK.

At any time, you can open the connection settings in one of the following ways:

  • In the Settings/Preferences dialog Ctrl+Alt+S, go to Tools | Big Data Tools Settings

  • Click settings on the Spark monitoring tool window toolbar.

Once you have established a connection to the Spark server, the Spark monitoring tool window appears.

Spark monitoring: jobs

The window consists of the several areas to monitor data for:

  • Application: a user application is being executed on Spark .

  • Job: a parallel computation consisting of multiple tasks.

  • Stage: a set of tasks within a job.

  • Environment: runtime information and Spark server properties.

  • Executor: a process launched for an application that runs tasks and keeps data in memory or disk storage across them.

  • Storage: server storage utilization.

  • SQL: specific details about SQL queries execution.

You can also preview info on Tasks, units of work that sent to one executor.

Refer to Spark documentation for more information about types of data.

Adjust layout

  • In the list of the application jobs, select a job to preview.

  • To focus on a particular stage, switch to the Stages tab.

    Job stages
  • To manage visibility of the monitoring areas, use the following buttons:

    ItemDescription
    Preview detailsShows details for the selected stage.
    Show tasksShows the list of the tasks executed during the selected stage.

    Showing stage details

  • Click Preview on web to preview any monitoring data in a browser.

Once you have set up the layout of the monitoring window, opened or closed some preview areas, you can filter the monitoring data to preview particular job parameters.

Filter out the monitoring data

  • Use the following buttons in the Applications, Jobs, and Stages tabs to show details for the jobs and stages with specific status.

    ItemDescription
    Running jobsShow running applications, jobs, or states
    Succeeded jobsShow succeeded applications, jobs, or stages
    Failed statusShow failed jobs or stages
    Unknown statusShow jobs or stages with unknown status
    Test skippedShow skipped stages
  • Manage content within a table:

    • Click a column header to change the order of data in the column.

    • Click Show/Hide columns on the toolbar to select the columns to be shown in the table:

      Select columns to show in the table

At any time, you can click Refresh on the Spark monitoring tool window to manually refresh the monitoring data. Alternatively, you can configure the automatic update within a certain time interval in the list located next to the Refresh button. You can select 5, 10, or 30 seconds.

Last modified: 13 July 2020