Big Data tools
The Big Data Tools plugin is available for IntelliJ IDEA 2019.2 and later. It provides specific capabilities to monitor and process data with Zeppelin, S3, Spark, and Hadoop Distributed File System (HDFS). You can create new or edit existing local or remote Zeppelin notebooks, execute code paragraphs, preview the resulting tables and graphs, and export the results to various formats.
The plugin supports many IDE features to work with notebooks:
Coding assistance for Scala
Inspection and quick-fixes, including fixes for notebook dependencies.
Notebook features
Browsing, creating, and deleting notebooks
Adding and deleting paragraphs
Executing paragraphs
Running SQL statements
Getting started with Big Data Tools in IntelliJ IDEA
The basic workflow for big data processing in IntelliJ IDEA includes the following steps:
Configure your environment
Work with notebooks
Execute the notebook.
-
Analyze your data:
Get familiar with the user interface
When you install the Big Data Tools plugin for IntelliJ IDEA, the following user interface elements appear:
Big Data Tools window
The Big Data Tools window appears in the rightmost group of the tool windows. The window displays the list of the configured servers, notebooks, and files structured by folders.
Basic operations on notebooks are available from the context menu.
You can navigate through the directories and preview columnar structures of .csv and .parquet files.
Basic operations on data files are available from the context menu. You can also move files by dragging them to the target directory on the target server.
For the basic operations with the servers, use the window toolbar:
Item | Description |
|---|---|
 | Adds a new connection to a server. |
 | Refreshes connections to all configured servers. |
 | Opens the connection settings for the selected server. |
Notebook editor
In the notebook editor, you can add and execute Scala and SQL code paragraphs. When editing your code paragraph, you can use all the coding assistance features available for a particular language. Code warnings and errors will be highlighted in the corresponding code constructs in the scrollbar. The results of paragraph execution are shown in the preview area below each paragraph.
Use the notebook editor toolbar for the basic operations with notebooks:
Item | Description |
|---|---|
 | Executes all paragraphs in the notebook. |
 | Stops execution of the notebook paragraphs. |
 | Clears output previews for all paragraphs. |
 | Opens the Interpreter Bindings dialog to configure interpreters for the selected notebook. |
 | Opens the notebook in the browser. |
 | Opens the module settings. |
 | Shows the minimap for quick navigation through the notebook. |
The notebook editor toolbar also shows the status of the last paragraph execution.
Zeppelin tool window
The Zeppelin tool window shows the connection status for configured Zeppelin servers and provides links to the server for quick navigation.
Spark tool windows
This window appears when you have connected to a Spark server by creating a new connection or by opening a Spark job in a running notebook.