DataSpell 2021.3 Help

Work with data files

Once you have established connection to a server, you can work with the data files. With the Big Data Tools plugin, you can easily perform basic file operations as well as quickly preview large structured files in tabular form.

Filter the list of buckets

  1. For the S3-like servers, you can filter the buckets to preview and work with. You can do it when configuring a new connection, or you can tweak earlier configured connection settings. Select a server in the Big Data Tool tool window and click connection settings on the window toolbar.

  2. In the settings dialog, specify a pattern in the Buckets filter list, then select the way the filter should be applied from the Filter type. If no filter is specified, all buckets will be shown in the Big Data Tool.

    Filtering the list of buckets

Manage server directories

  1. Expand the server node to preview its structure.

  2. Select a target directory and right-click to open the context menu.

    Context menu to preview the structure of servers in BDT window
  3. You can copy, paste, rename the directory, change its location, or delete it. Select Upload from disk to add more files to the directory.

    You can also save the directory and its files on the local drive.

Manage data files

  1. Expand the target directory and select a file.

  2. Right-click the file to open the context menu.

    Context menu to work with the data files
  3. You can copy, paste, rename the file, change its location, or delete it.

  4. To briefly preview details of a structured file, such as .csv or .parquet, expand its node in the directory structure. You should be able to see the columns and their types.

    Expanded csv file

    Select Show file info from the context menu to obtain more details about the file:

    File info
  5. To view a .csv or .parquet, double-click it or select the Preview command from the context menu. The file opens in the editor. You cannot edit it but you can preview it as a table or as text:

    Table view of the csv file
    Text view of the csv file

    In the table view, you can operate with table elements. Right-click to open the context menu and select a command to copy a raw or a column, or copy the entire table to the clipboard or file.

    Table specific commands

    You can also sort data in columns by clicking column headers.

    When you open .parquet files, the plugin only displays the first portion of the file content. This is especially useful when you work with very large files.

Create a new bucket

  1. To add a new bucket to the data storage, right-click the server in the Big Data Tool tool window and select Create Bucket from the context menu.

    Adding a new bucket
  2. Specify the new bucket name and click Ok to complete the task.

In case the server connection has been lost, the corresponding icon shows the disconnected status of the server Server connection got lost.

Disconnected server
Click Refresh connections to reestablish the connection to the server.

Last modified: 10 December 2021