PyCharm 2022.3 Help

AWS Glue

With the Big Data Tools plugin, you can monitor your AWS Glue platform.

Typical workflow:

  1. Establish connection to an AWS Glue server

  2. Preview your storage in the editor

  3. Preview your databases and partitions in a dedicated tool window

Connect to an AWS Glue server

  1. In the Big Data Tools window, click Add a connection and select AWS Glue.

  2. In the Big Data Tools dialog that opens, specify the connection parameters:

    Configure AWS Glue connection
    • Name: the name of the connection to distinguish it between the other connections.

    • Region: select a region to get buckets from.

    • Authentication type lets you select the authentication method:

      • Default credential providers chain: use the credentials from the default provider chain. For more info on the chain, refer to Using the Default Credential Provider Chain.

      • Explicit access key and secret key: enter your credentials manually.

      • Profile from credentials file: select a profile from your credentials file. Click Open Credentials to locate the directory where the credentials file is stored (for AWS, it's usually ~/.aws/credentials on Linux or macOS, or C:\Users\<USERNAME>\.aws\credentials on Windows). You can also select Use custom configs to use a profile file and credentials file from another directory.

    Optionally, you can set up:

    • Per project: select to enable these connection settings only for the current project. Deselect it if you want this connection to be visible in other projects.

    • Enable connection: deselect if you want to restrict using this connection. By default, the newly created connections are enabled.

    • HTTP Proxy: select if you want to use IDE proxy settings or if you want to specify custom proxy settings.

  3. Once you fill in the settings, click Test connection to ensure that all configuration parameters are correct. Then click OK.

View databases in the editor

You can open the AWS Glue or its particular catalogs, databases, and tables in a separate tab of the editor (similarly to other storages).

  1. In the Big Data Tools tool window, select a connection to your AWS Glue or expand it to open a catalog, a database, or a table.

  2. Right-click the selected element and select Open in Editor. Alternatively, click Open in Editor button.

    This will open the selected storage, catalog, database, or table in a separate tab of your editor.

  3. On the right side of the opened tab, use the Open Editor Preview button to show and hide the details about the selected element.

AWS Glue tab

Monitor databases

Once you have established a connection to the AWS Glue server, the AWS Glue tool window becomes available. You can use it to monitor your databases, view schemas and partitions, and configure the way the data is displayed:

  • Use the Filter list to show local, foreign, or all databases.

  • Select a database and click Show and hide columns icon to show or hide columns from the database view.

  • Click the column headers to sort data by a column.

The AWS Glue tool window

In the Location column, you can click the database URL to quickly open a directory in an S3 viewer. If the needed connection does not exist, this will open the connection creation form.

Last modified: 20 January 2023