JetBrains Space Help

Share Files Between Steps

In some cases, a file created in a step might be required in another step. For example, files generated during the build might be required by tests executed in a separate container. For this purpose, Automation provides a file share: separate external storage that is mounted to a host machine.

Accessing file share directly

If a host machine is a container, by default, the file share is mounted at $mountDir/share. Learn more

If a host machine is a self-hosted worker, by default, the file share is mounted at jetbrains/space/automation/worker/data/{temp-dir}/share. Learn more

To get the path to the file share, use the JB_SPACE_FILE_SHARE_PATH environment variable.

You can work with the file share directory directly using system commands. In the following example, the first container creates file.txt in the file share, and the second container shows the file share contents.

job("Example") { container(displayName = "Create file", image = "ubuntu") { shellScript { content = "touch ${'$'}JB_SPACE_FILE_SHARE_PATH/file.txt" } } container(displayName = "Show file", image = "ubuntu") { shellScript { content = "ls -la ${'$'}JB_SPACE_FILE_SHARE_PATH" } } }

Access file share in Kotlin code

Alternatively, there's a more high-level approach of accessing the file share: If your step runs Kotlin code in the kotlinScript block, you can use the special file-sharing API.

// java.io package provides functions for working with files import java.io.* job("Share files") { container(displayName = "Create file", image = "amazoncorretto:17-alpine") { kotlinScript { api -> // create a file in the user's directory val path = "~/file.txt" val content = "Hello World!" val sharedFile = File(path).apply { parentFile.mkdirs() createNewFile() writeText(content) } // copy the file to the file share api.fileShare().put(sharedFile, "file.txt") } } container(displayName = "Show file", image = "amazoncorretto:17-alpine") { kotlinScript { api -> // Get the file from the file share and print file contents. // Because the steps (containers) run sequentially, // file.txt is guaranteed to be available in the file share. api.fileShare().locate("file.txt")?.let { println(it.readText()) } } } }

File share size limit

The maximum capacity of the file share is limited to 2 GB per job. Let's use an example to see how this limit works:

  • A job consists of two steps (i.e. containers): A and B.

  • The steps run sequentially one after another.

  • None of the steps is allowed to allocate more than 2 GB in the file share.

  • Step A allocates 1.5 GB.

  • Let's suppose step B allocates 1 GB (the step is allowed to do this as it is less than the 2 GB limit). The size of the file share is 2.5 GB now.

  • If at the end of step B, the file share size is still 2.5 GB, the job will fail.

  • If step B cleans the file share so that its size doesn't exceed the 2 GB limit, the job will finish successfully.

You can remove files from the file share either directly (using container commands) or using the API: fileShare().remove().

Monthly limit on file share usage

Automation calculates file share usage twice for each step: First, when a file share volume is mounted to a step, and, for the second time, when the step finishes working (Automation calculates the size of new data allocated by the step). The monthly limit on file share usage is 200 GB for all subscription plans. Automation will warn you once the file share usage limit is reached.

Let's use an example to see how this limit works:

  • A job consists of three steps (i.e. containers): A, B, and C.

  • The steps run sequentially one after another.

  • Step A allocates 1 GB in the file share. Now, the file share size is 1 GB, file share usage is 1 GB.

  • Automation starts step B and mounts the file share to the step container. Even if step B does nothing with the file share, this is treated as read access to the file share. Now, the file share size is 1 GB, file share usage is 2 GB.

  • Step B allocates 0.5 GB in addition to the existing data. Now, the file share size is 1.5 GB, file share usage is 2.5 GB.

  • Step C gets the file share and does not put any new data to it. Now, the file share size is 1.5 GB, file share usage is 5 GB.

  • Job finishes its work with the resulting 5 GB file share usage. This means that you can run such a job only 40 times a month.

Parallel steps and file sharing

By default, steps in a job run sequentially: the next container starts only after the previous container finishes its work. This prevents conflicts in the file share. But what if two steps (A and B) run in parallel and put the same file in the file share? This behavior is undetermined: the file share will contain the result of either A or B. Consider an example:

job("Example") { parallel { container(displayName = "Write to file", image = "ubuntu") { shellScript { content = "echo Hello > /mnt/space/share/file.txt" } } container(displayName = "Write to same file", image = "ubuntu") { shellScript { content = "echo World > /mnt/space/share/file.txt" } } } }

Both containers run in parallel: the first container writes Hello to file.txt while the second one writes World to the same file. When the job finishes, file.txt will contain either Hello or World but neither Hello World nor World Hello.

File attributes and file sharing

File attributes are preserved when a file is copied to/from the file share.

File-sharing API

The fileShare() function provides access to the file sharing API:

Method or Property

Description

location: Path

(Read-only) Path to the file share (by default, /mnt/space/share).

put(file: File): Path

Copies file to the file share. Using relativePath, you can set a new file name and specify its relative path inside the file share (/mnt/space/share).

Returns full path to the file.

put(file: File, relativePath: String): Path

locate(relativePath: String): File?

Locates a file in the file share by its relativePath.

Returns file path.

remove(relativePath: String): Boolean

Deletes a file from the file share by its relativePath.

Returns true if the file was successfully deleted.

Last modified: 18 October 2022