JetBrains Space Help

Share Data Between Steps

Sometimes it is necessary to pass an execution context (for example, an access token, script run results, and so on) from one Automation step to another. Depending on the data type, you can share it using a parameter or a file.

Sharing parameters between steps

For this purpose, all steps in a job have access to a common parameter storage. To access the storage, you should use the parameters API. The API is available only inside the kotlinScript block: the container you use to run the step must include JRE/JDK 9 or later.

job("Reuse params") { container("openjdk:11") { kotlinScript { api -> api.parameters["myName"] = "Anna" } } container("openjdk:11") { kotlinScript { api -> println("Hi ${api.parameters["myName"]}!") } } }

Note that you can remove parameters from the storage by using parameters().remove(key: String), where key is a parameter name.

Sharing files between steps

In some cases, a file created in one step may be required in another step. For example, files generated during the build may be required by tests executed in a separate container. For this purpose, Automation provides a file share: a separate external storage that is mounted to a container.

Accessing file share directly

The file share is mounted to a step container at $mountDir/share (in general case, it is /mnt/space/share). You can work with this directory directly using container commands. In the following example, the first container creates file.txt in the file share, and the second container shows the file share contents.

job("Example") { container("ubuntu") { shellScript { content = "touch $mountDir/share/file.txt" } } container("ubuntu") { shellScript { content = "ls -la $mountDir/share" } } }

Accessing file share in Kotlin code

Alternatively, there's a more high-level approach of accessing the file share: If your step runs Kotlin code in the kotlinScript block, you can use the special file-sharing API. Note that the container must include JVM/JDK 8 or later.

import* job("Share files") { container("openjdk:11") { kotlinScript { api -> val path = "~/file.txt" val content = "Hello World!" val sharedFile = File(path).apply { parentFile.mkdirs() createNewFile() writeText(content) } api.fileShare().put(sharedFile, "file.txt") } } container("openjdk:11") { kotlinScript { api -> api.fileShare().locate("file.txt")?.let { println(it.readText()) } } } }

In this script:

  • The first container creates file.txt in the user's directory.

  • fileShare().put() copies the file to the file share. Note that you can change the file name during copying.

  • The second container uses fileShare().locate() to get the path to the file and then it prints the contents of the file.

  • Because the steps (containers) run sequentially, file.txt is guaranteed to be available in the file share when the second container starts.

  • import* imports the package as the script uses Java functions to work with files.

File share size limit

The maximum capacity of the file share is limited to 2 GB per job. Let's use an example to see how this limit works:

  • A job consists of two steps (i.e. containers): A and B.

  • The steps run sequentially one after another.

  • None of the steps is allowed to allocate more than 2 GB in the file share.

  • Step A allocates 1.5 GB.

  • Let's suppose step B allocates 1 GB (the step is allowed to do this as it is less than the 2 GB limit). The size of the file share is 2.5 GB now.

  • If at the end of step B, the file share size is still 2.5 GB, the job will fail.

  • If step B cleans the file share so that its size doesn't exceed the 2 GB limit, the job will finish successfully.

You can remove files from the file share either directly (using container commands) or using the API: fileShare().remove().

Monthly limit on file share usage

Automation calculates file share usage twice for each step: First, when a file share volume is mounted to a step, and, for the second time, when the step finishes working (Automation calculates the size of new data allocated by the step). The monthly limit on file share usage is 200 GB for all subscription plans. Automation will warn you once the file share usage limit is reached.

Let's use an example to see how this limit works:

  • A job consists of three steps (i.e. containers): A, B, and C.

  • The steps run sequentially one after another.

  • Step A allocates 1 GB in the file share. Now, the file share size is 1 GB, file share usage is 1 GB.

  • Automation starts step B and mounts the file share to the step container. Even if step B does nothing with the file share, this is treated as read access to the file share. Now, the file share size is 1 GB, file share usage is 2 GB.

  • Step B allocates 0.5 GB in addition to the existing data. Now, the file share size is 1.5 GB, file share usage is 2.5 GB.

  • Step C gets the file share and does not put any new data to it. Now, the file share size is 1.5 GB, file share usage is 5 GB.

  • Job finishes its work with the resulting 5 GB file share usage. This means that you can run such a job only 40 times a month.

Parallel steps and file sharing

By default, steps in a job run sequentially: the next container starts only after the previous container finishes its work. This prevents conflicts in the file share. But what if two steps (A and B) run in parallel and put the same file in the file share? This behavior is undetermined: the file share will contain the result of either A or B. Consider an example:

job("Example") { parallel { container("ubuntu") { shellScript { content = "echo Hello > /mnt/space/share/file.txt" } } container("ubuntu") { shellScript { content = "echo World > /mnt/space/share/file.txt" } } } }

Both containers run in parallel: the first container writes Hello to file.txt while the second one writes World to the same file. When the job finishes, file.txt will contain either Hello or World but neither Hello World nor World Hello.

File attributes and file sharing

File attributes are preserved when a file is copied to/from the file share.

File-sharing API

The fileShare() function provides access to the file sharing API:

Method or Property Description
location: Path (Read-only) Path to the file share (by default, /mnt/space/share).
put(file: File): Path

Copies file to the file share. Using relativePath, you can set a new file name and specify its relative path inside the file share (/mnt/space/share).

Returns full path to the file.

put(file: File, relativePath: String): Path
locate(relativePath: String): File?

Locates a file in the file share by its relativePath.

Returns file path.

remove(relativePath: String): Boolean

Deletes a file from the file share by its relativePath.

Returns true if the file was successfully deleted.

Last modified: 03 December 2020