JetBrains Space Help

External Workers

An external worker is a lightweight agent that you can run on your own host machine on Windows, Linux, and macOS. The external worker connects to Space Automation, gets jobs and source code, runs the jobs, and reports results back to Space.

External workers let you perform CI/CD workflows that are not possible with containers. For example:

  • Building full .NET Framework applications on Windows.

  • Using specific hardware. For example, building iOS and macOS applications on Apple hardware, running ML tasks on GPUs, and so on.

  • Using more powerful hardware configurations (the current limits for containers in Automation are 4 vCPU and 16GB RAM).

How does it work

  • The external worker agent is a Java application that is distributed either as a .zip archive or as a part of a specially-prepared Docker image. Learn more

  • After you register an external worker in Space, you should run the agent on the host machine. The running agent will regularly send its status to Space.

  • The agent uses pull communication with Space: The agent initiates the communication and Space only responds to the incoming requests. This means that no specific network configuration is required on the agent side. The only condition is access to your Space instance.

  • To run a job on the external worker, you should use the host blocks for job steps:

    job("Hello from External Worker") { host("Run echo") { shellScript { content = """ echo "Hello World!" """ } } }
  • When a job with at least one host step is triggered, Automation will randomly choose a worker suitable for the job. After the suitable worker is found, Automation initiates job run.

  • The worker agent:

    • creates a temporary directory on the host machine,

    • downloads the project source code including the Automation script,

    • and runs the job.

    Once the job is finished, the agent reports results to Automation and deletes the temporary directory.

Requirements to a host machine

The host machine which you want to use as an external worker must have the following software installed:

GitGit is used only to clone the project source code, so, basically, it can be any Git version.
Docker ComposeIf a job has not only host but also container steps, the host machine must have Docker and Docker Compose installed.

Important: The paths to Git and Docker must be specified in the system PATH.

Directory structure

Automation stores project data inside a temporary directory on this worker.

/ ├─── jetbrains/space/automation/worker/data // default {parent-dir} │ └─── {temp-dirs} // auto-generated subdirectories │ ├─── share // file share │ ├─── system // non-user system files │ └─── work │ ├─── {git-repo-name} // cloned git repository (working directory) │ ├─── ... // other project repos (only if specified in the script) │ └─── ... ...

In more detail:

  • The worker creates a parent directory, by default, /jetbrains/space/automation/worker/data. You can change the default location using the --dataDir agent argument.

  • By default, the project sources location is {parent-dir}/{temp-dir}/work/{git-repo-name}. Here {git-repo-name} stands for the project's Git repository name. For details on how to work with project repositories, refer to Check out Source Code.

  • The {parent-dir} also contains the share directory: external storage used for file sharing.

  • After the job is finished, {temp-dir} is deleted.

Adding external workers

  1. In Administration, open the Automation Workers page.

  2. Click Add worker.

  3. Specify worker Name and Tags. The Tags let you route jobs to specific workers. Automation will run jobs marked with a specific tag only on workers with the same tag. Learn more

    After you click Add worker, the worker will be created and you will see the worker settings page. The settings include the assigned tags, OS and available resources data (the worker will send it to Space by itself), and an authorization token.

  4. Your next task is to run the worker agent on a host machine. Before it happens the worker will be in not initialized state. The instructions on how to do it are provided on the worker settings page under Worker installation instructions: select a worker type and follow the instructions:

    • Docker for Windows / Linux / macOS: the host machine is a Docker container running on Windows, Linux, or macOS.

    • Binary for Windows / Linux / macOS: the host machine is a virtual or hardware machine running on Windows, Linux, or macOS.

    Typically, all you need to do is unzip the archive, make the script file executable and run the worker. For example, on Linux:

    unzip ./space-automation-worker.zip && \ chmod +x ./worker.sh && \ ./worker.sh start \ --serverUrl https://mycompany.jetbrains.space \ --token abc1234

    Available worker script arguments (note that you can set/get them also as environment variables):

    ArgumentEnv variableDescription
    --serverUrlSPACE_WORKER_SERVERURLA URL of your Space instance.
    --tokenSPACE_WORKER_TOKENAn authorization token issued to the worker. The token is shown only once during the worker registration. If you lose it, you will have to generate a new one.
    --dataDirSPACE_WORKER_DATADIRSets the path to the directory where the script will download all required data: project sources, file share, and Automation-specific data. To get this path in your Automation script, use the mountDir property.
    --cpu, --memSPACE_WORKER_CPU, SPACE_WORKER_MEMSpecifies how much CPU and RAM are available on the host machine. These parameters do not anyhow limit how much resources the agent is allowed to consume. They are used only to help Automation select a suitable worker for a job based on the amount of available resources. Learn more

    After you start a worker, its status must change to healthy:

    Healthy

Using external workers in steps

In order for Automation to run a job on a worker, the job must contain at least one host step.

job("Build solution") { host("Run msbuild") { shellScript { interpreter = "cmd" content = """ C:\"Program Files (x86)"\MSBuild\14.0\Bin\MsBuild.exe MySolution.sln """ } requirements { type = OSType.Windows } } }

Inside the host step you can use almost the same DSL elements as inside the container step. The main difference is that inside host, you can also:

Choosing specific workers for a job

You have a number of ways to route specific jobs to specific workers: by tags, by operating system, and by available system resources. To specify the routing condition, use the requirements block inside host. You can specify all three conditions inside requirements.

If Automation cannot find a suitable worker for a job, the job will switch to the Blocked state.

No external workers

Routing jobs by tags

Tagging is one of the ways to route a job to a specific worker: Both the worker and the job must be marked with the same tag. See the details below.

  1. In Administration | Automation Workers, choose the required worker.

  2. Click Edit.

  3. In Tags, specify a unique tag name.

    External worker settings
  4. Open the required .space.kts file and specify the same tag in the requirements block of a particular host. For example:

    job("Hello from host") { host("Run echo") { shellScript { content = """ echo Hello World! """ } requirements { workerTags("Pool1") } } }

Routing jobs by operating system

The worker agent automatically reports the operating system of its host machine to Space. To choose the suitable worker based on the operating systems, specify the required OS in the script:

requirements { os { // the possible values are // Windows, macOS, Linux type = OSType.Windows } }

Routing jobs by available resources

The worker agent automatically reports the system resources (CPU cores and RAM) of its host machine to Space. To choose the suitable worker based on host machine resources, specify the required resources in the script:

requirements { resources { // the job can run on any worker that // has 1 CPU core and 1GB RAM (or more) minCpu = 1.cpu minMemory = 1.gb } }

Note that the worker agent does not guarantee that all these resources will be available to the job: It knows nothing about how the resources are consumed by other processes running on the host machine.

You can manually set the available system resources for a particular worker. This may help, for example, if you don't want to run "heavy" jobs on a particular worker. To set the available resources, run the worker agent with --cpu (set in mCPU) and --mem (set in MB) arguments. For example:

./worker.sh start \ --serverUrl https://mycompany.jetbrains.space \ --token abc1234 \ --cpu 2000 \ --mem 8000

In the example above, regardless of real system resources, the worker will report to Space that the system has 2 CPU cores and 8GB RAM available: if a host in a job requires more resources, the worker will be considered as not suitable for this job. Note that these values do not anyhow prevent the worker agent from consuming more resources on the machine: They just help Space to choose the most suitable worker for a job.

Running container steps in external workers

External workers let you run not only host steps but container steps as well. To be able to run container steps, a worker must have Docker and Docker Compose installed.

Automation will try to run a job with container steps on an external worker in one of the following cases:

  • A job has both container and host steps.

  • A container step has the requirements section. Note that only selection by workerTags is available inside container.

  • A job has only container steps (with no requirements) and the Default resource to run jobs parameter in Administration | Automation is set to Automation Workers.

For example:

job("Example") { container(image = "openjdk:11") { // ... requirements { workerTags("Pool 1") } } }
Learn more

Suspending a worker

You can temporarily suspend a worker: The worker agent keeps running on the host machine, but Automation will not route jobs to the suspended worker. If the worker still has running jobs, it will first finish the jobs and only then switch to the suspended state.

To suspend a worker

  1. Find the required worker in Administration | Automation Workers.

  2. In the worker settings, click Suspend.

To activate the worker back

  1. Find the required worker in Administration | Automation Workers.

  2. In the worker settings, click Activate.

Deleting a worker

  1. Find the required worker in Administration | Automation Workers.

  2. In the worker settings, click Delete.

Note that deleting a worker is an instant operation: If the worker still has running jobs, the jobs will be terminated.

Viewing the history of executed steps

All steps executed by a particular external worker are saved in the worker history.

  1. Find the required worker in Administration | Automation Workers.

  2. In the worker settings, open the Activity tab.

Worker states

StateDescription
not initializedThe worker has never connected to Space before: This is the initial worker state after you register it in Space
healthyThe worker is up and running
unhealthyThe worker is low on disk space: Automation will not route jobs to this worker
suspendedThe worker was manually suspended
graceful shutdownThe worker is finishing the currently running jobs, then the worker agent process will be terminated. This happens when you terminate the agent process with SIGTERM (for example, when you kill the process)
disconnectedAutomation has no connection with the worker. This may happen, for example, if the worker agent is not running on the host machine, the machine is turned off or has no Internet connection
Last modified: 27 August 2021