Datalore 2024.1 Help

Configure agents

You can change the default agents configuration by mounting a config file into /opt/datalore/configs/agents-config.yaml inside the Datalore container. This config has the following format:

docker: network: <name-of-agents-network> dataloreHost: <host-of-datalore-server> instances: - <instance1-definition> - <instance2-definition> ... - <instanceN-definition>

By default, <name-of-agents-network> is datalore-agents-network, and <host-of-datalore-server is datalore.

Each instance definition has the following fields:

Mandatory fields

id: basic-agent

Specifies the unique id for the instance.

default: true

Specifies whether the instance is default. Make sure exactly one instance has this parameter set to true.

label: "docker-base"

Specifies the instance name shown in the UI.

image: docker.io/jetbrains/datalore-agent:2024.1

URI of the agent docker image, changed only to specify a custom agent image

Optional fields

volumes: array of strings

Specifies the local storages mounted to the agent. Find more details in the chapter below.

pullPolicy: "IF_ABSENT"

Specifies when the image will be pulled. Acceptable values are: NEVER, ALWAYS, IF_ABSENT (default).

cpus: 1.5

Specifies how much of the available CPU resources a container can use. For example, if the host machine has two CPUs and you set cpus to 1.5, the container is guaranteed at most one and a half of the CPUs.

memory: 4000000000

Specifies the maximum amount of memory the container can use.

env: - <env1-definition> - <env2-definition> ... - <envN-definition>

Defines environment variables on the agent. Each env definition contains a name and a value. Use DATALORE_SKIP_WARMUP to skip analysis warmup on agent initialization, which contributes to faster initialization at the cost of slower code completion for some time. Use DATALORE_MAX_BLOB_SIZE for every instance to customize output limit, specified in bytes.

Example:

env: - name: DATALORE_SKIP_WARMUP value: true - name: DATALORE_MAX_BLOB_SIZE value: 10_000_000

initialPoolSize: 1

Specifies the number of pre-warmed (pooled) agents. This is the minimum number of agents Datalore will keep started and yet unassigned to any computation. When a notebook is opened, Datalore assigns one of the pooled agents to that notebook and starts another agent to refill the pool. It might be useful to set initialPoolSize to a non-zero value if initialization of your agents takes a lot of time.

shortName: "S"

Specifies the short instance name.

numCPUs: 2

Specifies the number of CPUs to display.

cpuMemoryText: "8 GB"

Specifies the agent RAM description.

numGPUs: 1

Specifies the number of GPUs to display.

gpuMemoryText: "4 GB"

Specifies the GPU memory description.

features: - "1 vCPU" - "16 GB RAM" - "1 NVIDIA GPU"

Specifies specific properties of this agent to display in the UI.

Only properties containing the "RAM", "CPU", or "GPU" substrings will be displayed in the agent selection window.

Configure local storages for agents

You can mount a local server storage to your Datalore agents by using the volumes field for the respective instance definition. See the example below:

docker: instances: - id: docker-base label: "docker-base" description: "docker-base" volumes: - "/tmp/agent:/tmp"

The volume field contains an array of strings with the following format:

  • host-src:container-dest[:options] to bind-mount a host path into the container. Make sure both host-src and container-dest are an absolute path.

  • volume-name:container-dest[:options] to bind-mount a volume managed by a volume driver into the container. Make sure container-dest is an absolute path.

options is an optional, comma-delimited list of the following:

  • nocopy disables automatic copying of data from the container path to the volume. The nocopy flag only applies to named volumes.

  • [ro|rw] mounts a volume read-only or read-write respectively. If omitted or set to rw, volumes are mounted read-write.

  • [z|Z] applies SELinux labels to allow or deny multiple containers to read and write to the same volume.

    • z: a shared content label is applied to the content. This label indicates that multiple containers can share the volume content, for both reading and writing.

    • Z: a private unshared label is applied to the content. This label indicates that only the current container can use a private volume. Labeling systems such as SELinux require proper labels to be placed on volume content that is mounted into a container. Without a label, the security system can prevent a container's processes from using the content. By default, the labels set by the host operating system are not modified.

  • [[r]shared|[r]slave|[r]private] specifies mount propagation behavior. This only applies to bind-mounted volumes, not internal volumes or named volumes. Mount propagation requires the source mount point (the location where the source directory is mounted in the host operating system) to have the correct propagation properties. For shared volumes, the source mount point must be set to shared. For slave volumes, the mount must be set to either shared or slave.

Last modified: 26 January 2024