Kubernetes 运算符：在 Kubernetes 集群中部署 TeamCity 服务器

Kubernetes 提供了许多功能和工具，帮助实现应用程序的稳定、可靠、可扩展部署，以及零停机升级。

手动部署

在 Kubernetes 集群中部署服务器时，以下 TeamCity 功能可能会影响最终方案与整体策略。

多节点设置

与所有持续运行的进程一样，pod 和单独的容器迟早会失败或被逐出。如果应用程序运行在单个 pod 中，这些情况会导致整个工作流失败，尤其在 Kubernetes 管理员无法及时发现并手动替换故障 pod 的情况下。

为应对这一风险，提升 pod 的自愈能力，并最终确保 K8S 工作流的完整性，Kubernetes 支持副本——即在一个或多个节点上同时运行的相同 pod 实例。当某个 pod 失败时，空闲副本将顶替其位置，Kubernetes 会重启故障 pod 以恢复期望的副本数量。

TeamCity 不支持完全相同的服务器实例同时运行。当存在多个实例时，TeamCity 会将其视为多节点设置的一部分，并要求分别指定主/辅助节点。因此需要创建 N 个部署，使用不同的 TEAMCITY_SERVER_OPTS 环境变量值，为 TeamCity 节点分配正确的节点职责。

共享资源

生产环境下应确保所有 TeamCity 节点都能访问以下内容：

一个存储构建历史、用户和构建结果等的外部数据库。
一个存储全部配置文件、服务器设置及其它关键数据的数据目录。

数据目录必须采用 NFS/SMB 卷，以确保多个虚拟机（K8S 节点）可用。

示例 1：测试设置

下述清单展示了包含一个 TeamCity 节点及一个外部数据库的简单测试配置。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: teamcity-server
spec:
  replicas: 1 #always 1
  selector:
    matchLabels:
      app: teamcity-server
  template:
    metadata:
      labels:
        app: teamcity-server
    spec:
      containers:
        - name: teamcity-server
          image: jetbrains/teamcity-server:latest
          env:
            - name: TEAMCITY_SERVER_OPTS
              value: -Dteamcity.server.rootURL=http://$(POD_NAME).$(POD_NAMESPACE)
            - name: TEAMCITY_DATA_PATH
              value: /data/teamcity_server/datadir
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8111
          volumeMounts:
            - name: teamcity-data
              mountPath: /data/teamcity_server/datadir
      volumes:
        - name: teamcity-data
          emptyDir: {}

   

示例 2：在一台虚拟机上部署两个 TeamCity 副本

下面的示例实现了更稳定的解决方案：包含两个副本和一个外部数据库。不过，为了 Simplify 设置，二者都运行在同一台虚拟机上，并使用本地目录作为 TeamCity 数据目录。

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-initdb-config
data:
  init.sql: |
    CREATE DATABASE IF NOT EXISTS teamcity;
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      volumes:
        - name: mysql-initdb
          configMap:
            name: mysql-initdb-config
      containers:
        - image: mysql:8
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-initdb
              mountPath: /docker-entrypoint-initdb.d
---
#based on database.properties
apiVersion: v1
data:
  connectionProperties.password: cGFzc3dvcmQ=
  connectionProperties.user: cm9vdA==
  connectionUrl: amRiYzpteXNxbDovL215c3FsLmRlZmF1bHQ6MzMwNi90ZWFtY2l0eQ==
kind: Secret
metadata:
  name: database-properties
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: teamcity-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: teamcity-server
  template:
    metadata:
      labels:
        app: teamcity-server
    spec:
      containers:
        - name: teamcity-server
          image: jetbrains/teamcity-server:latest
          env:
            - name: TEAMCITY_DATA_PATH
              value: /data/teamcity_server/datadir
            - name: TEAMCITY_DB_USER
              valueFrom:
                secretKeyRef:
                  key: connectionProperties.user
                  name: database-properties
            - name: TEAMCITY_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: connectionProperties.password
                  name: database-properties
            - name: TEAMCITY_DB_URL
              valueFrom:
                secretKeyRef:
                  key: connectionUrl
                  name: database-properties
          ports:
            - containerPort: 8111
          volumeMounts:
            - name: teamcity-data
              mountPath: /data/teamcity_server/datadir
      volumes:
        - name: teamcity-data
          persistentVolumeClaim:
            claimName: teamcity-node-volume
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: teamcity-node-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard

   

示例 3：多个节点分工不同

下面的示例分别为主节点和辅助节点配置了两个独立的部署层。每个部署的 env 部分会设置 TEAMCITY_SERVER_OPTS 环境变量，以为每个节点分配不同职责。

#database deployment
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-initdb-config
data:
  init.sql: |
    CREATE DATABASE IF NOT EXISTS teamcity;
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      volumes:
        - name: mysql-initdb
          configMap:
            name: mysql-initdb-config
      containers:
        - image: mysql:8
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-initdb
              mountPath: /docker-entrypoint-initdb.d
---
apiVersion: v1
data:
  connectionProperties.password: qwerty
  connectionProperties.user: johndoe
  connectionUrl: foobar
kind: Secret
metadata:
  name: database-properties
---
apiVersion: apps/v1
kind: Deployment #Deployment #1, main TeamCity node
metadata:
  name: teamcity-server-main
spec:
  replicas: 1 #always 1
  selector:
    matchLabels:
      app: teamcity-server
  template:
    metadata:
      labels:
        app: teamcity-server
    spec:
      containers:
        - name: teamcity-server
          image: jetbrains/teamcity-server:latest
          env:
            - name: TEAMCITY_SERVER_OPTS
              value: -Dteamcity.server.nodeId=main-node -Dteamcity.server.rootURL=http://$(POD_NAME).$(POD_NAMESPACE)
                -Dteamcity.server.responsibilities=MAIN_NODE,CAN_PROCESS_BUILD_MESSAGES,CAN_CHECK_FOR_CHANGES,CAN_PROCESS_BUILD_TRIGGERS,CAN_PROCESS_USER_DATA_MODIFICATION_REQUESTS
            - name: TEAMCITY_DATA_PATH
              value: /data/teamcity_server/datadir
            - name: TEAMCITY_DB_USER
              valueFrom:
                secretKeyRef:
                  key: connectionProperties.user
                  name: database-properties
            - name: TEAMCITY_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: connectionProperties.password
                  name: database-properties
            - name: TEAMCITY_DB_URL
              valueFrom:
                secretKeyRef:
                  key: connectionUrl
                  name: database-properties
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8111
          volumeMounts:
            - name: teamcity-data
              mountPath: /data/teamcity_server/datadir
      volumes:
        - name: teamcity-data
          persistentVolumeClaim:
            claimName: teamcity-node-volume
---
apiVersion: apps/v1
kind: Deployment #Deployment #2, secondary TeamCity node
metadata:
  name: teamcity-server-additional-node-0
spec:
  replicas: 1 #always 1
  selector:
    matchLabels:
      app: teamcity-server
  template:
    metadata:
      labels:
        app: teamcity-server
    spec:
      containers:
        - name: teamcity-server
          image: jetbrains/teamcity-server:latest
          env:
            - name: TEAMCITY_SERVER_OPTS
              value: -Dteamcity.server.nodeId=secondary-node-0 -Dteamcity.server.rootURL=http://$(POD_NAME).$(POD_NAMESPACE)
                -Dteamcity.server.responsibilities=CAN_PROCESS_BUILD_MESSAGES,CAN_CHECK_FOR_CHANGES,CAN_PROCESS_BUILD_TRIGGERS,CAN_PROCESS_USER_DATA_MODIFICATION_REQUESTS
            - name: TEAMCITY_DATA_PATH
              value: /data/teamcity_server/datadir
            - name: TEAMCITY_DB_USER
              valueFrom:
                secretKeyRef:
                  key: connectionProperties.user
                  name: database-properties
            - name: TEAMCITY_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: connectionProperties.password
                  name: database-properties
            - name: TEAMCITY_DB_URL
              valueFrom:
                secretKeyRef:
                  key: connectionUrl
                  name: database-properties
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8111
          volumeMounts:
            - name: teamcity-data
              mountPath: /data/teamcity_server/datadir
      volumes:
        - name: teamcity-data
          persistentVolumeClaim:
            claimName: teamcity-node-volume
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: teamcity-node-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard

   

通过 Helm 部署 TeamCity 服务器

Helm 是 Kubernetes 的软件包管理器，可用于模板、版本编排、配置及复用原始 YAML 清单。这样可以避免复制粘贴 YAML，方便管理多个环境（开发、暂存、生产等），并能更安全地更新或回滚部署。

这些优势在部署 TeamCity 时同样适用。例如，上述最适用于真实应用程序的示例 3 展示了如下问题：

重复——主/辅助 TeamCity 部署有约 90% 内容相同。
硬编码值——镜像标记、节点 ID、职责、存储大小等均为原始值。
强耦合——数据库配置、密钥、PVC 与应用部署混杂在一起。
可扩展性差——每增加一个辅助节点都需复制一个新的部署。

为解决这些问题，可以将清单向下拆分为多个独立文件。您的 Helm Chart 结构可如下所示：

teamcity/
├── Chart.yaml
├── values.yaml             # The majority of your edits will happen here
├── templates/
│   ├── mysql/              # The database template
│   │   ├── service.yaml
│   │   ├── deployment.yaml
│   │   └── configmap.yaml
│   ├── teamcity/           # Template for TeamCity nodes
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   ├── secrets.yaml        # Sensitive data
│   ├── pvc.yaml            # The storage template
│   └── _helpers.tpl

  

values.yaml

该文件将大部分变量和唯一值单独分层存放。可以修改这些值（如 TeamCity 服务器镜像标记），而无需编辑全局清单。

teamcity:
  image: jetbrains/teamcity-server
  tag: latest
  dataPath: /data/teamcity_server/datadir

  mainNode:
    enabled: true
    nodeId: main-node
    responsibilities:
      - MAIN_NODE
      - CAN_PROCESS_BUILD_MESSAGES
      - CAN_CHECK_FOR_CHANGES
      - CAN_PROCESS_BUILD_TRIGGERS
      - CAN_PROCESS_USER_DATA_MODIFICATION_REQUESTS

  secondaryNodes:
    - name: secondary-node-0
      responsibilities:
        - CAN_PROCESS_BUILD_MESSAGES
        - CAN_CHECK_FOR_CHANGES
        - CAN_PROCESS_BUILD_TRIGGERS
        - CAN_PROCESS_USER_DATA_MODIFICATION_REQUESTS

database:
  secretName: database-properties

    

通过在 secondaryNodes 部分添加新条目，即可方便地扩展 TeamCity 方案，无需让清单因非唯一内容而膨胀……

secondaryNodes:
  - name: secondary-node-0
      responsibilities:
        ...
  - name: secondary-node-1
      responsibilities:
        ...
  - name: secondary-node-2
    responsibilities:
      ...

    

…以及执行 helm upgrade teamcity ./teamcity 命令。

模板/teamcity/部署.yaml

配置好 values.yml 文件后，核心节点部署清单可精简如下：

# ---------------------------------------------------------
# Main TeamCity node
# ---------------------------------------------------------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: teamcity-server-main
spec:
  replicas: 1
  selector:
    matchLabels:
      app: teamcity-server
      node: main
  template:
    metadata:
      labels:
        app: teamcity-server
        node: main
    spec:
      containers:
        - name: teamcity-server
          image: {{ .Values.teamcity.image }}:{{ .Values.teamcity.tag }}
          env:
            - name: TEAMCITY_SERVER_OPTS
              value: >
                -Dteamcity.server.nodeId={{ .Values.teamcity.mainNode.nodeId }}
                -Dteamcity.server.rootURL=http://$(POD_NAME).$(POD_NAMESPACE)
                -Dteamcity.server.responsibilities={{ join "," .Values.teamcity.mainNode.responsibilities }}
            - name: TEAMCITY_DATA_PATH
              value: {{ .Values.teamcity.dataPath }}
            - name: TEAMCITY_DB_USER
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.database.secretName }}
                  key: connectionProperties.user
            - name: TEAMCITY_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.database.secretName }}
                  key: connectionProperties.password
            - name: TEAMCITY_DB_URL
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.database.secretName }}
                  key: connectionUrl
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8111
          volumeMounts:
            - name: teamcity-data
              mountPath: {{ .Values.teamcity.dataPath }}
      volumes:
        - name: teamcity-data
          persistentVolumeClaim:
            claimName: teamcity-node-volume

# ---------------------------------------------------------
# Secondary TeamCity nodes (rendered from values.yaml)
# ---------------------------------------------------------
{{- range $index, $node := .Values.teamcity.secondaryNodes }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: teamcity-server-{{ $node.name }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: teamcity-server
      node: {{ $node.name }}
  template:
    metadata:
      labels:
        app: teamcity-server
        node: {{ $node.name }}
    spec:
      containers:
        - name: teamcity-server
          image: {{ $.Values.teamcity.image }}:{{ $.Values.teamcity.tag }}
          env:
            - name: TEAMCITY_SERVER_OPTS
              value: >
                -Dteamcity.server.nodeId={{ $node.name }}
                -Dteamcity.server.rootURL=http://$(POD_NAME).$(POD_NAMESPACE)
                -Dteamcity.server.responsibilities={{ join "," $node.responsibilities }}
            - name: TEAMCITY_DATA_PATH
              value: {{ $.Values.teamcity.dataPath }}
            - name: TEAMCITY_DB_USER
              valueFrom:
                secretKeyRef:
                  name: {{ $.Values.database.secretName }}
                  key: connectionProperties.user
            - name: TEAMCITY_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ $.Values.database.secretName }}
                  key: connectionProperties.password
            - name: TEAMCITY_DB_URL
              valueFrom:
                secretKeyRef:
                  name: {{ $.Values.database.secretName }}
                  key: connectionUrl
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8111
          volumeMounts:
            - name: teamcity-data
              mountPath: {{ $.Values.teamcity.dataPath }}
      volumes:
        - name: teamcity-data
          persistentVolumeClaim:
            claimName: teamcity-node-volume
{{- end }}

    

无论 TeamCity 辅助节点数量如何，这一单一定义都无需变更。 Helm 会自动将该文件渲染为多个 Kubernetes 部署：主节点一个，加上 N 个辅助节点部署，具体由 range 针对 values.yml 文件中声明的列表生成。

模板/mysql/部署.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "teamcity.mysqlName" . }}
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: {{ .Values.mysql.image }}
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: {{ .Values.mysql.rootPassword | quote }}
          ports:
            - containerPort: 3306

    

模板/mysql/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-initdb-config
data:
  init.sql: |
    CREATE DATABASE IF NOT EXISTS {{ .Values.mysql.database }};

    

模板/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: teamcity-node-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: {{ .Values.persistence.size }}
  storageClassName: {{ .Values.persistence.storageClass }}

    

还可以利用 _helpers.tpl 定义的 Go 模板持续优化该方案。例如，可以创建通用容器配置……

{{- define "teamcity.containerBase" -}}
image: {{ .Values.teamcity.image }}:{{ .Values.teamcity.tag }}
ports:
  - containerPort: 8111
volumeMounts:
  - name: teamcity-data
    mountPath: {{ .Values.teamcity.dataPath }}
env:
  - name: TEAMCITY_DATA_PATH
    value: {{ .Values.teamcity.dataPath }}
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
{{ include "teamcity.dbEnv" . | indent 2 }}
{{- end -}}

  

……以及 TEAMCITY_SERVER_OPTS 构建器……

{{- define "teamcity.serverOpts" -}}
-Dteamcity.server.nodeId={{ .nodeId }}
-Dteamcity.server.rootURL=http://$(POD_NAME).$(POD_NAMESPACE)
-Dteamcity.server.responsibilities={{ join "," .responsibilities }}
{{- end -}}

  

……让 deployment.yaml 文件更加简洁：

containers:
  - name: teamcity-server
  {{ include "teamcity.containerBase" . | indent 4 }}
env:
  - name: TEAMCITY_SERVER_OPTS
    value: >
      {{ include "teamcity.serverOpts" (dict
        "nodeId" .Values.teamcity.mainNode.nodeId
        "responsibilities" .Values.teamcity.mainNode.responsibilities
      ) }}

  

2026年 8月 6日