Bio-OS Deployment

Before starting the deployment of Bio-OS, ensure that you have obtained the open-source Helm Charts for Bio-OS (https://github.com/Bio-OS/helm-charts) in your environment. Bio-OS supports deployment on both ARM64 and x86_64 architectures. In addition, Bio-OS requires Kubernetes as the underlying platform, and you may choose either the community edition of Kubernetes or Minikube. This documentation is based on the x86 architecture; if your environment uses ARM architecture, please make sure to select the ARM-compatible images and tools for installation. (See Prerequisite.

一. K8s Environment Preparation

1.1 Start Minikube

It is assumed here that Minikube has already been installed. For installation instructions, see: Minikube Start.

Here, Minikube is used as an example to build a local environment, while for production environments it is recommended to use Kubespray for high-availability deployment.

minikube start \
    --image-mirror-country=cn \
    --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
    --force \
    --addons ingress \
    --cni flannel \
    --kubernetes-version 1.24.15 \
    --nodes 4 \
    --ports 80,443 \
    --service-cluster-ip-range '10.255.0.0/16' \# 可自行修改
    --subnet '10.254.0.0/16' # 可执行修改

# Shortcut configuration
echo 'alias kubectl="minikube kubectl -- "' >> ~/.bashrc
source ~/.bashrc

1.2 Helm Deployment

Install the Helm package (note: Helm version must be greater than 2.0; if Helm is already installed, you may skip this step). For the ARM architecture, update the Helm download URL to:https://get.helm.sh/helm-v3.12.0-linux-arm64.tar.gz .

wget https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz
tar xvf helm-v3.12.0-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin

1.3 NFS-Server Deployment

• Here, Ubuntu is used as an example to deploy the NFS service, where the variable NFS_PATH corresponds to the NFS storage directory. (Execute as root user)

#!/bin/bash
#
# Desc: Install NFS Server. OS: Ubuntu
# 
# Install NFS Server
# 
# 
function installNFSServer() {
    apt update && apt install -y nfs-kernel-server nfs-common
}

# Config NFS Server
function configNFS() {
    NFS_PATH=/nfs
    mkdir -p ${NFS_PATH}/bioos-storage && chmod -R 777 ${NFS_PATH}
    chown -R nobody:nogroup ${NFS_PATH}
    echo "${NFS_PATH} *(insecure,rw,sync,root_squash,no_subtree_check,all_squash)" >> /etc/exports
    systemctl restart nfs-server
    systemctl restart rpcbind
    exportfs -arv
    showmount -e localhost
}

# Set up auto-start on boot.
function autoStart() {
    systemctl enable nfs-server
    systemctl enable rpcbind
}

function main() {
    installNFSServer
    configNFS
    autoStart
}

main

Post-installation check list：

root@registry:/home/vagrant# systemctl status nfs-server
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
    Drop-In: /run/systemd/generator/nfs-server.service.d
             └─order-with-mounts.conf
     Active: active (exited) since Thu 2023-05-18 21:33:05 CDT; 10s ago
    Process: 35987 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 35988 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS)
   Main PID: 35988 (code=exited, status=0/SUCCESS)
        CPU: 6ms

May 18 21:33:05 registry systemd[1]: Starting NFS server and services...
May 18 21:33:05 registry systemd[1]: Finished NFS server and services...

1.4 Install the NFS-CSI storage service

# Doc：https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/charts/README.md
# Add nfs-csi helm repo
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts

# Create NFS Secret
kubectl -n kube-system create secret generic mount-options --from-literal mountOptions="nfsvers=3,hard"

# install nfs-csi helm chart
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs \
        --namespace kube-system \
        --version v4.4.0 \
        --set image.nfs.repository=dyrnq/nfsplugin \
        --set image.csiProvisioner.repository=dyrnq/csi-provisioner \
        --set image.csiSnapshotter.repository=dyrnq/csi-snapshotter \
        --set image.livenessProbe.repository=dyrnq/livenessprobe \
        --set image.nodeDriverRegistrar.repository=dyrnq/csi-node-driver-registrar \
        --set image.externalSnapshotter.repository=dyrnq/snapshot-controller

Install the StorageClass. Since nfs-csi does not provide an SC resource by default, you need to manually install one (reference: https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/deploy/example/storageclass-nfs.yaml). Copy the following command and execute it in the shell.

cat << EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: ${NFS-SERVER-IPADDRESS}   # 这里填 NFS Server 地址
  share: /nfs
  csi.storage.k8s.io/provisioner-secret-name: "mount-options"
  csi.storage.k8s.io/provisioner-secret-namespace: "kube-system"
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1
  - hard

EOF

1.5 Verify that the PVC can be mounted successfully

Copy the following command and execute it in the shell.

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  storageClassName: nfs-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pvc-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: test-pvc
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage
EOF

Check the Pod status

# If the Pod status is Running and the PVC status is Bound, the NFS server is available.
root@n37-043-061:~# kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
test-pvc-pod   1/1     Running   0          8d
root@n37-043-061:~# kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-pvc   Bound    pvc-8dd031d1-2316-44b0-bd46-bdaeea7f9fd5   1Gi        RWO            nfs-csi        8d

二.Install Bio-OS services

The deployment of Bio-OS depends on MySQL, JupyterHub, and Cromwell. Before installing the Bio-OS service, you must first install MySQL, JupyterHub, and Cromwell. The installation order is: MySQL → JupyterHub → Cromwell → Bio-OS (this order must be followed).

2.1 Install the MySQL service

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install mysql bitnami/mysql \
        --create-namespace \
        --namespace bioos \
        --version 9.12.5 \
        --set auth.rootPassword=Bytedance2023 \
        --set primary.persistence.storageClass=nfs-csi

The rootPassword in the command can be changed to your desired password, but you must make the corresponding modifications in the subsequent steps.

2.2 Install the Jupyterhub service

Bio-OS is packaged with Helm and currently consists of four subcharts. Note: after deploying JupyterHub, you must obtain a new token in the browser before proceeding to deploy the bioos-server and web services. Install JupyterHub with Helm (make sure to choose images that match your environment’s architecture):

helm repo add bioos https://bio-os.github.io/helm-charts/charts

helm install jupyterhub bioos/jupyterhub \
        --namespace bioos \
        --create-namespace \
        --set hub.db.url=mysql+pymysql://root:[e-mail]:3306/bioos \
        --set hub.db.password=Bytedance2023

# Expose the JupyterHub service so that it can be accessed directly from the browser. Once you obtain the token, you may close the port forwarding.

kubectl -n bioos port-forward --address 0.0.0.0 service/hub 8081:8081

Open the browser and go to: http://<server-ip>/jupyterhub to request a new token.

2.3 Install the Cromwell service

There are two ways to install Cromwell: via Helm or via a JAR package.

• Helm installation: simpler to set up, but workflows cannot use Docker images during execution.

• JAR package deployment: allows workflows to use Docker images during execution.

1. Install Cromwell with Helm

helm repo add bioos https://bio-os.github.io/helm-charts/charts

helm install cromwell bioos/cromwell \
        --create-namespace \
        --namespace bioos \
        --set persistence.internal=true \
        --set db.mysql.host=mysql.bioos.svc.cluster.local \
        --set db.mysql.username=root \
        --set db.mysql.password=Bytedance2023

1. Install Cromwell with JAR package

Download cromwell-85.jar from the link ：https://github.com/broadinstitute/cromwell/releases/tag/85

To enable Cromwell to communicate with MySQL, you need to modify the MySQL connection configuration:

kubectl edit svc mysql
# Change type
type: NodePort

# Check the K8S internal communication port (the one after 3306).
kubectl get svc mysql
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mysql   NodePort   10.108.158.83   <none>        3306:30276/TCP   2d1h

Obtain the InternalIP using the following command:

kubectl get node -o wide

The obtained InternalIP is shown as follows:

NAME        STATUS   ROLES           AGE    VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
tf-server   Ready    control-plane   2d2h   v1.27.4   192.168.200.146   <none>        Ubuntu 22.04.2 LTS   5.15.0-83-generic   containerd://1.7.1

Edit application.conf and update the MySQL connection string with the obtained InternalIP and Port.

The application.conf is shown as follows:

include required(classpath("application"))
webservice {
  port = 8000
}
workflow-options {
  workflow-log-dir = /nfs/bioos-storage/cromwell-workflow-logs
  workflow-log-temporary = false
}
call-caching {
  enabled = true
  invalidate-bad-cache-results = true
}
database {
  profile = "slick.jdbc.MySQLProfile$"
  db {
    driver = "com.mysql.cj.jdbc.Driver"
    url = "jdbc:mysql://192.168.200.146:30276/cromwell?rewriteBatchedStatements=true&useSSL=false"
    port = 3306
    user = "root"
    password = "admin"
    connectionTimeout = 5000
  }
}
backend {
  default = "Local"
  providers {
    Local {
      config {
        root = "/nfs/bioos-storage/cromwell-executions"
        filesystem {
          local {
           localization: [
                  "hard-link", "soft-link", "copy"
           ]

            caching {
              duplication-strategy: [
                "hard-link", "soft-link", "copy"
              ]
              hashing-strategy: "md5"
              check-sibling-md5: false
            }
          }
        }
      }
    }
  }
}

Run Cromwell with Java (note: use nohup to ensure Cromwell runs in the background):

nohup java -jar -Dconfig.file=application.conf cromwell-85.jar server >log1 2>logerr &

2.4 Install the Bio-OS service

helm repo add bioos https://bio-os.github.io/helm-charts/chartss

helm install bioos bioos/bioos \
        --create-namespace \
        --namespace bioos \
        --set mysql.hostname=mysql.bioos.svc.cluster.local \
        --set mysql.database=bioos \
        --set mysql.username=root \
        --set mysql.password=Bytedance2023 \
        --set wes.endpoint=http://cromwell.bioos.svc.cluster.local:8000 \ # 之前cromwell如果是helm部署，此处可以使用svc地址，如果之前cromwell位jar包部署，此处需要修改为cromwell宿主机ip和port
        --set jupyterhub.endpoint=http://{{INGRESS-ADDRESS:PORT}}/jupyterhub/ \   # 需要手工替换成Ingress的地址
        --set jupyterhub.adminToken=${jupyterhub-token}  # 这里填写2.2步骤中获取的token

# Forward port ：
kubectl -n ingress-nginx port-forward --address 0.0.0.0 service/ingress-nginx-controller 8888:80

三. Deployment validation

Run kubectl get po,pv,pvc,ing to check the application deployment status. Bio-OS uses two storage volumes—one for MySQL and one for Bio-OS data—and defines several Ingress rules to expose services via subpaths.

kubectl get po,pv,pvc,ing

The command output should appear as follows:

Open the browser and visit: http://<server-ip>/workspace. This completes the installation of Bio-OS.