Google Cloud Storage Fuse CSI Driver

gcsFuse 實驗

GCP

本文介紹 GcsFuseCsiDriver 的使用方法和實驗。

環境描述

arch

客戶在 GKE 上安裝壓測工具 K6 Operator，並希望將壓測報告上傳到 Google Cloud Storage，方便隨時取閱。本篇文章將會紀錄 gcs-fuse-csi-driver 的安裝及使用，並測試看看是否正常運作。

Kubernetes version: 1.27.8-gke.1067004

前置作業

需要先安裝 K6 Operator 在 Kubernetes 內:

https://github.com/grafana/k6-operator

測試步驟

  1. 啟動 GcsFuseCsiDriver

此步驟會等非常久，超過 20 分鐘

請替換 <cluster-name>

gcloud beta container clusters update <cluster-name> \
    --update-addons GcsFuseCsiDriver=ENABLED \
    --region=<cluster-name>
  1. 建立 bucket

locations: https://cloud.google.com/storage/docs/locations#location-r

請替換 <bucket-name> 及 <region>

gcloud storage buckets create gs://<bucket-name> --location=<region>
  1. 確認是否開啟 gke-metadata-server 功能

meta

  1. 建立 IAM & Kubernetes 的 service account，並且 assign role，可以使用以下腳本:

請替換 <bucket-name> 及 <project-id>

account

#!/bin/bash
# export GCS_BUCKET="<bucket-name>" \
# export PROJECT_ID="<project-id>" \
# export GCP_SA_NAME="gcs-sa" \
# export K8S_SA_NAME="k8s-gcs-sa" \
# export K8S_NAMESPACE="k6"

service_account=${GCP_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
# roles=("roles/storage.objectAdmin" "roles/storage.admin")
roles=("roles/storage.objectAdmin")

echo "service account: ${service_account}"

# Create a GCP service account in the Cloud Storage bucket project.
gcloud iam service-accounts create ${GCP_SA_NAME} --project=${PROJECT_ID}

for role in ${roles[@]}; do
    echo "assign permission role: ${role}"
    gcloud storage buckets add-iam-policy-binding gs://${GCS_BUCKET} \
        --member "serviceAccount:${service_account}" \
        --role "${role}"
done
#kubectl create ns ${K8S_NAMESPACE}
kubectl create sa ${K8S_SA_NAME} -n ${K8S_NAMESPACE}
gcloud iam service-accounts add-iam-policy-binding ${service_account} \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${K8S_SA_NAME}]"

kubectl annotate serviceaccount ${K8S_SA_NAME} \
    --namespace ${K8S_NAMESPACE} \
    iam.gke.io/gcp-service-account=${service_account}
  1. (Option) 按造 Configure access to Cloud Storage buckets using GKE Workload Identity Federation 步驟進行權限測試:

https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#authentication

  1. 參考以下範例，建立測試用 k6 程式 k6.js:

以下範例程式將在 60s 內使用 100 個 virtual users 對 http://${__ENV.SERVICE_NAME}/heathly 進行壓力測試

import http from "k6/http";
import { check } from "k6";

// Test configuration
export const options = {
  vus: 100,
  duration: '60s'
};

// Simulated user behavior
export default function () {
  let res = http.get(`http://${__ENV.SERVICE_NAME}/heathly`, {});
  // Validate response status
  check(res, { "status was 200": (r) => r.status == 200 });
}
  1. 建立 configmap:
kubectl create configmap k6-stress --from-file k6.js -n k6
  1. 參考以下範例，建立K6 TestRun:

以下設定將建立 4 個 Testing Job 進行壓力測試，產生報告時，以 name 欄位命名

撰寫本教學文章時，K6 Operator 在 TestRun 預設的 Image 尚未支援 Dashboard Report 的功能，因此需要指定 image: 0.49.0

請替換 <bucket-name>, <service-account> 及 <end-point>

12345 是 k6 Dockerfile 使用的 UID: https://github.com/grafana/k6/blob/master/Dockerfile#L11

k6

cat <<EOF | kubectl create -n k6 -f -
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: stress
  namespace: k6
spec:
  parallelism: 4
  script:
    configMap:
      name: k6-stress
      file: k6.js
  runner:
    image: grafana/k6:0.49.0
    securityContext:
      fsGroup: 12345
    metadata:
      annotations:
        gke-gcsfuse/volumes: "true"
    nodeSelector:
      role: sre
    env:
      - name: SERVICE_NAME
        value: <end-point>
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      - name: K6_WEB_DASHBOARD_EXPORT
        value: "/k6-report/report-\$(POD_NAME).html"
      - name: K6_WEB_DASHBOARD
        value: "true"
    serviceAccountName: <service-account>
    resources:
      limits:
        cpu: 2000m
        memory: 1000Mi
      requests:
        cpu: 500m
        memory: 500Mi
    volumeMounts:
    - mountPath: /k6-report
      name: gcs-fuse-csi-ephemeral
    volumes:
    - name: gcs-fuse-csi-ephemeral
      csi:
        driver: gcsfuse.csi.storage.gke.io
        volumeAttributes:
          bucketName: <bucket-name>
          mountOptions: "implicit-dirs"

運行結果

  1. 查看 Google Cloud Storage

運行成功後，將在 Google Cloud Storage 看到如以下的 report files:

report

  1. 查看 Report 內容

content

Reference