Use Filestore Multishares for Google Kubernetes Engine

Stay organized with collections Save and categorize content based on your preferences.

This guide shows you how to use Filestore Multishares for Google Kubernetes Engine with the GKE Filestore CSI driver.

Before you begin

  1. Before you begin, complete the setup steps needed to use Filestore.

  2. Enable the GKE Filestore CSI driver (GKE version 1.23 or later).

Use Filestore Multishares with multiple applications

This section shows you how to deploy two applications, one Deployment and one Statefulset, each using a Filestore Multishares StorageClass. You'll also see how GKE bin packs (a process to efficiently pack applications into your GKE nodes) all volumes in the same underlying Filestore Enterprise instance.

  1. Use the GKE-provided StorageClass or create a custom StorageClass.

    After the GKE Filestore CSI driver is enabled, users can access the GKE-provided multishare StorageClass enterprise-multishare-rwx with the following configuration. Referencing this StorageClass, the GKE Filestore CSI driver uses dynamic volume provisioning to automatically create Persistent Volumes (PVs) for new Persistent Volume Claims (PVCs) as GKE workload demand requires:

    $ kubectl describe sc enterprise-multishare-rwx
    Name:                  enterprise-multishare-rwx
    IsDefaultClass:        No
    Annotations:           components.gke.io/component-name=filestorecsi,components.gke.io/component-version=0.7.2,components.gke.io/layer=addon
    Provisioner:           filestore.csi.storage.gke.io
    Parameters:            instance-storageclass-label=enterprise-multishare-rwx,multishare=true,tier=enterprise
    AllowVolumeExpansion:  True
    MountOptions:          <none>
    ReclaimPolicy:         Delete
    VolumeBindingMode:     WaitForFirstConsumer
    Events:                <none>
    

    If needed, users can create a custom StorageClass based on this template, being mindful of the following naming requirements:

  2. Create a Deployment with multiple pod replicas using a single PVC.

    Create a YAML configuration file similar to the following:

    cat <<EOF | kubectl apply -f -
    
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web-server-multishare
      labels:
        app: nginx
    spec:
      replicas: 5
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            volumeMounts:
            - mountPath: /usr/share/nginx/html
              name: mypvc
          volumes:
          - name: mypvc
            persistentVolumeClaim:
              claimName: test-pvc-fs
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: test-pvc-fs
    spec:
      accessModes:
        - ReadWriteMany
      storageClassName: enterprise-multishare-rwx
      resources:
        requests:
          storage: 100Gi
    
    EOF
    
  3. Check pod replicas.

    a. From the command line, run the following command to check the PVC status:

      $ kubectl get pvc
    

    You should see something similar to the following response:

      NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
      test-pvc-fs   Bound    pvc-056d769d-a709-4bb2-b6d3-0361871b27a2   100Gi      RWX            enterprise-multishare-rwx  35m
    

    b. From the command line, run the following command to check the pod status:

      $ kubectl get pod
    

    You should see something similar to the following response:

      NAME                                     READY   STATUS    RESTARTS   AGE
      web-server-multishare-76c9ffb4b5-2dhml   1/1     Running   0          35m
      web-server-multishare-76c9ffb4b5-7mtcb   1/1     Running   0          35m
      web-server-multishare-76c9ffb4b5-csdbd   1/1     Running   0          35m
      web-server-multishare-76c9ffb4b5-rgx82   1/1     Running   0          35m
      web-server-multishare-76c9ffb4b5-zjl27   1/1     Running   0          35m
    
  4. Scale replicas.

    a. From the command line, run the following command to edit the Deployment:

      $ kubectl edit deployment web-server-multishare
    

    b. The file will open in the command line. Find the spec.replicas field and update the value to 10.

    c. From the command line, run the following command to see the applied change:

      $ kubectl get pod
    

    You should see something similar to the following response:

      NAME                                     READY   STATUS    RESTARTS   AGE
      web-server-multishare-76c9ffb4b5-2dhml   1/1     Running   0          36m
      web-server-multishare-76c9ffb4b5-5ctkf   1/1     Running   0          3s
      web-server-multishare-76c9ffb4b5-7mtcb   1/1     Running   0          36m
      web-server-multishare-76c9ffb4b5-8dwmw   1/1     Running   0          2s
      web-server-multishare-76c9ffb4b5-csdbd   1/1     Running   0          36m
      web-server-multishare-76c9ffb4b5-lndcq   1/1     Running   0          2s
      web-server-multishare-76c9ffb4b5-rgx82   1/1     Running   0          36m
      web-server-multishare-76c9ffb4b5-vtd6p   1/1     Running   0          3s
      web-server-multishare-76c9ffb4b5-xm49s   1/1     Running   0          3s
      web-server-multishare-76c9ffb4b5-zjl27   1/1     Running   0          36m
    

    Notice ten pods are running.

    d. From the command line, run the following command:

      $ kubectl get deployment
    

    You should see something similar to the following response:

      NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
      web-server-multishare   10/10   10           10          36m
    

    e. From the command line, run the following command to check the PVC bound status:

      $ kubectl get pvc
    

    You should see something similar to the following response:

      NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
      test-pvc-fs   Bound    pvc-056d769d-a709-4bb2-b6d3-0361871b27a2   100Gi      RWX            enterprise-multishare-rwx  37m
    

    f. From the command line, run the following command to edit the Deployment:

      $ kubectl edit deployment web-server-multishare
    

    g. The file will open in the command line. Find the spec.replicas field and update the value to 2.

    h. From the command line, run the following command to see the applied change:

      $ kubectl get pod
    

    You should see something similar to the following response:

      NAME                                     READY   STATUS    RESTARTS   AGE
      web-server-multishare-76c9ffb4b5-2dhml   1/1     Running   0          38m
      web-server-multishare-76c9ffb4b5-7mtcb   1/1     Running   0          38m
    
  5. Deploy a Statefulset.

    Deploy a second application that shares the underlying Filestore instance.

    To do so, provision 200 GiB of space and validate that it uses the same underlying Filestore instance as the first application.

    You'll then scale the application to nine replicas using 900 GiB in total (9 replicas using 100 GiB each) and verify that GKE uses the same Filestore instance by sharing the instance.

    Create a YAML configuration file similar to the following:

    cat <<EOF | kubectl apply -f -
    
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: web
    spec:
      serviceName: "nginx"
      replicas: 2
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: k8s.gcr.io/nginx-slim:0.8
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: test-pvc-multishare
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: test-pvc-multishare
        spec:
          accessModes: [ "ReadWriteMany" ]
          storageClassName: enterprise-multishare-rwx
          resources:
            requests:
              storage: 100Gi
    
    EOF
    
    
  6. Check Statefulset replicas and volumes.

    From the command line, run the following command:

    $ kubectl get pod
    

    You should see something similar to the following response:

    NAME                                     READY   STATUS    RESTARTS   AGE
    web-0                                    1/1     Running   0          4m48s
    web-1                                    1/1     Running   0          3m32s
    web-server-multishare-76c9ffb4b5-2dhml   1/1     Running   0          57m
    web-server-multishare-76c9ffb4b5-7mtcb   1/1     Running   0          57m
    

    Notice that the first two pods are associated with the Statefulset. The last two pods are associated with the Deployment.

    From the command line, run the following command:

    $ kubectl get statefulset
    

    You should see something similar to the following response:

    NAME   READY   AGE
    web    2/2     2m8s
    

    From the command line, run the following command:

    $ kubectl get pvc
    

    You should see something similar to the following response:

    NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
    test-pvc-fs                 Bound    pvc-056d769d-a709-4bb2-b6d3-0361871b27a2   100Gi      RWX            enterprise-multishare-rwx   54m
    test-pvc-multishare-web-0   Bound    pvc-7aa21b5a-5343-4547-b7d7-414c16af15a7   100Gi      RWX            enterprise-multishare-rwx   114s
    test-pvc-multishare-web-1   Bound    pvc-8b37cd6e-d764-4d38-80d7-d74228536cfe   100Gi      RWX            enterprise-multishare-rwx   38s
    

    The PVC test-pvc-fs is associated with the Deployment web-server-multishare.

    The PVCs test-pvc-multishare-web-0 and test-pvc-multishare-web-1 are associated with the Statefulset.

  7. Scale the Statefulset replicas.

    Increase the replica count to nine. As the count increases, the corresponding PVCs are created.

    a. From the command line, run the following command:

    $ kubectl  edit statefulset web
    

    b. The file will open in the command line. Find the spec.replicas field and update the value to 9.

    c. From the command line, run the following command to see the applied change:

    $ kubectl get statefulset
    

    You should see something similar to the following response:

    NAME   READY   AGE
    web    9/9     13m
    

    d. From the command line, run the following command:

    $ kubectl get deployment
    

    You should see something similar to the following response:

    NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
    web-server-multishare   2/2     2            2           65m
    

    e. From the command line, run the following command:

    $ kubectl get pvc
    

    You should see something similar to the following response:

    NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
    test-pvc-fs                 Bound    pvc-056d769d-a709-4bb2-b6d3-0361871b27a2   100Gi      RWX            enterprise-multishare-rwx   65m
    test-pvc-multishare-web-0   Bound    pvc-7aa21b5a-5343-4547-b7d7-414c16af15a7   100Gi      RWX            enterprise-multishare-rwx   13m
    test-pvc-multishare-web-1   Bound    pvc-8b37cd6e-d764-4d38-80d7-d74228536cfe   100Gi      RWX            enterprise-multishare-rwx   12m
    test-pvc-multishare-web-2   Bound    pvc-3fcbd132-939f-4364-807a-7c8ac6a3e64e   100Gi      RWX            enterprise-multishare-rwx   5m12s
    test-pvc-multishare-web-3   Bound    pvc-5894afa5-2502-4ee7-9d5c-b7378cb85479   100Gi      RWX            enterprise-multishare-rwx   4m57s
    test-pvc-multishare-web-4   Bound    pvc-ebbe452b-bc8f-4624-a830-a2094cce0d67   100Gi      RWX            enterprise-multishare-rwx   4m36s
    test-pvc-multishare-web-5   Bound    pvc-5a73a698-d174-44cb-a3a1-e767966c3417   100Gi      RWX            enterprise-multishare-rwx   4m20s
    test-pvc-multishare-web-6   Bound    pvc-102da6a9-2ca6-4f9e-9896-8fe14709db7a   100Gi      RWX            enterprise-multishare-rwx   3m55s
    test-pvc-multishare-web-7   Bound    pvc-160e81cd-c5bf-4ae6-966e-518e8249e02d   100Gi      RWX            enterprise-multishare-rwx   3m38s
    test-pvc-multishare-web-8   Bound    pvc-9b52d773-2e9a-40de-881c-dc06945ba3d7   100Gi      RWX            enterprise-multishare-rwx   118s
    
  8. Verify the Filestore instance state.

    You now have a Deployment with two replica pods, and a Statefulset with nine replica pods, and a total of ten PVCs, each 100 GiB in size. All of the volumes are packed onto a single Filestore Multishare instance.

    a. From the command line, run the following instances list command:

    $  gcloud beta filestore instances list --project=YOUR_PROJECT_ID --region=REGION
    

    where:

    • YOUR_PROJECT_ID is the name of the project being used. For example, my-project.

    • REGION is the name of the region being used. For example, us-central1.

    You should see something similar to the following response:

    INSTANCE_NAME                            LOCATION     TIER        CAPACITY_GB  FILE_SHARE_NAME  IP_ADDRESS   STATE  CREATE_TIME
    fs-a767cef8-738e-4c8e-b70b-09cbb872d016  us-central1  ENTERPRISE  1024         N/A              10.192.53.2  READY  2022-06-21T21:15:30
    

    b. From the command line, run the following instances describe command:

    $ gcloud filestore instances describe fs-a767cef8-738e-4c8e-b70b-09cbb872d016 --project=YOUR_PROJECT_ID --region=REGION
    capacityGb: '1024'
    capacityStepSizeGb: '256'
    createTime: '2022-06-21T21:15:30.464237089Z'
    labels:
      storage_gke_io_created-by: filestore_csi_storage_gke_io
      storage_gke_io_storage-class-id: enterprise-multishare-rwx
    maxCapacityGb: '10240'
    maxShareCount: '10'
    multiShareEnabled: true
    name: projects/YOUR_PROJECT_ID/locations/REGION/instances/fs-a767cef8-738e-4c8e-b70b-09cbb872d016
    networks:
    - connectMode: DIRECT_PEERING
      ipAddresses:
      - 10.192.53.2
      modes:
      - MODE_IPV4
      network: csi-filestore-test-network
      reservedIpRange: 10.192.53.0/26
    state: READY
    tier: ENTERPRISE
    
    

    where:

    • YOUR_PROJECT_ID is the name of the project being used. For example, my-project.

    • REGION is the name of the region being used. For example, us-central1.

Expand a PVC and verify the Filestore instance

This section shows you how to expand an existing PVC and verify the Filestore instance size.

  1. Expand a PVC.

    PVCs—backed by shares in a Filestore Multishare instance— can grow to a maximum size of 1 TiB. To verify this, expand one of the volumes associated with the Statefulset while the pod is using it.

    From the command line, run the following command to check the current PVC size of replica 0:

    $ kubectl get pvc test-pvc-multishare-web-0 -o json
    {
        "apiVersion": "v1",
        "kind": "PersistentVolumeClaim",
        "metadata": {
            "annotations": {
                "pv.kubernetes.io/bind-completed": "yes",
                "pv.kubernetes.io/bound-by-controller": "yes",
                "volume.beta.kubernetes.io/storage-provisioner": "filestore.csi.storage.gke.io",
                "volume.kubernetes.io/storage-provisioner": "filestore.csi.storage.gke.io"
            },
            "creationTimestamp": "2022-06-21T22:07:42Z",
            "finalizers": [
                "kubernetes.io/pvc-protection"
            ],
            "labels": {
                "app": "nginx"
            },
            "name": "test-pvc-multishare-web-0",
            "namespace": "default",
            "resourceVersion": "48395",
            "uid": "7aa21b5a-5343-4547-b7d7-414c16af15a7"
        },
        "spec": {
            "accessModes": [
                "ReadWriteMany"
            ],
            "resources": {
                "requests": {
                    "storage": "100Gi"
                }
            },
            "storageClassName": "enterprise-multishare-rwx",
            "volumeMode": "Filesystem",
            "volumeName": "pvc-7aa21b5a-5343-4547-b7d7-414c16af15a7"
        },
        "status": {
            "accessModes": [
                "ReadWriteMany"
            ],
            "capacity": {
                "storage": "100Gi"
            },
            "phase": "Bound"
        }
    }
    
    
  2. From the command line, run the following command to increase the size to 500 GiB:

    $ kubectl edit pvc test-pvc-multishare-web-0
    
    
  3. The file will open in the command line. Find the spec.resources.requests.storage field and update the value to 500Gi.

  4. From the command line, run the following command to see the applied change:

    $ kubectl get pvc test-pvc-multishare-web-0
    

    You should see something similar to the following response:

    NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
    test-pvc-multishare-web-0   Bound    pvc-7aa21b5a-5343-4547-b7d7-414c16af15a7   500Gi      RWX            enterprise-multishare-rwx   28m
    

    The Filestore CSI driver accepted the request, first expanding the underlying Filestore instance, and then expanding the share backing the PVC.

    Specifically, the Filestore CSI driver automatically expanded the instance to 1536 Gi to accommodate the new share size of 500Gi.

  5. From the command line, run the following instances describe command to verify the capacity of the Filestore instance:

    $ gcloud filestore instances describe fs-a767cef8-738e-4c8e-b70b-09cbb872d016 --project=YOUR_PROJECT_ID --region=REGION
    capacityGb: '1536'
    capacityStepSizeGb: '256'
    createTime: '2022-06-21T21:15:30.464237089Z'
    labels:
      storage_gke_io_created-by: filestore_csi_storage_gke_io
      storage_gke_io_storage-class-id: enterprise-multishare-rwx
    maxCapacityGb: '10240'
    maxShareCount: '10'
    multiShareEnabled: true
    name: projects/YOUR_PROJECT_ID/locations/us-central1/instances/fs-a767cef8-738e-4c8e-b70b-09cbb872d016
    networks:
    - connectMode: DIRECT_PEERING
      ipAddresses:
      - 10.192.53.2
      modes:
      - MODE_IPV4
      network: csi-filestore-test-network
      reservedIpRange: 10.192.53.0/26
    state: READY
    tier: ENTERPRISE
    

    where:

    • YOUR_PROJECT_ID is the name of the project being used. For example, my-project.

    • REGION is the name of the region being used. For example, us-central1.

Dynamic provisioning on a Shared VPC

Filestore CSI driver for GKE supports dynamic provisioning of volumes in a service project under a shared VPC. The following section shows how to use the Filestore CSI driver to dynamically provision volumes on Filestore Multishare instances in a service project under a Shared VPC network.

  1. Complete the setup steps for a Shared VPC network and private service access.

  2. Create a StorageClass to dynamically provision volumes backed by a Filestore Multishares instance on a shared VPC.

    Run the following command to deploy a StorageClass resource:

    cat <<EOF | kubectl apply -f -
    
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: csi-filestore-multishare-sharedvpc
    provisioner: filestore.csi.storage.gke.io
    parameters:
      network: "projects/HOST_PROJECT_ID/global/networks/SHARED_VPC_NAME"
      connect-mode: PRIVATE_SERVICE_ACCESS
      tier: enterprise
      multishare: "true"
    allowVolumeExpansion: true
    
    EOF
    

    where:

    • HOST_PROJECT_ID is the ID or name of the host project of the Shared VPC network. For example, my-host-project.

    • SHARED_VPC_NAME the name of the Shared VPC network. For example, my-shared-vpc.

    If you want to deploy your resource within a reserved IP address range, add the following line to the parameters used in the command:

    reserved-ip-range: RESERVED_NAME
    

    Where RESERVED_NAME is the name of the reserved IP address range within which a Filestore instance can be provisioned. For example, filestore-reserved-ip-range. If a reserved IP range is specified, it must be a named address range instead of a direct CIDR value.

    For more information, see Allocate IP address ranges or Configuring a reserved IP address range. To see an example of how to create a reserved name using the Google Cloud console, see Create an IP allocation.

  3. Create a Deployment.

    Run the following command to create a Deployment resource:

    cat <<EOF | kubectl apply -f -
    
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web-server-multishare
      labels:
        app: nginx
    spec:
      replicas: 5
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            volumeMounts:
            - mountPath: /usr/share/nginx/html
              name: mypvc
          volumes:
          - name: mypvc
            persistentVolumeClaim:
              claimName: test-pvc-fs-sharedvpc
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: test-pvc-fs-sharedvpc
    spec:
      accessModes:
        - ReadWriteMany
      storageClassName: csi-filestore-multishare-sharedvpc
      resources:
        requests:
          storage: 100Gi
    
    EOF
    
    

CMEK-enabled Filestore instances

You can create GKE volumes hosted on CMEK-enabled Filestore Multishare instances. In this section, you'll see how to setup a customer-managed encryption key (CMEK) for your Filestore instance.

Customer-managed key details can be provided in the StorageClass. Any instance dynamically created by the Filestore CSI driver, which references this StorageClass, will have CMEK enabled.

  1. Create a CMEK-enabled StorageClass.

    a. Run the following command:

    cat <<EOF | kubectl apply -f -
    
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: csi-filestore-multishare-cmek
    provisioner: filestore.csi.storage.gke.io
    parameters:
      tier: enterprise
      multishare: "true"
      instance-encryption-kms-key: projects/KEY_PROJECT_ID/locations/REGION/keyRings/RING_NAME/cryptoKeys/KEY_NAME
    
    allowVolumeExpansion: true
    
    EOF
    

    where:

    • KEY_PROJECT_ID is the name of the project where the key is located. For example, my-key-project.

    • REGION is the name of the region being used. For example, us-central1.

    • RING_NAME the key ring name. For example, my-key-ring-name.

    • KEY_NAME the key name. For example, my-key-name.

  2. Create a Deployment.

    b. Run the following command to create a Deployment resource:

    cat <<EOF | kubectl apply -f -
    
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web-server-multishare
      labels:
        app: nginx
    spec:
      replicas: 5
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            volumeMounts:
            - mountPath: /usr/share/nginx/html
              name: mypvc
          volumes:
          - name: mypvc
            persistentVolumeClaim:
              claimName: test-pvc-fs-cmek
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: test-pvc-fs-cmek
    spec:
      accessModes:
        - ReadWriteMany
      storageClassName: csi-filestore-multishare-cmek
      resources:
        requests:
          storage: 100Gi
    
    EOF
    
    

Map PVCs to Filestore instances

In this section, you'll see how to map your PVCs to your Filestore instances.

With Filestore Multishare instances, each PVC is hosted on a Filestore instance by the Filestore CSI driver. The details of the underlying Filestore instance hosting the volume and the share representing the Kubernetes volume, is captured in the volumeHandle field of the Persistent Volumes specification. The volume handle format is as follows:

   modeMultishare/<storageclass-prefix>/<project>/<region>/<filestore-instance-name>/<filestore-share-name>

The following kubectl command can be used to quickly determine the mappings between a PVC, PV, Filestore Instance, and a Filestore share.

From the command line, run the following command:

   $ kubectl get pv -o jsonpath='{range .items[*]}{"pv="}{.metadata.name}{",pvc="}{.spec.claimRef.name}{",volumeHandle="}{.spec.csi.volumeHandle}{"\n"}{end}'

You should see something similar to the following response:

   pv=pvc-67ad9abd-f25e-4130-b7ca-64d28bd29525,pvc=test-pvc-multishare,volumeHandle=modeMultishare/csi-filestore-multishare-sharedvpc/YOUR_PROJECT_ID/us-central1/fs-2109f680-3f04-4ada-b4bc-2a1c7fc47b88/pvc_67ad9abd_f25e_4130_b7ca_64d28bd29525

   pv=pvc-c80f4de0-9916-4957-b8ae-b21206650ac0,pvc=test-pvc-fs-sharedvpc,volumeHandle=modeMultishare/csi-filestore-multishare-sharedvpc/YOUR_PROJECT_ID/us-central1/fs-2109f680-3f04-4ada-b4bc-2a1c7fc47b88/pvc_c80f4de0_9916_4957_b8ae_b21206650ac0

where:

  • YOUR_PROJECT_ID is the name of the project being used. For example, my-project.

Notice that two persistent volumes in the cluster are hosted on a single Filestore instance.

What's next