By default, workloads running on one Google Distributed Cloud connected rack node cannot access the local storage of another Distributed Cloud connected rack node. However, you can configure Distributed Cloud connected racks to use Rakuten Symcloud Storage, which is a third-party solution that acts as a local storage abstraction layer on each Distributed Cloud connected node and makes its local storage available to workloads running on other Distributed Cloud connected nodes. Symcloud Storage is the default and only storage option on Google Distributed Cloud connected servers.
Symcloud Storage is deployed from Google Cloud Marketplace and is subject to the terms stated therein. Google provides limited support for using Symcloud Storage with Distributed Cloud connected and might engage the third-party provider for assistance. Software updates for Symcloud Storage are included in the Distributed Cloud connected software updates.
Symcloud Storage classes
This section describe the storage classes that Symcloud Storage can enable on your
Distributed Cloud connected cluster. Symcloud Storage on
Distributed Cloud connected does not support the robin-rwx
storage class nor any custom-configured RWX file system mode volumes.
For more information on Symcloud Storage classes, see Using Robin CNS in Kubernetes.
robin
storage class
The robin
storage class is a basic Read Write-Once (RWO) storage class. The
following example illustrates the instantiation of the class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: robin
labels:
app.kubernetes.io/instance: robin
app.kubernetes.io/managed-by: robin.io
app.kubernetes.io/name: robin
provisioner: robin
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
robin-immediate
storage class
The robin-immediate
storage class is the same as robin
except that the
persistent volume is created immediately after creating the corresponding
persistent volume claim. The following example illustrates the instantiation
of the class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: robin-immediate
labels:
app.kubernetes.io/instance: robin
app.kubernetes.io/managed-by: robin.io
app.kubernetes.io/name: robin
provisioner: robin
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate
robin-repl-3
storage class
The robin-repl-3
is an RWO storage class with three replicas that span across
multiple Distributed Cloud nodes. The following example
illustrates the instantiation of the class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: robin-repl-3
labels:
app.kubernetes.io/instance: robin
app.kubernetes.io/managed-by: robin.io
app.kubernetes.io/name: robin
provisioner: robin
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
replication: "3"
faultdomain: host
Prerequisites
Before you begin, complete the following steps:
- Configure logging and monitoring for the target Distributed Cloud connected project.
- Create the target Distributed Cloud connected cluster.
- Configure your Distributed Cloud networking so that Pods in the target Distributed Cloud connected cluster can reach the Google Cloud data center.
- Bind each
local-block
persistent volume on each Distributed Cloud node that you don't want to be abstracted by Symcloud Storage. If you unbind a boundlocal-block
persistent volume, then installing Symcloud Storage wipes the contents of that persistent volume. For instructions, see Binding in the Kubernetes documentation.
Install Symcloud Storage on a Distributed Cloud connected node
To install Symcloud Storage on a Distributed Cloud connected node, complete the following steps:
Use the following command to apply the Symcloud Storage license to your cluster. Replace
LICENSE_FILE
with the full path and name of the Symcloud Storage license file.kubectl apply -f LICENSE_FILE -n robin-admin
Use the following command to verify the status of the
RobinCluster
service and all the Symcloud Storage nodes:kubectl describe robinclusters -n robinio
The command returns output similar to the following:
[...] Status: [...] Phase: Ready robin_node_status: [...] Status: Ready [...] Status: Ready [...] Status: Ready [...]
The expected status for the service and nodes is
Ready
.
Set Symcloud Storage as the default storage class
Use the following command to set Symcloud Storage as the default storage class
on your Distributed Cloud connected cluster. Replace
STORAGE_CLASS
with one of the
Symcloud Storage classes.
kubectl patch storageclass STORAGE_CLASS -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
For more information about setting the default storage class, see Change the default StorageClass in the Kubernetes documentation.
Configure abstracted Symcloud Storage volumes for workloads
This section provides examples for how to use Symcloud Storage classes to configure abstracted storage for your Distributed Cloud connected workloads. For more details about configuring Symcloud Storage volumes, see Using Robin CNS in Kubernetes.
Configure an ext4
RWO volume in file system mode
The following example illustrates how to configure a persistent volume claim for
an RWO volume in file system mode with the ext4
file system. Replace
STORAGE_CLASS
with one of the
Symcloud Storage classes.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: rwo-fs-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: STORAGE_CLASS
Configure an RWO volume in block mode
The following example illustrates how to configure a persistent volume claim for
an RWO volume in block mode. Replace STORAGE_CLASS
with one
of the Symcloud Storage classes.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: rwo-block-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: STORAGE_CLASS volumeMode: Block
Modify the configuration of an existing volume
The following example illustrates how to modify the configuration of an existing
Symcloud Storage LZ4 compressed RWO volume by using annotations.
ReplaceSTORAGE_CLASS
with one of the Symcloud Storage classes.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: compressed-rwo-fs-pvc annotations: robin.io/compression: LZ4 spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: STORAGE_CLASS
The following example illustrates how to modify the configuration of an existing
Symcloud Storage RWO volume with the xfs
file system by using annotations.
ReplaceSTORAGE_CLASS
with one of the Symcloud Storage classes.
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: rwo-xfs-pvc annotations: robin.io/fstype: xfs spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: STORAGE_CLASS
Configure the Symcloud Storage CLI client
Symcloud Storage provides a command-line interface (CLI) client that you can use to manage your Symcloud Storage configuration. To configure the client on your Distributed Cloud connected cluster, complete the following steps:
Get the Symcloud Storage image path used by the
RobinCluster
service instance deployed on your Distributed Cloud connected cluster and set your environment variables as follows:image_robin=$(kubectl get robincluster -o jsonpath='{.items[].spec.image_robin}') image_registry_path=$(kubectl get robincluster -o jsonpath='{.items[].spec.image_registry_path}') ROBIN_CNS_IMAGE="$image_registry_path/$image_robin"
Create a
robincli
resource with the following contents:kind: Deployment apiVersion: apps/v1 metadata: name: robincli namespace: default labels: name: robincli spec: replicas: 1 selector: matchLabels: name: robincli template: metadata: annotations: product: robin labels: name: robincli spec: containers: - name: robincli image: ROBIN_CNS_IMAGE workingDir: /root command: ["/bin/bash","-c","mkdir -p /root/.robin; ln -s -t /usr/lib/python3.7/site-packages/ /opt/robin/current/python3/site-packages/robincli /opt/robin/current/python3/site-packages/stormgr_def.py /opt/robin/current/python3/site-packages/stormgr_lib.py; /opt/robin/current/bin/robin client add-context robin-master.robinio --set-current; while true; do sleep 10000; done"] resources: requests: memory: "10Mi" cpu: "100m"
Replace
ROBIN_CNS_IMAGE
with the full repository path and name of the image that you obtained in step 1.Apply the
robincli
resource onto your Distributed Cloud connected cluster.Upon initial installation, Symcloud Storage generates a
default-admin-user
secret in therobinio
namespace with a randomized password. Use the following commands to obtain these logon credentials:Obtain the user name:
kubectl -n robinio get secret default-admin-user -o jsonpath='{.data.username}' | base64 -d
Obtain the password:
kubectl -n robinio get secret default-admin-user -o jsonpath='{.data.password}' | base64 -d
Log on to the newly created Pod and run the client:
kubectl exec -it robincli -- bash
Limitations of Symcloud Storage
When using Symcloud Storage with Distributed Cloud connected, you can only achieve high availability if your Distributed Cloud connected cluster consists of three or more Distributed Cloud connected nodes.
Removing nodes that use Symcloud Storage from a cluster
Symcloud Storage volume replicas are stored on worker nodes within your Distributed Cloud connected cluster. If you remove a node from the cluster, the Symcloud Storage volume data stored on that node becomes unavailable. To prevent this, you must do one of the following:
If you are tearing down the entire cluster, remove the workloads and their corresponding Symcloud Storage persistent volumes before you tear down the cluster itself.
If you are removing specific nodes from the cluster, you must migrate the workload data stored on those nodes before you remove those nodes from the cluster. For instructions, see Evacuating volumes from a disk.