Backup and recovery with custom storage

This section discusses how to configure backup and recovery of your Cassandra database using ssh and your file system instead of using Google Cloud. See also:

What is Cassandra backup and recovery with custom storage?

Backup with custom storage stores backups of your Cassandra database to compressed files in the file system of a server you specify. Backups occur on a schedule you specify in your overrides file. The connection to the server is by secure SSH.

Setting up backups without Cloud Services

The following steps include common examples for completing specific tasks, like creating an SSH key pair. Use the methods that are appropriate to your installation.

The procedure has the following parts:

Set up the server and SSH
Set the schedule and destination for backup
Set up for recovery

Set up the server and SSH

Designate a Linux or Unix server for your backups. This server must be reachable using SSH from your Apigee hybrid runtime plane. It must have enough storage for your backups.
Set up an SSH server on the server, or ensure that it has a secure SSH server configured.
Caution: For security purposes, make sure your SSH server is up to date.

Create an SSH key pair and store the private key file in a path that is accessible from your hybrid runtime plane. You must use a blank password for your key pair or the backup will fail. For example:

ssh-keygen -t rsa -b 4096 -C exampleuser@example.com
  Enter file in which to save the key (/Users/exampleuser/.ssh/id_rsa): $APIGEE_HOME/hybrid-files/certs/ssh_key
  Enter passphrase (empty for no passphrase):
  Enter same passphrase again:
  Your identification has been saved in ssh_key
  Your public key has been saved in ssh_key.pub
  The key fingerprint is:
  SHA256:DWKo334XMZcZYLOLrd/8HNpjTERPJJ0mc11UYmrPvSA exampleuser@example.com
  The key's randomart image is:
  +---[RSA 4096]----+
  |          +.  ++X|
  |     .   . o.=.*+|
  |    . o . . o==o |
  |   . . . =oo+o...|
  |  .     S +E oo .|
  |   . .   .. . o .|
  |    . . .  . o.. |
  |     .  ...o ++. |
  |      .. .. +o+. |
  +----[SHA256]-----+

Create a user account on the backup server with the name apigee. Make sure the new apigee user has a home directory under /home.
On the backup server, create an ssh directory in the new /home/apigee directory.
Copy the public key (ssh_key.pub in the previous example) into a file named authorized_keys in the new /home/apigee/ssh directory. For example:
```
cd /home/apigee
mkdir .ssh
cd .ssh
vi authorized_keys
```
On your backup server, create a backup directory within the /home/apigee/ directory. The backup directory can be any directory as long as the apigee user has access to it. For example:
```
cd /home/apigee
mkdir cassandra-backup
```
Test the connection. You need to make sure that your Cassandra pods can connect to your backup server using SSH:
1. Log into the shell of your Cassandra pod. For example:
```
kubectl exec -it -n apigee APIGEE_CASSANDRA_DEFAULT_0 -- /bin/bash
```
  Where APIGEE_CASSANDRA_DEFAULT_0 is the name of a Cassandra pod. Change this to the name of the pod you want to connect from.
2. Connect by SSH to your backup server, using the server IP address:
```
ssh apigee@BACKUP_SERVER_IP
```
  Note: You may see a warning at this point saying your server's fingerprint is unrecognized and asks if you would like to continue. For purposes of this test, this means you can successfully reach your backup server from your Cassandra pod. You do not need to continue.

Set the schedule and destination for backup

You set the schedule and destination for backups in your overrides.yaml file.

Add the following parameters to your overrides.yaml file:

Parameters

cassandra:
  backup:
    enabled: true
    keyFile: "PATH_TO_PRIVATE_KEY_FILE"
    server: "BACKUP_SERVER_IP"
    storageDirectory: "/home/apigee/BACKUP_DIRECTORY"
    cloudProvider: "HYBRID" # required verbatim "HYBRID" (all caps)
    schedule: "SCHEDULE"

Example

cassandra:
  backup:
    enabled: true
    keyFile: "/Users/exampleuser/apigee-hybrid/hybrid-files/service-accounts/private.key"
    server: "34.56.78.90"
    storageDirectory: "/home/apigee/cassbackup"
    cloudProvider: "HYBRID"
    schedule: "0 2 * * *"

Where:

Property	Description
`backup:enabled`	Backup is disabled by default. You must set this property to `true`.
`backup:keyFile`	`PATH_TO_PRIVATE_KEY_FILE` The path on your local file system to the SSH private key file (named `ssh_key` in the step where you created the SSH key pair).
`backup:server`	`BACKUP_SERVER_IP` The IP address of your backup server.
`backup:storageDirectory`	`BACKUP_DIRECTORY` The name of the backup directory on your backup server. This must be a directory within `home/apigee` (the backup directory is named `cassandra_backup` in the step where you created the backup directory).
`backup:cloudProvider`	`HYBRID` The `cloudProvider: "HYBRID"` property is required.
`backup:schedule`	`SCHEDULE` The time when the backup starts, specified in standard crontab syntax. Default: `0 2 * * ` Note:* Avoid scheduling a backup that starts a short time after you apply the backup configuration to your cluster. When you apply the backup configuration, Kubernetes recreates the Cassandra nodes. If the backup starts before the nodes restart (possibly several minutes) the backup will fail.

Use apigeectl to apply the backup configuration to the storage scope of your cluster:
```
$APIGEECTL_HOME/apigeectl --datastore -f YOUR_OVERRIDES_FILE
```
Where YOUR_OVERRIDES_FILE is the path to the overrides file you just edited.

Configure restore

Restoration takes your data from the backup location and restores the data into a new Cassandra cluster with the same number of nodes. No data is taken from the old Cassandra cluster.

The restoration instructions below are for single region deployments that do not use Google Cloud Storage for backups. For other deployments, see the following:

For single region deployments that do use Google Cloud Storage for backups, see Cassandra backup and recovery.
For multi-region deployments, see Multi-region deployment on GKE and GKE on-prem.

To restore Cassandra backups:

Create a new namespace within the existing Kubernetes cluster that will be used to restore the hybrid huntime deployment. Do not use the original namespace name for the new namespace. Do not use the old namespace for restoration.
In the root hybrid installation directory, create a new overrides-restore.yaml file.
Copy the complete Cassandra configuration from your original overrides.yaml file into the new overrides-restore.yaml file. For example:
```
cp ./overrides.yaml ./overrides-restore.yaml
```

Add a namespace element to the new overrides-restore.yaml file.

Parameters

namespace: YOUR_RESTORE_NAMESPACE
cassandra:
  ...
  restore:
    enabled: true
    keyFile: "PATH_TO_PRIVATE_KEY_FILE"
    server: "BACKUP_SERVER_IP"
    storageDirectory: "/home/apigee/BACKUP_DIRECTORY"
    cloudProvider: "HYBRID"  # required verbatim "HYBRID" (all caps)
    snapshotTimestamp: "BACKUP_TO_RESTORE"
  ...

Example

namespace: cassandra-restore
cassandra:
  restore:
    enabled: true
    keyFile: "/Users/exampleuser/apigee-hybrid/hybrid-files/service-accounts/private.key"
    server: "34.56.78.90"
    storageDirectory: "/home/apigee/cassbackup"
    cloudProvider: "HYBRID"
    snapshotTimestamp: "20201001183903"

Where:

Property	Description
`namespace`	`YOUR_RESTORE_NAMESPACE` The name of the new namespace you created in step 1 for the new Cassandra cluster. Do not use the same namespace you used for your original cluster.
`restore:enabled`	Restore is disabled by default. You must set this property to `true`.
`restore:keyFile`	`PATH_TO_PRIVATE_KEY_FILE` The path on your local file system to the SSH private key file (named `ssh_key` in the step where you created the SSH key pair).
`restore:server`	`BACKUP_SERVER_IP` The IP address of your backup server.
`restore:storageDirectory`	`BACKUP_DIRECTORY` The name of the backup directory on your backup server. This must be a directory within `home/apigee` (the backup directory is named `cassandra_backup` in the step where you created the backup directory).
`restore:cloudProvider`	`HYBRID` The `cloudProvider: "HYBRID"` property is required.
`restore:snapshotTimestamp`	`BACKUP_TO_RESTORE` The specific backup you want to restore, specified in standard crontab syntax (no wildcards allowed).

Change the app label on any Cassandra nodes in the old namespace by executing the following command:

kubectl label pods --overwrite --namespace=OLD_NAMESPACE -l app=apigee-cassandra app=apigee-cassandra-old

Create a new hybrid runtime deployment. This will create a new Cassandra cluster and begin restoring the backup data into the cluster:
```
./apigeectl init  -f ../overrides-restore.yaml
```
```
./apigeectl apply  -f ../overrides-restore.yaml
```

Once the restoration is complete, the traffic must be switched to use the Cassandra cluster in the new namespace. Run the following commands to switch the traffic:

kubectl get rs -n OLD_NAMESPACE # look for the 'apigee-connect' replicaset

kubectl patch rs -n OLD_NAMESPACE APIGEE_CONNECT_RS_NAME -p '{"spec":{"replicas" : 0}}'

Once the traffic switch is complete, you can reconfigure backups on the restored cluster by removing the restore configuration and adding the backup configuration to the overrides-restore.yaml file. Replace YOUR_RESTORE_NAMESPACE with the new namespace name created in step 1.
```
namespace: YOUR_RESTORE_NAMESPACE
cassandra:
  ...
  backup:
    enabled: true
    serviceAccountPath: SA_JSON_FILE_PATH
    dbStorageBucket: CLOUD_STORAGE_BUCKET_PATH
    schedule: BACKUP_SCHEDULE_CODE
  ...
```
Then apply the backup configuration with the following command:
```
./apigeectl apply  -f ../overrides-restore.yaml
```