Restoring in a single region

This page describes how to restore Cassandra in a single region.

In a single region deployment, Apigee hybrid is deployed in a single data center or a region. If you have multiple Apigee organizations in your deployment, the restore process restores data for all the organizations. In a multi-organization setup, you cannot restore a specific organization.

Note: Before you begin restoring a single region, consider whether the following prerequisite steps are applicable:

If you want to preserve an existing setup for troubleshooting and root cause analysis (RCA), you should delete all the org and env components from the Kubernetes cluster except the Apigee controller, and then retain the cluster. The cluster will contain the existing Apigee datastore (Cassandra) which you can use for troubleshooting. Create a new Kubernetes cluster and then restore Cassandra in the new cluster.
If your hybrid installation was set up with multiple organizations, get the overrides files for each organization before proceeding with restore in a single region. You can add the restore configuration as described in Step 3 to any one of the overrides files. Do not add the restore configuration to any other overrides file.

Restoring a region from a backup

Choose the instructions below for the management tool you are using for Apigee hybrid:

Helm

Update the Cassandra restore details in the overrides.yaml file:

namespace: YOUR_RESTORE_NAMESPACE # Use the same namespace as in your original cluster.
cassandra:
  hostNetwork: false
...
restore:
  enabled: true
  serviceAccountPath: "SA_JSON_FILE_PATH"
  dbStorageBucket: "CLOUD_STORAGE_BUCKET_PATH"
  cloudProvider: "GCP"  # required verbatim "GCP" (all caps)
  snapshotTimestamp: "TIMESTAMP"
...
backup:
  enabled: false
...

Where:

Property	Description
`namespace`	`YOUR_RESTORE_NAMESPACE` Namespace for restore. Use the same namespace as in your original cluster.
`cassandra:hostNetwork`	`hostNetwork` is required and should always be set to `false`.
`restore:enabled`	Restore is disabled by default. You must set this property to `true`.
`restore:serviceAccountPath`	`SA_JSON_FILE_PATH` The path on your filesystem to the service account you created for the backup.
`restore:dbStorageBucket`	`CLOUD_STORAGE_BUCKET_PATH` The Cloud Storage bucket path where your backup data is stored in the following format: `gs://BUCKET_NAME`. The `gs://` is required.
`restore:cloudProvider`	`GCP` The `cloudProvider: "GCP"` property is required.
`restore:snapshotTimestamp`	`TIMESTAMP` The timestamp of the backup snapshot to restore. To check what timestamps can be used, go to the `dbStorageBucket` and look at the files that are present in the bucket. Each file name contains a timestamp value. For example, `backup_20210203213003_apigee-cassandra-default-0.tgz` Where `20210203213003` is the `snapshotTimestamp` value you would use if you wanted to restore the backups created at that point in time.
`backup:enabled`	You should set this property to `false` in case it had been previously set to `true`. Note: Backup must not be enabled when restoring if you are using Helm. This setting is optional if you are using `apigeectl`.

In case you do not have a clean cluster to start out with, follow the Decommission a hybrid region for helm documentation to bring your existing Hybrid installation into a clean state (you can leave the Cert Manager installed). This would bring you to an equal state as if you would have followed Helm runtime setup manual until the beginning of Step 11.

Verify there are no pods remaining in the Apigee namespaces:

kubectl get pods -n apigee
        kubectl get pods -n apigee-system

If you are using CSI backup, make sure that you can see the volumesnapshots you want to use for the restoration process by running:
```
kubectl get volumesnapshot -n apigee
          
```
Install all Hybrid components one by one as described in Step 11 on the installation manual. Note that the apigee-cassandra-restore pod will get created once you run the command to install the datastore, but it will only go into running state after you install the apigee-org component.

Important: Omit the ‑‑atomic flag for all upgrade commands for migrated clusters.

See Cassandra backup overview for more details on Cassandra backup and restore.

`apigeectl`

In your configuration, the Cassandra backup can reside either on Cloud Storage or on a remote server. In either case, perform the following steps to restore:

Verify the hybrid version.
```
apigeectl version
```
Ensure the version is the same version that created the backup files in storage.
Tip: Run every apigeectl command with the --dry-run=true flag first to check for any apigeectl errors.
Confirm that the Kubernetes cluster you are restoring to does not have a prior Apigee hybrid installation. If you are restoring to the existing cluster, use the following commands to delete the existing Apigee hybrid installation:
```
apigeectl delete -f overrides.yaml
kubectl -n apigee get apigeedatastore,apigeeredis,apigeetelemetry,org,env,arc # The output should be empty.
apigeectl delete --all -f overrides.yaml
```

Open your overrides.yaml file and set the restore properties to the desired values:

Cloud Storage

Parameters

namespace: YOUR_RESTORE_NAMESPACE # Use the same namespace as in your original cluster.
cassandra:
  hostNetwork: false
  ...
  restore:
    enabled: true
    serviceAccountPath: "SA_JSON_FILE_PATH"
    dbStorageBucket: "CLOUD_STORAGE_BUCKET_PATH"
    cloudProvider: "GCP"  # required verbatim "GCP" (all caps)
    snapshotTimestamp: "TIMESTAMP"
  ...
  backup:
    enabled: false
    serviceAccountPath: "SA_JSON_FILE_PATH"
    dbStorageBucket: "CLOUD_STORAGE_BUCKET_PATH"
    cloudProvider: "GCP"  # required verbatim "GCP" (all caps)
    schedule: "SCHEDULE"

Example

namespace: apigee
cassandra:
  hostNetwork: false
  ...
  restore:
    enabled: true
    serviceAccountPath: "/Users/myhome/.ssh/my_cassandra_backup.json"
    dbStorageBucket: "gs://myname-cassandra-backup"
    cloudProvider: "GCP"
    snapshotTimestamp: "20201001183903"

  ...
  backup:
    enabled: false
    serviceAccountPath: "/Users/myhome/.ssh/my_cassandra_backup.json"
    dbStorageBucket: "gs://myname-cassandra-backup"
    cloudProvider: "GCP"
    schedule: "0 2 * * *"
  ...

Where:

Property	Description
`namespace`	`YOUR_RESTORE_NAMESPACE` Namespace for restore. Use the same namespace as in your original cluster.
`cassandra:hostNetwork`	`hostNetwork` is required and should always be set to `false`.
`restore:enabled`	Restore is disabled by default. You must set this property to `true`.
`restore:serviceAccountPath`	`SA_JSON_FILE_PATH` The path on your filesystem to the service account you created for the backup.
`restore:dbStorageBucket`	`CLOUD_STORAGE_BUCKET_PATH` The Cloud Storage bucket path where your backup data is stored in the following format: `gs://BUCKET_NAME`. The `gs://` is required.
`restore:cloudProvider`	`GCP` The `cloudProvider: "GCP"` property is required.
`restore:snapshotTimestamp`	`TIMESTAMP` The timestamp of the backup snapshot to restore. To check what timestamps can be used, go to the `dbStorageBucket` and look at the files that are present in the bucket. Each file name contains a timestamp value. For example, `backup_20210203213003_apigee-cassandra-default-0.tgz` Where `20210203213003` is the `snapshotTimestamp` value you would use if you wanted to restore the backups created at that point in time.
`backup:enabled`	You should set this property to `false` in case it had been previously set to `true`. Note: Backup must not be enabled when restoring if you are using Helm. This setting is optional if you are using `apigeectl`.
`backup:serviceAccountPath`	`SA_JSON_FILE_PATH` The path on your filesystem to the service account JSON file that was downloaded when you ran `./tools/create-service-account`
`backup:dbStorageBucket`	`CLOUD_STORAGE_BUCKET_PATH` The Cloud Storage bucket path in this format: `gs://BUCKET_NAME`. The `gs://` is required.
`backup:cloudProvider`	`GCP` The `cloudProvider: "GCP"` property is required.
`backup:schedule`	`SCHEDULE` The time when the backup starts, specified in standard crontab syntax. Default: `0 2 * * ` Note:* Avoid scheduling a backup that starts a short time after you apply the backup configuration to your cluster. When you apply the backup configuration, Kubernetes recreates the Cassandra nodes. If the backup starts before the nodes restart (possibly several minutes) the backup will fail.

Non-Cloud Storage

Parameters

  namespace: YOUR_RESTORE_NAMESPACE # Use the same namespace as in your original cluster.
  cassandra:
    hostNetwork: false
    ...
    restore:
      enabled: true
      keyFile: "PATH_TO_PRIVATE_KEY_FILE"
      server: "BACKUP_SERVER_IP"
      storageDirectory: "/home/apigee/BACKUP_DIRECTORY"
      cloudProvider: "HYBRID"  # required verbatim "HYBRID" (all caps)
      snapshotTimestamp: "TIMESTAMP"
    ...
    backup:
      enabled: false
      keyFile: "PATH_TO_PRIVATE_KEY_FILE"
      server: "BACKUP_SERVER_IP"
      storageDirectory: "/home/apigee/BACKUP_DIRECTORY"
      cloudProvider: "HYBRID" # required verbatim "HYBRID" (all caps)
      schedule: "SCHEDULE"

Example

  namespace: apigee
  cassandra:
    hostNetwork: false
    ...
    restore:
      enabled: true
      keyFile: "/Users/exampleuser/apigee-hybrid/hybrid-files/service-accounts/private.key"
      server: "34.56.78.90"
      storageDirectory: "/home/apigee/cassbackup"
      cloudProvider: "HYBRID"
      snapshotTimestamp: "20201001183903"
    ...
    backup:
      enabled: false
      keyFile: "/Users/exampleuser/apigee-hybrid/hybrid-files/service-accounts/private.key"
      server: "34.56.78.90"
      storageDirectory: "/home/apigee/cassbackup"
      cloudProvider: "HYBRID"
      schedule: "0 2 * * *"
    ...

Where:

Property	Description
`namespace`	`YOUR_RESTORE_NAMESPACE` Namespace for restore. Use the same name namespace as in your original cluster.
`cassandra:hostNetwork`	`hostNetwork` is required and should always be set to `false`.
`restore:enabled`	Restore is disabled by default. You must set this property to `true`.
`restore:keyFile`	`PATH_TO_PRIVATE_KEY_FILE` The path on your local file system to the SSH private key file (named `ssh_key` in the step where you created the SSH key pair).
`restore:server`	`BACKUP_SERVER_IP` The IP address of your backup server.
`restore:storageDirectory`	`BACKUP_DIRECTORY` The name of the backup directory on your backup server. This must be a directory within `home/apigee` (the backup directory is named `cassandra_backup` in the step where you created the backup directory).
`restore:cloudProvider`	`HYBRID` The `cloudProvider: "HYBRID"` property is required.
`restore:snapshotTimestamp`	`TIMESTAMP` The timestamp of the backup snapshot to restore. To check what timestamps can be used, go to the `dbStorageBucket` and look at the files that are present in the bucket. Each file name contains a timestamp value. For example, `backup_20210203213003_apigee-cassandra-default-0.tgz` Where `20210203213003` is the `snapshotTimestamp` value you would use if you wanted to restore the backups created at that point in time.
`backup:enabled`	You should set this property to `false` in case it has been previously set to `true`. Note: Backup must not be enabled when restoring if you are using Helm. This setting is optional if you are using `apigeectl`.
`backup:keyFile`	`PATH_TO_PRIVATE_KEY_FILE` The path on your local file system to the SSH private key file (named `ssh_key` in the step where you created the SSH key pair).
`backup:server`	`BACKUP_SERVER_IP` The IP address of your backup server.
`backup:storageDirectory`	`BACKUP_DIRECTORY` The name of the backup directory on your backup server. This must be a directory within `home/apigee` (the backup directory is named `cassandra_backup` in the step where you created the backup directory).
`backup:cloudProvider`	`HYBRID` The `cloudProvider: "HYBRID"` property is required.
`backup:schedule`	`SCHEDULE` The time when the backup starts, specified in standard crontab syntax. Default: `0 2 * * ` Note:* Avoid scheduling a backup that starts a short time after you apply the backup configuration to your cluster. When you apply the backup configuration, Kubernetes recreates the Cassandra nodes. If the backup starts before the nodes restart (possibly several minutes) the backup will fail.

Note: Apigee restores Cassandra from a Cloud Storage backup or a remote server backup based on the backup configuration in the overrides.yaml file.

Create a new hybrid runtime deployment. This will create a new Cassandra cluster and begin restoring the backup data into the cluster:

${APIGEECTL_HOME}/apigeectl init  -f overrides/overrides.yaml

${APIGEECTL_HOME}/apigeectl check-ready -f overrides/overrides.yaml

${APIGEECTL_HOME}/apigeectl apply -f overrides/overrides.yaml --restore

${APIGEECTL_HOME}/apigeectl check-ready -f overrides/overrides.yaml

Verify the restoration job progress and confirm that apigeeds and all the other pods are up:

Check apigeeds:
```
kubectl get apigeeds -n apigee
```
Check all other pods:
```
kubectl get pods -n apigee
```

Upon successful completion of the restore and confirmation that the runtime components are healthy, we recommend configuring a backup on the cluster:

Remove the restore configuration from the overrides-restore.yaml file.
Add the backup configuration to the overrides-restore.yaml file.

Apply the backup configuration with the following command:

Helm

helm upgrade datastore apigee-datastore/ \
--install \
--namespace apigee \
--atomic \
-f overrides.yaml

`apigeectl`

./apigeectl apply -f ../overrides.yaml