Migrating an org to another cluster

This page describes how to migrate an Apigee hybrid org from one Kubernetes cluster to another. Some cases in which you might need to migrate an org to another cluster are the following:

The data center hosting the existing cluster has no more capacity, or is being decommissioned.
The cluster is running old infrastructure or an old version of Kubernetes, and you want to migrate to a cluster with newer infrastructure.
You want to move orgs from multi-org clusters into separate clusters.

Note that there are risks and limitations when migrating an org to another hybrid cluster. Read the details in the Limitations section before performing a migration.

Limitations

The following limitations apply when migrating a hybrid org to another Kubernetes cluster:

There is a risk of data loss when moving org data to a new Kubernetes cluster. You should back up the data for all orgs in the Kubernetes cluster, using the hybrid backup instructions, before migrating an org.
The maximum data size supported for org migration is 5GB across all keyspaces of an org, excluding cache and quota.
Cache data will not be migrated. Hybrid rebuilds cache data.
Quota data will not be migrated. Hybrid resets quota data.
You can only migrate orgs to a Kubernetes cluster that contains no existing hybrid deployment. Migrating to a cluster with an existing hybrid deployment is not supported.
The org being migrated can only be moved to a new cluster with a single region deployment. After the single region deployment is up and running, you can follow the region expansion process, described in Multi-region deployment, to expand to other regions.
The Cassandra cluster should be operating in good health across all regions.

Migrating an org

Follow the instructions below to migrate a hybrid org from one Kubernetes cluster to another:

If not already enabled, enable backups on the Kubernetes cluster containing the hybrid org to be migrated. See Cassandra backup overview for information on hybrid backups.
Start a hybrid backup job using the following command:
```
kubectl create job -n apigee --from=cronjob/apigee-cassandra-backup <backup job name>
```
The <backup job name> can be any valid container name.
Once the backup job is completed, use the instructions in the following sections of Monitoring backups to verify that the backup has successfully completed:
- "Check the status of the backup job"
- "Check the backup logs"
After verifying that the backup was successful, make a note of the ID number at the end of the backup log. For example, a successful backup log should contain a line like the following:
```
INFO: completed upload for 20230207004250
```
Make a note of the multi-digit number at the end of the line. You will need this number later.
Switch the Kubernetes context to the destination Kubernetes cluster:
```
kubectl config use-context <destination cluster name> # <destination cluster name>
```
where <destination cluster name> is the name of the destination Kubernetes cluster.
Restore the backup data into the destination Kubernetes cluster using the instructions in Restoring in a single region.
- Use the overrides.yaml file for the org that is being migrated to the destination hybrid deployment.
- Remember to set the restore:snapshotTimestamp value to the multi-digit number shown by the backup log in step 4. See Restoring in a single region.
Once the restoration is complete, delete any org data, other than the data for the org being migrated, from the destination Kubernetes cluster. Hybrid backup files contain the data for all orgs, including ones you may not want to migrate. After the destination hybrid deployment is restored, you need to remove any extra org data that was copied to the deployment, using the following steps:
1. Verify that the current context is the correct context for the destination Kubernetes cluster:
```
kubectl config current-context
```
2. Exec into the apigee-cassandra-default-0 pod:
```
kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
```
3. Execute the following command:
```
find /opt/apigee/data/apigee-cassandra/ -iname '*_hybrid' -not -iname '*<migrated org name>*' -type d -maxdepth 2 -printf "%f\n"
```
  See Get the name of the migrated org for instructions on finding <migrated org name>.
  
  Copy the list of all names that are shown in the output. You will need this list in step 7. f.
4. Exit the apigee-cassandra-default-0 pod.
5. Create a Cassandra debug client pod using the instructions in Create a client container for debugging. Move on to the next step after getting a cqlsh prompt.
6. Execute the following commands in the cqlsh prompt:
  - ```
  desc keyspaces;
```
  Make sure this command returns no errors.
- For each name in the list created in step 7. c., run the following command:
```
  drop keyspace <name>
```
7. Exit the Cassandra debug client pod.
8. After executing the cqlsh commands, run the following commands on all Cassandra pods in the destination Kubernetes cluster:
  - ```
  kubectl exec -it -n apigee  -- /bin/bash
```
- ```
find /opt/apigee/data/apigee-cassandra/ -iname '*_hybrid' -not -iname '*<migrated org name>*' -type d -maxdepth 2
```
    See Get the name of the migrated org for instructions on finding <migrated org name>.
  - ```
  find /opt/apigee/data/apigee-cassandra/ -iname '*_hybrid' -not -iname '**' -type d -maxdepth 2 -exec rm -rf {} +
```
9. Exit the Cassandra pod.
Switch the Kubernetes context to the source Kubernetes cluster:
```
kubectl config use-context <source cluster name>
```
where <source cluster name> is the name of the source Kubernetes cluster.

Delete the migrated org from the source Kubernetes cluster. Be sure to use the overrides.yaml file for the org in the delete command:

Verify the current context is the correct context for the source Kubernetes cluster:

kubectl config current-context

If needed, set the kubernetes contexts to the cluster and org needs to be decommissioned.

List your current contexts to see the context name for each cluster:

kubectl config get-contexts

Set the context to the cluster and region you want to decommission:

kubectl config use-context CONTEXT_NAME

Where CONTEXT_NAME is the context name for the cluster and region.

For example:

    kubectl config get-contexts
    CURRENT   NAME                                                   CLUSTER                                                AUTHINFO                                               NAMESPACE
              gke_example-org-1_us-central1_example-cluster-1        gke_example-org-1_us-central1_example-cluster-1        gke_example-org-1_us-central1_example-cluster-1        apigee
    *         gke_example-org-1_us-central1_example-cluster-2        gke_example-org-1_us-central1_example-cluster-2        gke_example-org-1_us-central1_example-cluster-2        apigee
              gke_example-org-1_us-west1_example-cluster-2           gke_example-org-1_us-west1_example-cluster-2           gke_example-org-1_us-west1_example-cluster-2           apigee

    kubectl config use-context gke_example-org-1_us-west1_example-cluster-2

Delete the virtualhost.

Helm

Repeat this for each environment group:

helm -n apigee delete ENV_GROUP_NAME

`apigeectl`

$APIGEECTL_HOME/apigeectl delete --settings virtualhost -f OVERRIDES_FILE.yaml

Delete the environments.

Helm

Repeat this for each environment:

helm -n apigee delete ENV_NAME

`apigeectl`

$APIGEECTL_HOME/apigeectl delete --all-envs -f OVERRIDES_FILE.yaml

Delete the Apigee org.

Helm

helm -n apigee delete ORG_NAME

`apigeectl`

$APIGEECTL_HOME/apigeectl delete -f OVERRIDES_FILE.yaml --org

Exec into the apigee-cassandra-default-0 pod:

kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash

Execute the following command:
```
find /opt/apigee/data/apigee-cassandra/ -iname '*<migrated org name>_hybrid' -type d -maxdepth 2 -printf "%f\n"
```
See Get the name of the migrated org for instructions on finding <migrated org name>.

Copy the list of all the names that are shown in the output. You will need this list in step 9. j.
Exit the apigee-cassandra-default-0 pod.
Create a Cassandra debug client pod using the instructions in Create a client container for debugging. Move on to the next step after getting a cqlsh prompt.
Execute the following commands at the cqlsh prompt:
```
desc keyspaces;
```
Make sure this command returns no errors.
For each name in the list created in step 10. f., run the following command:
```
drop keyspace <name>;
```
Exit the Cassandra debug client pod.

cqlsh

kubectl exec -it -n apigee <cassandra pod name> -- /bin/bash

find /opt/apigee/data/apigee-cassandra/ -iname '*<migrated org name>_hybrid' -type d -maxdepth 2

See Get the name of the migrated org for instructions on finding <migrated org name>.

find /opt/apigee/data/apigee-cassandra/ -iname '*<migrated org name>_hybrid' -type d -maxdepth 2 -exec rm -rf {} +

Exit the Cassandra pod.

Get the name of the migrated org

Several of the steps in the procedure described in the previous section require the name of the migrated org. To get the migrated org name, do the following:

Get the org name from the org's overrides.yaml file. Make sure to check the overrides.yaml file for the org being migrated.
If the org name contains any dashes "-", replace all dashes "-" with underscores "_".