This page describes how to migrate an Apigee hybrid org from one Kubernetes cluster to another. Some cases in which you might need to migrate an org to another cluster are the following:
- The data center hosting the existing cluster has no more capacity, or is being decommissioned.
- The cluster is running old infrastructure or an old version of Kubernetes, and you want to migrate to a cluster with newer infrastructure.
- You want to move orgs from multi-org clusters into separate clusters.
Note that there are risks and limitations when migrating an org to another hybrid cluster. Read the details in the Limitations section before performing a migration.
Limitations
The following limitations apply when migrating a hybrid org to another Kubernetes cluster:
- There is a risk of data loss when moving org data to a new Kubernetes cluster. You should back up the data for all orgs in the Kubernetes cluster, using the hybrid backup instructions, before migrating an org.
- The maximum data size supported for org migration is 5GB across all keyspaces of an org, excluding cache and quota.
- Cache data will not be migrated. Hybrid rebuilds cache data.
- Quota data will not be migrated. Hybrid resets quota data.
- You can only migrate orgs to a Kubernetes cluster that contains no existing hybrid deployment. Migrating to a cluster with an existing hybrid deployment is not supported.
- The org being migrated can only be moved to a new cluster with a single region deployment. After the single region deployment is up and running, you can follow the region expansion process, described in Multi-region deployment, to expand to other regions.
- The Cassandra cluster should be operating in good health across all regions.
Migrating an org
Follow the instructions below to migrate a hybrid org from one Kubernetes cluster to another:
- If not already enabled, enable backups on the Kubernetes cluster containing the hybrid org to be migrated. See Cassandra backup overview for information on hybrid backups.
- Start a hybrid backup job using the following command:
kubectl create job -n apigee --from=cronjob/apigee-cassandra-backup <backup job name>
The
<backup job name>
can be any valid container name. - Once the backup job is completed, use the instructions in the following sections of
Monitoring
backups to verify that the backup has successfully completed:
- "Check the status of the backup job"
- "Check the backup logs"
- After verifying that the backup was successful, make a note of the ID number at the end
of the backup log.
For example, a successful backup log should contain a line like the following:
Make a note of the multi-digit number at the end of the line. You will need this number later.INFO: completed upload for 20230207004250
- Switch the Kubernetes context to the destination Kubernetes cluster:
kubectl config use-context <destination cluster name> # <destination cluster name>
where
<destination cluster name>
is the name of the destination Kubernetes cluster. - Restore the backup data into the destination Kubernetes cluster using the instructions in
Restoring in a single region.
- Use the overrides.yaml file for the org that is being migrated to the destination hybrid deployment.
- Remember to set the
restore:snapshotTimestamp
value to the multi-digit number shown by the backup log in step 4. See Restoring in a single region.
- Once the restoration is complete, delete any org data, other than the data for the org being
migrated, from the destination Kubernetes cluster. Hybrid backup files contain the data for all
orgs, including ones you may not want to migrate. After the destination hybrid
deployment is restored, you need to remove any extra org data that was copied to the
deployment, using the following steps:
- Verify that the current context is the correct context for the destination Kubernetes cluster:
kubectl config current-context
- Exec into the
apigee-cassandra-default-0
pod:kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
- Execute the following command:
find /opt/apigee/data/apigee-cassandra/ -iname '*_hybrid' -not -iname '*<migrated org name>*' -type d -maxdepth 2 -printf "%f\n"
See Get the name of the migrated org for instructions on finding
<migrated org name>
.Copy the list of all names that are shown in the output. You will need this list in step 7. f.
- Exit the
apigee-cassandra-default-0
pod. - Create a Cassandra debug client pod using the instructions in
Create a client container for debugging. Move on to the next step after getting a
cqlsh
prompt. - Execute the following commands in the
cqlsh
prompt:-
desc keyspaces;
Make sure this command returns no errors.
- For each name in the list created in step 7. c., run the following command:
drop keyspace <name>
-
- Exit the Cassandra debug client pod.
- After executing the
cqlsh
commands, run the following commands on all Cassandra pods in the destination Kubernetes cluster:kubectl exec -it -n apigee
-- /bin/bash find /opt/apigee/data/apigee-cassandra/ -iname '*_hybrid' -not -iname '*<migrated org name>*' -type d -maxdepth 2
See Get the name of the migrated org for instructions on finding
<migrated org name>
.find /opt/apigee/data/apigee-cassandra/ -iname '*_hybrid' -not -iname '*
*' -type d -maxdepth 2 -exec rm -rf {} +
- Exit the Cassandra pod.
- Verify that the current context is the correct context for the destination Kubernetes cluster:
- Switch the Kubernetes context to the source Kubernetes cluster:
kubectl config use-context <source cluster name>
where
<source cluster name>
is the name of the source Kubernetes cluster. - Delete the migrated org from the source Kubernetes cluster. Be sure to use the
overrides.yaml
file for the org in the delete command:- Verify the current context is the correct context for the source Kubernetes cluster:
kubectl config current-context
If needed, set the kubernetes contexts to the cluster and org needs to be decommissioned.
List your current contexts to see the context name for each cluster:
kubectl config get-contexts
Set the context to the cluster and region you want to decommission:
kubectl config use-context CONTEXT_NAME
Where CONTEXT_NAME is the context name for the cluster and region.
For example:
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE gke_example-org-1_us-central1_example-cluster-1 gke_example-org-1_us-central1_example-cluster-1 gke_example-org-1_us-central1_example-cluster-1 apigee * gke_example-org-1_us-central1_example-cluster-2 gke_example-org-1_us-central1_example-cluster-2 gke_example-org-1_us-central1_example-cluster-2 apigee gke_example-org-1_us-west1_example-cluster-2 gke_example-org-1_us-west1_example-cluster-2 gke_example-org-1_us-west1_example-cluster-2 apigeekubectl config use-context gke_example-org-1_us-west1_example-cluster-2
- Delete the virtualhost.
Helm
Repeat this for each environment group:
helm -n apigee delete ENV_GROUP_NAME
apigeectl
$APIGEECTL_HOME/apigeectl delete --settings virtualhost -f OVERRIDES_FILE.yaml
- Delete the environments.
Helm
Repeat this for each environment:
helm -n apigee delete ENV_NAME
apigeectl
$APIGEECTL_HOME/apigeectl delete --all-envs -f OVERRIDES_FILE.yaml
- Delete the Apigee org.
Helm
helm -n apigee delete ORG_NAME
apigeectl
$APIGEECTL_HOME/apigeectl delete -f OVERRIDES_FILE.yaml --org
- Exec into the apigee-cassandra-default-0 pod:
kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
- Execute the following command:
find /opt/apigee/data/apigee-cassandra/ -iname '*<migrated org name>_hybrid' -type d -maxdepth 2 -printf "%f\n"
See Get the name of the migrated org for instructions on finding
<migrated org name>
.Copy the list of all the names that are shown in the output. You will need this list in step 9. j.
- Exit the
apigee-cassandra-default-0
pod. - Create a Cassandra debug client pod using the instructions in
Create a client container for debugging. Move on to the next step
after getting a
cqlsh
prompt. - Execute the following commands at the
cqlsh
prompt:desc keyspaces;
Make sure this command returns no errors.
- For each name in the list created in step 10. f.,
run the following command:
drop keyspace <name>;
- Exit the Cassandra debug client pod.
After executing the - Verify the current context is the correct context for the source Kubernetes cluster:
-
kubectl exec -it -n apigee <cassandra pod name> -- /bin/bash
-
find /opt/apigee/data/apigee-cassandra/ -iname '*<migrated org name>_hybrid' -type d -maxdepth 2
See Get the name of the migrated org for instructions on finding
<migrated org name>
. -
find /opt/apigee/data/apigee-cassandra/ -iname '*<migrated org name>_hybrid' -type d -maxdepth 2 -exec rm -rf {} +
- Exit the Cassandra pod.
cqlsh
commands, run the following commands on all Cassandra pods
in the source Kubernetes cluster:
Get the name of the migrated org
Several of the steps in the procedure described in the previous section require the name of the migrated org. To get the migrated org name, do the following:
- Get the org name from the org's overrides.yaml file. Make sure to check the overrides.yaml file for the org being migrated.
- If the org name contains any dashes "-", replace all dashes "-" with underscores "_".