Decommission a hybrid org

This document outlines the steps to decommission an org from a hybrid deployment. Decommissioning an org means deleting all data related to the org across all Cassandra pods in all Kubernetes clusters.

Limitations

Only one org can be decommissioned at a time. Decommissioning multiple orgs simultaneously is not supported.

Get the org name

Some commands in these instructions require you to use a properly formatted org name.

To get the org name for use in commands on this page:

Retrieve the org name from the org's overrides.yaml file.
If the org name contains any dashes ("-"), replace them with underscores ("_").

Instructions

Follow these instructions to decommission an org from a hybrid deployment.

Back up the org
1. If it's not already enabled, enable backups on the hybrid deployment. In multi-region setups, use the hybrid deployment operating in the primary region. See Cassandra backup overview for information on hybrid backups.
2. Trigger a hybrid backup job using the following command:
```
kubectl create job -n apigee --from=cronjob/apigee-cassandra-backup BACKUP_JOB_NAME
```
  The BACKUP_JOB_NAME can be any valid container name.
3. Once the backup job completes, use the "Check the status of the backup job" and "Check the backup logs" instructions in Monitoring backups to verify the backup was successful.
(Optional) If you have configured Apigee Telemetry (Metrics and Logger) on the org to be deleted, follow these steps to reconfigure them so that the metrics and log data apply to a new org/project.

Run this command for the org where you want to send the data. Be sure to use the overrides.yaml file for the org. For example, if the org to be decommissioned is "test-dev" the overrides.yaml file should contain an org: test-dev org field.
Helm
```
helm upgrade telemetry apigee-telemetry/ \
  --namespace apigee \
  --atomic \
  -f OVERRIDES_FILE.yaml
```
Note: If you see an error saying Error: UPGRADE FAILED: "ORG_NAME" has no deployed releases, replace upgrade with install and try the command again.
apigeectl
```
$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE.yaml --telemetry
```

Run this command, making sure to use the correct org/project:

kubectl -n apigee get apigeetelemetry apigee-telemetry -oyaml | grep `gcpProjectID:`

Delete the org's Kubernetes resources from the hybrid deployment.

Helm

Verify the current context is the correct context for the source Kubernetes cluster:

kubectl config current-context

If needed, set the Kubernetes context.

List your current contexts to see the context name for each cluster:

kubectl config get-contexts

Set the context to the cluster and region you want to decommission:

kubectl config use-context CONTEXT_NAME

Where CONTEXT_NAME is the context name for the cluster and region.

For example:

    kubectl config get-contexts
    CURRENT   NAME                                                   CLUSTER                                                AUTHINFO                                               NAMESPACE
              gke_example-org-1_us-central1_example-cluster-1        gke_example-org-1_us-central1_example-cluster-1        gke_example-org-1_us-central1_example-cluster-1        apigee
    *         gke_example-org-1_us-central1_example-cluster-2        gke_example-org-1_us-central1_example-cluster-2        gke_example-org-1_us-central1_example-cluster-2        apigee
              gke_example-org-1_us-west1_example-cluster-2           gke_example-org-1_us-west1_example-cluster-2           gke_example-org-1_us-west1_example-cluster-2           apigee

    kubectl config use-context gke_example-org-1_us-west1_example-cluster-2

Delete the virtualhost.
Repeat this for each environment group:
```
helm -n apigee delete ENV_GROUP_NAME
```
Delete the environments. Repeat this for each environment:
```
helm -n apigee delete ENV_NAME
```
Delete the Apigee org.
```
helm -n apigee delete ORG_NAME
```

`apigeectl`

Be sure to use the overrides.yaml file for the org. For example, if the org to be decommissioned is "test-dev" the overrides.yaml file should contain an org: test-dev org field.

For multi-region hybrid deployments, run these commands against each hybrid deployment in each region.

kubectl config current-context # Verify the current context is the correct context for the hybrid deployment
apigeectl check-ready -f overrides.yaml # Check the deployment status
apigeectl delete --settings virtualhost -f overrides.yaml
apigeectl check-ready -f overrides.yaml # Check the deployment status
apigeectl delete --all-envs -f overrides.yaml
apigeectl check-ready -f overrides.yaml # Check the deployment status
apigeectl delete --org -f overrides.yaml
apigeectl check-ready -f overrides.yaml # Check the deployment status

Delete the org data from the hybrid deployment. Once this step is complete all org data will be gone from the hybrid deployment.
1. Exec into the apigee-cassandra-default-0 pod:
```
kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
```
2. Execute the following command. Copy the list of all the names that are shown in the output. This list will be needed later.
```
find /opt/apigee/data/apigee-cassandra/ -iname '*ORG_NAME_hybrid' -type d -maxdepth 2 -printf "%f\n"
```
  See Get org name for instructions on how to find and prepare the ORG_NAME.
  
  Exit from the apigee-cassandra-default-0 pod.
3. Create a Cassandra debug client pod as described in Create a client container for debugging. Move on to the next step after getting a cqlsh prompt.
4. Execute the following commands in the cqlsh prompt:
```
desc keyspaces;
```
  Make sure this command returns no errors.
  
  For each name in the list created earlier from the apigee apigee-cassandra-default-0, run the following commands:
```
drop keyspace ;
```
  Exit from the Cassandra debug client pod.
5. Perform a rolling restart of all Cassandra pods. Restarting the Cassandra pods can be done in any order as long as only one Cassandra pod is restarted at a time. For multi-region deployments, perform a rolling restart on all Cassandra pods in each hybrid region.
  Run the following command and verify the state shows "Running":
```
kubectl get apigeeds -n apigee
```
  Restart a single Cassandra pod with the following command:
```
kubectl delete pod -n apigee CASSANDRA_POD_NAME
```
  Wait for the pod to reach Running state using:
```
kubectl get pods -n apigee
```
  Restart the next Cassandra pod.
6. After executing the cqlsh commands, run the following commands on all Cassandra pods in the hybrid deployment. For multi-region hybrid deployments, run the commands on all Cassandra pods in all hybrid regions.
```
kubectl exec -it -n apigee CASSANDRA_POD_NAME -- /bin/bash
```
```
find /opt/apigee/data/apigee-cassandra/ -iname '*ORG_NAME_hybrid' -type d -maxdepth 2
```
  See Get org name for instructions on how to find and prepare the ORG_NAME.
```
find /opt/apigee/data/apigee-cassandra/ -iname '*ORG_NAME_hybrid' -type d -maxdepth 2 -exec rm -rf {} +
```
7. Exit from the Cassandra pod.