Cassandra 备份和恢复

本部分介绍了如何为在 Apigee Hybrid 运行时层面中安装的 Apache Cassandra 数据库环配置数据备份和恢复。另请参阅 Cassandra 数据存储区。

关于 Cassandra 备份，您需要了解哪些信息？

Cassandra 是复制的数据库，配置为在每个区域或数据中心至少有 3 个数据副本。Cassandra 使用流式复制和读取修复来维护在每个区域或数据中心中给定时间点的数据副本。

在混合环境中，Cassandra 备份默认不启用。不过，最好启用 Cassandra 备份，以防数据被意外删除。

备份了哪些内容？

本主题中介绍的备份配置可备份以下实体：

  • Cassandra 架构，包括用户架构（Apigee 密钥空间定义）
  • 每个节点的 Cassandra 分区令牌信息
  • Cassandra 数据的快照

备份数据存储在何处？

备份数据存储在您必须创建的 Google Cloud Storage 存储桶中。本主题涵盖创建和配置存储桶。

安排 Cassandra 备份

备份作为运行时层面中的 cron 作业进行安排。如需安排 Cassandra 备份，请执行以下操作：

  1. 运行以下 create-service-account 命令以创建具有标准 roles/storage.objectAdmin 角色的 Google Cloud 服务账号 (SA)。此 SA 角色允许您将备份数据写入 Cloud Storage。在 Hybrid 安装根目录下执行以下命令： 
    ./tools/create-service-account apigee-cassandra OUTPUT_DIR
    例如： 
    ./tools/create-service-account apigee-cassandra ./service-accounts
    如需详细了解 Google Cloud 服务账号，请参阅创建和管理服务账号。
  2. create-service-account 命令会保存包含服务账号私钥的 JSON 文件。该文件会保存在执行命令的目录中。执行以下步骤将需要此文件的路径。
  3. 创建 Cloud Storage 存储桶。 为存储桶指定合理的数据保留政策。Apigee 建议采用 15 天的数据保留政策。
  4. 打开您的 overrides.yaml 文件。
  5. 添加以下 cassandra.backup 属性以启用备份。请勿移除任何已配置的属性。

    参数

    cassandra:
  ...

  backup:
    enabled: true
    serviceAccountPath: SA_JSON_FILE_PATH
    dbStorageBucket: CLOUD_STORAGE_BUCKET_PATH
    schedule: BACKUP_SCHEDULE_CODE

  ...

    示例

    ...

cassandra:
  storage:
    type: gcepd
    capacity: 50Gi
    gcepd:
      replicationType: regional-pd
  sslRootCAPath: "/Users/myhome/ssh/cassandra.crt"
  sslCertPath: "/Users/myhome/ssh/cassandra.crt"
  sslKeyPath: "/Users/myhome/ssh/cassandra.key"
  auth:
    default:
      password: "abc123"
    admin:
      password: "abc234"
    ddl:
      password: "abc345"
    dml:
      password: "abc456"
  nodeSelector:
    key: cloud.google.com/gke-nodepool
    value: apigee-data
  backup:
    enabled: true
    serviceAccountPath: "/Users/myhome/.ssh/my_cassandra_backup.json"
    dbStorageBucket: "gs://myname-cassandra-backup"
    schedule: "45 23 * * 6"

  ...
    6. 其中：
    属性 说明
    backup:enabled 备份默认处于停用状态。您必须将此属性设为 true。
    backup:serviceAccountPath

    SA_JSON_FILE_PATH

    文件系统上指向运行 ./tools/create-service-account 时所下载的服务账号 JSON 文件的路径
    backup:dbStorageBucket

    CLOUD_STORAGE_BUCKET_PATH

    Cloud Storage 存储桶路径，格式如下：gs://BUCKET_NAME。gs:// 是必需的。
    backup:schedule

    BACKUP_SCHEDULE_CODE

    备份的开始时间，使用标准 crontab 语法指定。默认值：0 2 * * *
  6. 将配置更改应用到新集群。例如： 
    ./apigeectl apply -f overrides.yaml

恢复备份

恢复功能会从备份位置获取数据，并将其恢复到具有相同节点数的新 Cassandra 集群中。不会从旧的 Cassandra 集群获取数据。

以下恢复说明适用于使用 Google Cloud Storage 进行备份的单区域部署。对于多区域部署，请参阅 GKE 和 GKE On-Prem 上的多区域部署。

如需恢复 Cassandra 备份，请执行以下操作：

  1. 在现有 Kubernetes 集群中创建一个将用于恢复 Hybrid 运行时部署的新命名空间。请勿将原始命名空间名称用于新命名空间。请勿使用旧命名空间进行恢复。
  2. 在根混合安装目录中，创建新的 overrides-restore.yaml 文件。
  3. 将完整的 Cassandra 配置从原始 overrides.yaml 文件复制到新的 overrides-restore.yaml 文件中。请参阅以下示例命令。 
    cp ./overrides.yaml ./overrides-restore.yaml
  4. 将命名空间元素添加到新的 overrides-restore.yaml 文件。请勿使用与原始集群相同的命名空间。

    参数

    namespace: YOUR_RESTORE_NAMESPACE
cassandra:
  ...
  restore:
    enabled: true
    snapshotTimestamp: TIMESTAMP
    serviceAccountPath: SA_JSON_FILE_PATH
    dbStorageBucket: CLOUD_STORAGE_BUCKET_PATH
         image:
           pullPolicy: Always
  ...

    示例

    ...
    namespace: cassandra-restore
    cassandra:
      storage:
        type: gcepd
        capacity: 50Gi
        gcepd:
          replicationType: regional-pd
      sslRootCAPath: "/Users/myhome/ssh/cassandra.crt"
      sslCertPath: "/Users/myhome/ssh/cassandra.crt"
      sslKeyPath: "/Users/myhome/ssh/cassandra.key"
      auth:
        default:
          password: "abc123"
        admin:
          password: "abc234"
        ddl:
          password: "abc345"
        dml:
          password: "abc456"
      nodeSelector:
        key: cloud.google.com/gke-nodepool
        value: apigee-data

      restore:
        enabled: true
        snapshotTimestamp: "20210203213003"
        serviceAccountPath: "/Users/myhome/.ssh/my_cassandra_backup.json"
        dbStorageBucket: "gs://myname-cassandra-backup"
        image:
          pullPolicy: Always
    ...

    其中：

    属性 说明
    namespace

    YOUR_RESTORE_NAMESPACE

    您在第 1 步中为新的 Cassandra 集群创建的新命名空间的名称。请勿使用原始集群所用的命名空间。
    restore:enabled 恢复功能默认处于停用状态。您必须将此属性设为 true。
    restore:snapshotTimestamp

    TIMESTAMP

    要恢复的备份快照的时间戳。如需查看可以使用哪些时间戳，请转到 dbStorageBucket 并查看存储桶中存在的文件。每个文件名都包含时间戳值，如下所示：

    backup_20210203213003_apigee-cassandra-default-0.tgz

    其中，20210203213003 是您在希望恢复该时间点创建的备份时使用的 snapshotTimestamp 值。
    restore:serviceAccountPath

    SA_JSON_FILE_PATH

    文件系统上指向为备份创建的服务账号的路径。
    restore:dbStorageBucket

    CLOUD_STORAGE_BUCKET_PATH

    存储备份数据的 Cloud Storage 存储桶路径，格式如下：

    gs://BUCKET_NAME

    gs:// 是必需的。

  5. 执行以下命令，更改旧命名空间中任何 Cassandra 节点上的 app 标签： 
    kubectl label pods --overwrite --namespace=OLD_NAMESPACE -l app=apigee-cassandra app=apigee-cassandra-old
  6. 创建新的 Hybrid 运行时部署。这将创建新的 Cassandra 集群并开始将备份数据恢复到该集群中： 
    ./apigeectl init  -f ../overrides-restore.yaml

    ./apigeectl apply  -f ../overrides-restore.yaml

  7. 恢复完成后，必须切换流量才能使用新命名空间中的 Cassandra 集群。运行以下命令切换流量：

    kubectl get rs -n OLD_NAMESPACE # look for the 'apigee-connect' replicaset

    kubectl patch rs -n OLD_NAMESPACE APIGEE_CONNECT_RS_NAME -p '{"spec":{"replicas" : 0}}'
  8. 流量切换完成后，您可以通过移除 restore 配置并将 backup 配置添加到 overrides-restore.yaml 文件来重新配置已恢复集群上的备份。将 YOUR_RESTORE_NAMESPACE 替换为第 1 步中创建的新命名空间名称。
    namespace: YOUR_RESTORE_NAMESPACE
cassandra:
  ...
   backup:
    enabled: true
    serviceAccountPath: SA_JSON_FILE_PATH
    dbStorageBucket: CLOUD_STORAGE_BUCKET_PATH
    schedule: BACKUP_SCHEDULE_CODE
  ...

    然后使用以下命令应用 backup 配置：

    ./apigeectl apply  -f ../overrides-restore.yaml

查看恢复日志

您可以检查恢复作业日志，并使用 grep 检查 error，以确保恢复日志没有错误。

验证恢复已完成

使用以下命令检查恢复操作是否已完成：

kubectl get pods

输出内容类似如下： 

NAME                           READY     STATUS      RESTARTS   AGE
apigee-cassandra-default-0     1/1       Running     0          1h
apigee-cassandra-default-1     1/1       Running     0          1h
apigee-cassandra-default-2     1/1       Running     0          59m
apigee-cassandra-restore-b4lgf 0/1       Completed   0          51m

查看恢复日志

使用以下命令查看恢复日志：

kubectl logs -f apigee-cassandra-restore-b4lgf

输出内容类似如下： 

Restore Logs:

Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
to download file gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1/backup_20190405011309_schema.tgz
INFO: download successfully extracted the backup files from gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
finished downloading schema.cql
to create schema from 10.32.0.28

Warnings :
dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0

dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0


Warnings :
dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0

dclocal_read_repair_chance table option has been deprecated and will be removed in version 4.0

INFO: the schema has been restored
starting apigee-cassandra-default-0 in default
starting apigee-cassandra-default-1 in default
starting apigee-cassandra-default-2 in default
84 95 106
waiting on waiting nodes $pid to finish  84
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
INFO: restore downloaded  tarball and extracted the file from  gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: restore downloaded  tarball and extracted the file from  gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: restore downloaded  tarball and extracted the file from  gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO  12:02:28 Configuration location: file:/etc/cassandra/cassandra.yaml
…...

INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed

Summary statistics:
   Connections per host    : 3
   Total files transferred : 2
   Total bytes transferred : 0.378KiB
   Total duration          : 5048 ms
   Average transfer rate   : 0.074KiB/s
   Peak transfer rate      : 0.075KiB/s

progress: [/10.32.1.155]0:1/1 100% 1:1/1 100% [/10.32.0.28]1:1/1 100% 0:1/1 100% [/10.32.3.220]0:1/1 100% 1:1/1 100% total: 100% 0.000KiB/s (avg: 0.074KiB/s)
INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed
progress: [/10.32.1.155]0:1/1 100% 1:1/1 100% [/10.32.0.28]1:1/1 100% 0:1/1 100% [/10.32.3.220]0:1/1 100% 1:1/1 100% total: 100% 0.000KiB/s (avg: 0.074KiB/s)
INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed
INFO  12:02:41 [Stream #e013ee80-5863-11e9-8458-353e9e3cb7f9] All sessions completed
INFO: ./apigee/data/cassandra/data/ks1/user-9fbae960571411e99652c7b15b2db6cc restored successfully
INFO: Restore 20190405011309 completed
INFO: ./apigee/data/cassandra/data/ks1/user-9fbae960571411e99652c7b15b2db6cc restored successfully
INFO: Restore 20190405011309 completed
waiting on waiting nodes $pid to finish  106
Restore finished

验证备份作业

您还可以在安排备份 Cron 作业后验证备份作业。安排 Cron 作业后，您应该会看到类似如下的内容：

kubectl get pods

输出内容类似如下： 

NAME                                       READY     STATUS      RESTARTS   AGE
apigee-cassandra-default-0                 1/1       Running     0          2h
apigee-cassandra-default-1                 1/1       Running     0          2h
apigee-cassandra-default-2                 1/1       Running     0          2h
apigee-cassandra-backup-1554515580-pff6s   0/1       Running     0          54s

检查备份日志

备份作业：

  • 创建 schema.cql 文件。
  • 将它上传到您的存储桶。
  • 回应节点以同时备份数据和上传数据。
  • 等待所有数据上传完毕。
kubectl logs -f apigee-cassandra-backup-1554515580-pff6s

输出内容类似如下： 

myusername-macbookpro:cassandra-backup-utility myusername$ kubectl logs -f apigee-cassandra-backup-1554577680-f9sc4
starting apigee-cassandra-default-0 in default
starting apigee-cassandra-default-1 in default
starting apigee-cassandra-default-2 in default
35 46 57
waiting on process  35
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
Requested creating snapshot(s) for [all keyspaces] with snapshot name [20190406190808] and options {skipFlush=false}
Snapshot directory: 20190406190808
INFO: backup created cassandra snapshot 20190406190808
tar: Removing leading `/' from member names
/apigee/data/cassandra/data/ks1/mytest3-37bc2df0587811e98e8d875b0ed64754/snapshots/
/apigee/data/cassandra/data/ks1/mytest3-37bc2df0587811e98e8d875b0ed64754/snapshots/20190406190808/
/apigee/data/cassandra/data/ks1/mytest3-37bc2df0587811e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Data.db
Requested creating snapshot(s) for [all keyspaces] with snapshot name [20190406190808] and options {skipFlush=false}
Requested creating snapshot(s) for [all keyspaces] with snapshot name [20190406190808] and options {skipFlush=false}
Snapshot directory: 20190406190808
INFO: backup created cassandra snapshot 20190406190808
tar: Removing leading `/' from member names
/apigee/data/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/snapshots/
/apigee/data/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/snapshots/20190406190808/
/apigee/data/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/snapshots/
/apigee/data/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/snapshots/20190406190808/
/apigee/data/cassandra/data/system/prepared_statements-18a9c2576a0c3841ba718cd529849fef/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d/snapshots/
/apigee/data/cassandra/data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d/snapshots/20190406190808/
/apigee/data/cassandra/data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/peer_events-59dfeaea8db2334191ef109974d81484/snapshots/
/apigee/data/cassandra/data/system/peer_events-59dfeaea8db2334191ef109974d81484/snapshots/20190406190808/
/apigee/data/cassandra/data/system/peer_events-59dfeaea8db2334191ef109974d81484/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/system/built_views-4b3c50a9ea873d7691016dbc9c38494a/snapshots/
/apigee/data/cassandra/data/system/built_views-4b3c50a9ea873d7691016dbc9c38494a/snapshots/20190406190808/
/apigee/data/cassandra/data/system/built_views-4b3c50a9ea873d7691016dbc9c38494a/snapshots/20190406190808/manifest.json
……
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Filter.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-CompressionInfo.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Index.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Statistics.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Data.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Index.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Statistics.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-TOC.txt
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Statistics.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Summary.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Filter.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Summary.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Index.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/manifest.json
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Filter.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-2-big-Digest.crc32
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Summary.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Data.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-TOC.txt
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/schema.cql
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-CompressionInfo.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Digest.crc32
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-TOC.txt
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-Data.db
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-3-big-Digest.crc32
/apigee/data/cassandra/data/ks2/user-d6d39d70586311e98e8d875b0ed64754/snapshots/20190406190808/mc-1-big-CompressionInfo.db
……
/tmp/tokens.txt
/ [1 files][    0.0 B/    0.0 B]
Operation completed over 1 objects.
/ [1 files][    0.0 B/    0.0 B]
Operation completed over 1 objects.
INFO: backup created tarball and transferred the file to gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: removing cassandra snapshot
INFO: backup created tarball and transferred the file to gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
INFO: removing cassandra snapshot
Requested clearing snapshot(s) for [all keyspaces]
INFO: Backup 20190406190808 completed
waiting on process  46
Requested clearing snapshot(s) for [all keyspaces]
INFO: Backup 20190406190808 completed
Requested clearing snapshot(s) for [all keyspaces]
waiting on process  57
INFO: Backup 20190406190808 completed
waiting result
to get schema from 10.32.0.28
INFO: /tmp/schema.cql has been generated
Activated service account credentials for: [apigee-cassandra-backup-svc@gce-myusername.iam.gserviceaccount.com]
tar: removing leading '/' from member names
tmp/schema.cql
Copying from <TDIN>...
/ [1 files][    0.0 B/    0.0 B]
Operation completed over 1 objects.
INFO: backup created tarball and transferred the file to gs://gce-myusername-apigee-cassandra-backup/apigeecluster/dc-1
finished uploading schema.cql