Mantenha tudo organizado com as coleções
Salve e categorize o conteúdo com base nas suas preferências.
Está a ver a documentação do Apigee e do Apigee Hybrid.
Veja a documentação do
Apigee Edge.
Sintoma
Quando replicar dados durante uma expansão multirregional, o estado CassandraDataReplication
pode apresentar um estado de erro e a replicação de dados pode falhar.
Mensagem de erro
Quando usa
kubectl para ver o estado da reconstrução:
kubectl -n apigee get apigeeds \
-o jsonpath="{.items[].status.cassandraDataReplication}{'\n'}"
Vê que um ou mais pods do Cassandra mostram um estado de erro e uma mensagem a indicar que a reconstrução falhou. Por exemplo:
{
"rebuildDetails": {
"apigee-cassandra-default-0": {
"message": "failed to rebuild from us-west1: java.lang.IllegalStateException : Unable to find sufficient sources for streaming range (-8567285182390470134,-8567154549835592965] in keyspace system_distributed",
"state": "error",
"updated": 1641581899
},
…
}
}
Pode haver problemas de conetividade de rede entre os pods do Cassandra em diferentes centros de dados.
Apigee Hybrid
Passos de diagnóstico comuns
Obtenha o estado da replicação de dados:
kubectl -n apigee get apigeeds \ -o jsonpath="{.items[].status.cassandraDataReplication}{'\n'}"
Se vir um erro com uma mensagem semelhante à especificada em
Mensagem de erro, significa que está a
observar este problema.
Causa: região de origem incorreta
Se especificar uma região de origem (datacenter) no ficheiro YAML de replicação de dados que seja diferente da origem real datacenter, a replicação de dados falha. Execute os passos em Diagnóstico para analisar este cenário e
execute os passos em Resolução para o corrigir.
Diagnóstico
Liste todos os pods do Cassandra na região de origem:
kubectl -n apigee get pods -l app=apigee-cassandra
Obtenha o valor datacenter real de qualquer um dos pods do Cassandra devolvidos
no passo 1:
Obtenha o valor usado para source.region no ficheiro de recursos personalizados (YAML) de replicação de dados do Cassandra que criou em
Implementação em várias regiões. Se estiver a usar o nome de ficheiro de exemplo encontrado na documentação de implementação em várias regiões, o ficheiro deve ter o nome datareplication.yaml.
Datacenter: dc-1================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.104.13.2 491.84 KiB 256 100.0% 7254711c-fe0a-4b34-b50f-861109f97936 ra-1
UN 10.104.11.3 527.36 KiB 256 100.0% 5ec389f0-fd67-4de6-9f21-172d5899ff78 ra-1
UN 10.104.12.7 838.46 KiB 256 100.0% 7a88be82-1f81-4117-86e3-2cda434c0878 ra-1
Tenha em atenção que o valor source.region (us-west1) do ficheiro
datareplication.yaml não corresponde ao valor datacenter real
(dc-1) da saída do estado nodetool. Siga os passos na secção
Resolução para corrigir a configuração.
Resolução
Para corrigir a replicação de dados, tem de eliminar a tarefa de replicação de dados e criá-la com o nome datacenter correto. Siga estes passos:
Elimine o processo de replicação de dados atual. Se usar o nome de ficheiro de exemplo encontrado na documentação de
Implementação em várias regiões, o ficheiro deve ter o nome
datareplication.yaml.
kubectl delete -f datareplication.yaml
Atualize o nome da região no ficheiro YAML para o valor datacenter correto, por exemplo:
dc-1:
O erro de replicação de dados também pode ser o resultado de problemas de conetividade entre os nós do Cassandra.
Diagnóstico
Siga estes passos para analisar este cenário:
Apresente todos os pods do Cassandra:
# list cassandra pods
kubectl -n=apigee get pods -l app=apigee-cassandra
Execute os seguintes comandos para usar o Telnet no primeiro pod do Cassandra no
segundo centro de dados (dc-2) a partir do primeiro pod do Cassandra no primeiro centro de dados
(dc-1) através da porta 7001:
k debug -it apigee-cassandra-default-0 -n apigee --image=nginx -- sh curl -v telnet://DC_2_APIGEE_CASSANDRA_DEFAULT_0_POD_IP:7001
Se o telnet for bem-sucedido, é apresentado um resultado semelhante ao seguinte:
* Rebuilt URL to: telnet://10.0.4.10:7001/
* Trying 10.0.4.10...
* TCP_NODELAY set
* Connected to 10.0.4.10 (10.0.4.10) port 7001 (#0)
Caso contrário, é apresentado um erro semelhante ao seguinte:
* Rebuilt URL to: telnet://10.0.4.10:7001/
* Trying 10.0.4.10...
* TCP_NODELAY set
* connect to 10.0.4.10 port 7001 failed: Connection refused
* Failed to connect to 10.0.4.10 port 7001: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.0.4.10 port 7001: Connection refused
A falha de conetividade do pod do Cassandra num centro de dados para o pod do Cassandra noutro centro de dados indica que tem de haver uma restrição de firewall ou algum tipo de problema de conetividade de rede.
Resolução
Se esta implementação híbrida do Apigee estiver no GKE, verifique se existem regras de firewall definidas que
bloqueiam o tráfego de um centro de dados para outro e analise o problema de
conectividade de rede consultando a
vista geral das regras de firewall da VPC.
Se esta implementação híbrida do Apigee estiver no GKE-on-prem, trabalhe com a equipa de rede relevante e analise o problema de conetividade de rede.
Se o problema persistir mesmo depois de seguir as instruções acima, reúna as seguintes informações de diagnóstico e, em seguida, contacte o apoio ao cliente da Google Cloud:
O ID do projeto do Google Cloud.
A organização do Apigee Hybrid.
Os ficheiros overrides.yaml de ambas as regiões de origem e novas,
ocultando todas as informações confidenciais.
[[["Fácil de entender","easyToUnderstand","thumb-up"],["Meu problema foi resolvido","solvedMyProblem","thumb-up"],["Outro","otherUp","thumb-up"]],[["Difícil de entender","hardToUnderstand","thumb-down"],["Informações incorretas ou exemplo de código","incorrectInformationOrSampleCode","thumb-down"],["Não contém as informações/amostras de que eu preciso","missingTheInformationSamplesINeed","thumb-down"],["Problema na tradução","translationIssue","thumb-down"],["Outro","otherDown","thumb-down"]],["Última atualização 2025-09-04 UTC."],[[["\u003cp\u003eThis documentation addresses data replication failures in Apigee hybrid multi-region deployments, specifically when the \u003ccode\u003eCassandraDataReplication\u003c/code\u003e status indicates an error.\u003c/p\u003e\n"],["\u003cp\u003eA common cause of replication failure is an incorrect source region specified in the Cassandra data replication YAML file, which can be diagnosed by comparing the \u003ccode\u003esource.region\u003c/code\u003e with the \u003ccode\u003edatacenter\u003c/code\u003e value from the \u003ccode\u003enodetool status\u003c/code\u003e command.\u003c/p\u003e\n"],["\u003cp\u003eAnother potential cause of replication failure is network connectivity issues between Cassandra pods in different data centers, which can be diagnosed using \u003ccode\u003ecurl\u003c/code\u003e and \u003ccode\u003etelnet\u003c/code\u003e to check for connection refusals.\u003c/p\u003e\n"],["\u003cp\u003eResolution for an incorrect source region involves deleting the existing data replication job, updating the YAML file with the correct \u003ccode\u003edatacenter\u003c/code\u003e value, and reapplying the configuration.\u003c/p\u003e\n"],["\u003cp\u003eResolution for network issues involves checking firewall rules in GKE or working with the networking team in GKE-on-prem, ensuring connectivity between Cassandra nodes across regions.\u003c/p\u003e\n"]]],[],null,["# Cassandra data replication failure\n\n*You're viewing **Apigee** and **Apigee hybrid** documentation.\nView [Apigee Edge](https://docs.apigee.com/api-platform/troubleshoot/404-support-d) documentation.*\n| **Note:** This document is applicable for Apigee hybrid users only.\n\nSymptom\n-------\n\n\nWhen replicating data during a multi-region expansion, the `CassandraDataReplication`\nstatus may show an error state and data replication may fail.\n\nError message\n-------------\n\nWhen you use\n[kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) to view the rebuild status: \n\n```\n kubectl -n apigee get apigeeds \\\n -o jsonpath=\"{.items[].status.cassandraDataReplication}{'\\n'}\"\n```\n\nYou see that one or more Cassandra pods show an error state and a message stating the rebuild\nhas failed. For example: \n\n```scdoc\n{\n \"rebuildDetails\": {\n \"apigee-cassandra-default-0\": {\n \"message\": \"failed to rebuild from us-west1: java.lang.IllegalStateException : Unable to find sufficient sources for streaming range (-8567285182390470134,-8567154549835592965] in keyspace system_distributed\",\n \"state\": \"error\",\n \"updated\": 1641581899\n },\n …\n }\n}\n```\n\nPossible causes\n---------------\n\nCommon diagnosis steps\n----------------------\n\n1. Fetch the status of the data replication: \n\n ```\n kubectl -n apigee get apigeeds \\\n -o jsonpath=\"{.items[].status.cassandraDataReplication}{'\\n'}\"\n ```\n2. If you see an error with a message similar to the one specified in [Error message](#error-message), then it indicates that you are observing this issue.\n\nCause: Incorrect source region\n------------------------------\n\n\nIf you specify a source region (`datacenter`) in your data replication YAML file that\nis different from the actual source `datacenter`, then the data replication will\nfail. Perform the steps in [Diagnosis](#diagnosis) to analyze this scenario and\nperform the steps in [Resolution](#resolution) to correct it.\n\n### Diagnosis\n\n1. List all the Cassandra pods in the source region: \n\n ```\n kubectl -n apigee get pods -l app=apigee-cassandra\n ```\n2. Get the actual `datacenter` value from any one of the Cassandra pods returned in [step 1](#step-1): \n\n ```\n kubectl -n apigee exec -it apigee-cassandra-default-0 -- \\\n nodetool -u JMX_user -pw JMX_password status\n ```\n3. Get the value used for `source.region` in the Cassandra data replication custom resource (YAML) file you created in [Multi-region deployment](/apigee/docs/hybrid/latest/multi-region#configure-apigee-hybrid-for-multi-region). If you are using the example file name found in the Multi-region deployment documentation, then the file should be named `datareplication.yaml`. \n\n ```\n cat datareplication.yaml\n ```\n\n Example results: \n\n ```actionscript-3\n apiVersion: apigee.cloud.google.com/v1alpha1\n kind: CassandraDataReplication\n metadata:\n name: region-expansion\n namespace: apigee\n spec:\n organizationRef: apigee-hybrid-example-org\n force: false\n source:\n region: \"us-west1\"\n ```\n4. Check the result of the `nodetool status` output and verify whether or\n not the `datacenter` value matches the `source.region` value:\n\n ```\n kubectl -n apigee exec -it apigee-cassandra-default-0 -- \\\n nodetool -u jmxuser -pw iloveapis123 status\n ```\n\n Example results: \n\n ```restructuredtext\n Datacenter: dc-1\n ================\n Status=Up/Down\n |/ State=Normal/Leaving/Joining/Moving\n -- Address Load Tokens Owns (effective) Host ID Rack\n UN 10.104.13.2 491.84 KiB 256 100.0% 7254711c-fe0a-4b34-b50f-861109f97936 ra-1\n UN 10.104.11.3 527.36 KiB 256 100.0% 5ec389f0-fd67-4de6-9f21-172d5899ff78 ra-1\n UN 10.104.12.7 838.46 KiB 256 100.0% 7a88be82-1f81-4117-86e3-2cda434c0878 ra-1\n ```\n5. Note that the `source.region` (`us-west1`) from the `datareplication.yaml` file does not match the actual `datacenter` value (`dc-1`) from the `nodetool` status output. Follow the steps in [Resolution](#resolution) to correct the configuration.\n\n### Resolution\n\n\nTo correct the data replication, you'll need to delete the data replication job and create it\nwith the correct `datacenter` name. Perform the following steps:\n\n1. Delete the current data replication process. If using the example file name found in the [Multi-region deployment](/apigee/docs/hybrid/latest/multi-region#configure-apigee-hybrid-for-multi-region) documentation, then the file should be named `datareplication.yaml`. \n\n ```\n kubectl delete -f datareplication.yaml\n ```\n2. Update the region name in the YAML file to the correct `datacenter` value, e.g. `dc-1`: \n\n ```actionscript-3\n apiVersion: apigee.cloud.google.com/v1alpha1\n kind: CassandraDataReplication\n metadata:\n name: region-expansion\n namespace: apigee\n spec:\n organizationRef: apigee-hybrid-example-org\n force: false\n source:\n region: \"dc-1\"\n ```\n3. Apply the updated data replication: \n\n ```\n kubectl apply -f datareplication.yaml\n ```\n4. Check the rebuild status using the following command and verify that you no longer see the error state previously reported: \n\n ```\n kubectl -n apigee get apigeeds \\\n -o jsonpath=\"{.items[].status.cassandraDataReplication}{'\\n'}\"\n ```\n5. If the problem still persists, proceed to [Cause: Network connectivity issues.](#network-connectivity-issues)\n\nCause: Network connectivity issues\n----------------------------------\n\nThe data replication error may also be the result of connectivity issues between Cassandra\nnodes.\n| **Note:** Connectivity must be checked between all Cassandra nodes in all regions.\n\n### Diagnosis\n\n\nPerform the following steps to analyze this scenario:\n\n1. List all the Cassandra pods: \n\n ```text\n # list cassandra pods\n kubectl -n=apigee get pods -l app=apigee-cassandra\n ```\n2. Execute the following commands to telnet to the first Cassandra pod in the second data center (`dc-2`) from the first Cassandra pod in the first data center (`dc-1`) using port `7001`: \n\n ```scdoc\n k debug -it apigee-cassandra-default-0 -n apigee --image=nginx -- sh curl -v telnet://DC_2_APIGEE_CASSANDRA_DEFAULT_0_POD_IP:7001\n ```\n3. If telnet was successful, an output similar to following is displayed: \n\n ```scdoc\n * Rebuilt URL to: telnet://10.0.4.10:7001/\n * Trying 10.0.4.10...\n * TCP_NODELAY set\n * Connected to 10.0.4.10 (10.0.4.10) port 7001 (#0)\n ```\n4. Otherwise, an error similar to following is displayed: \n\n ```scdoc\n * Rebuilt URL to: telnet://10.0.4.10:7001/\n * Trying 10.0.4.10...\n * TCP_NODELAY set\n * connect to 10.0.4.10 port 7001 failed: Connection refused\n * Failed to connect to 10.0.4.10 port 7001: Connection refused\n * Closing connection 0\n curl: (7) Failed to connect to 10.0.4.10 port 7001: Connection refused\n ```\n\n\n The connectivity failure from the Cassandra pod in one data center to the Cassandra pod in\n another data center indicates that there must be a firewall restriction or some sort of\n network connectivity issue.\n\n### Resolution\n\n1. If this Apigee hybrid deployment is on GKE, check to see if any firewall rules are set that block the traffic from one data center to another and analyze the network connectivity problem by referring to [VPC firewall rules overview](https://cloud.google.com/vpc/docs/firewalls).\n2. If this Apigee hybrid deployment is on GKE-on-prem, work with the relevant networking team and analyze the network connectivity problem.\n\nIf the problem still persists, go to [Must gather diagnostic information](#must-gather).\n\nMust gather diagnostic information\n----------------------------------\n\n\nIf the problem persists even after following the above instructions, gather the following\ndiagnostic information and then contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/):\n\n1. The Google Cloud Project ID.\n2. The Apigee hybrid organization.\n3. The `overrides.yaml` files from both source and new regions, masking any sensitive information.\n4. The `CassandraDataReplication` YAML file.\n5. Cassandra `nodetool status` output: \n\n ```\n kubectl -n apigee exec -it apigee-cassandra-default-0 -- \\\n nodetool -u JMX_user -pw JMX_password status\n ```\n6. Cassandra `nodetool describecluster` output: \n\n ```\n kubectl -n apigee exec -it apigee-cassandra-default-0 -- \\\n nodetool -u JMX_user -pw JMX_password describecluster\n ```"]]
