NOTE: Some aspects of this product are in Beta. The hybrid installation options are GA. To join the Beta program, reach out to your Apigee representative.

Runtime logs

Use the runtime logs to help diagnose problems with the apigee-runtime pods. Each runtime pod contains a message processor, which is responsible for processing API proxy traffic, executing policies, routing requests, and more.

Using runtime logs

Logs are a good way to troubleshoot problems with your installation. See Logging for details.

Use the logs command

You can view runtime system logs using the kubectl logs command. For example:

kubectl logs -f -n hybrid apigee-runtime-apigee-gcp-prod1-test-blue-hd77q

Viewing logs on the filesystem

You can find the message processor log files on the apigee-runtime pod's filesystem. To view the files, open a shell in an apigee-runtime pod:

kubectl exec -it -n namespace apigee-runtime-pod-name bash

For example:

kubectl exec -it -n hybrid apigee-runtime-apigee-gcp-prod1-test-blue-cnj5x bash

Once you have the pod shell open, you can find hybrid message processor logs under /opt/apigee/var/log/apigee-runtime/config.log.

Contract version information

The following table briefly describes log entries related to loading the contract from the management plane:

Log Entry Meaning
hybrid-runtime Apigee-Timer-9 INFO RUNTIME-SYNC-SERVICE - RuntimeSyncServiceImpl.updateToVersion() : Switched to newer version 750 from version null The MP changed to a newer version of the configuration contract.
hybrid-runtime Apigee-Timer-6 INFO RUNTIME-SYNC-SERVICE - RuntimeSyncServiceImpl.loadLatestVersionIfAvailable() : Found a newer version 750 The MP loaded the latest contract configuration version from version.properties.
hybrid-runtime Apigee-Timer-6 INFO RUNTIME-SYNC-SERVICE - RuntimeSyncServiceImpl.updateToVersion() : Attempting to update to a newer version 750 The MP is updating to a newer version of the configuration contract.
hybrid-runtime Apigee-Timer-6 INFO RUNTIME-SYNC-SERVICE - RuntimeSyncServiceImpl.updateToVersion() : Switched to newer version 751 from version 750 The MP loaded the latest contract configuration version from version.properties.

Deployment errors

The following runtime errors relate to proxy deployments:

Log Entry Meaning
hybrid-runtime org:hybrid env:prod Apigee-Main-6 ERROR i.a.h.e.i.HermesEngineImpl - HermesEngineImpl.handleHermesException() : Keyspace cache_hybrid_hybrid does not exist Keyspace cache_orgname_hybrid is not provisioned in Cassandra.
hybrid-runtime org:hybrid Apigee-Timer-9 ERROR c.a.s.e.k.HybridEncryptionKeyServiceImpl - HybridEncryptionKeyServiceImpl.getEncryptionKey() : Encryption key does not exist for organization hybrid and type KmsKey Keyspace kms_orgname_hybrid is not provisioned in Cassandra.

Message processor deployment error logs

Look for the following deployment error entries in the message processor logs:

Log entry Description
hybrid-runtime org:hybrid env:prod Apigee-Main-6 ERROR i.a.h.e.i.HermesEngineImpl - HermesEngineImpl.handleHermesException() : Keyspace cache_hybrid_hybrid does not exist Keyspace cache_orgname_hybrid is not provisioned in Cassandra.
hybrid-runtime org:hybrid Apigee-Timer-9 ERROR c.a.s.e.k.HybridEncryptionKeyServiceImpl - HybridEncryptionKeyS erviceImpl.getEncryptionKey() : Encryption key does not exist for organization hybrid and type KmsKey Keyspace kms_orgname_hybrid is not provisioned in Cassandra.

Message processor error codes

Error codes for the message processor start with runtime.sync.error-name. For example:

Log Entry Meaning
runtime.sync.ContractLoadFailedMessage: Error loading spec file Reason: The error could be caused by an invalid schema, corrupted contract specification file, or invalid contract. Any of these issues could prevent the message processor from loading the spec file.

Remedy: Make sure either the contract specification file is available, the file system is not corrupted, or the schema/contract is valid.