MessageLogging policy

You're viewing Apigee X documentation.
View Apigee Edge documentation.

policy icon

What

The MessageLogging policy lets you log custom messages to Cloud Logging or syslog. You can use the information in the logs for various tasks, such tracking down problems in the API runtime environment.

There are two ways to use the MessageLogging policy:

  • The <CloudLogging> element logs messages to Cloud Logging. To use this method, you need to enable the Cloud Logging APIs for your GCP project. For more information about enabling APIs for a GCP project, see Enabling and Disabling Services.
  • The <Syslog> element logs messages to syslog, a standard protocol for sending system log or event messages to a specific server. To use this method, you must have a syslog server available. If you don't, you can use public log management services, such a Splunk, Sumo Logic, and Loggly. See Configuring third-party log management services.

Note: You cannot use both the <CloudLogging> element and the <Syslog> element in the same policy.

<MessageLogging> element

Defines a <MessageLogging> policy.

Default value See Default Policy tab, below
Required? Required
Type TYPE
Parent Element n/a
Child Elements <CloudLogging>
<Syslog>

The <MessageLogging> element uses the following syntax:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging continueOnError="false" enabled="true" name="Message-Logging-1">
    <DisplayName>Message Logging-1</DisplayName>
    <Syslog>
<!-- Note: You cannot use both the <Syslog> element and the <CloudLogging> element in the same policy. -->
        <Message>Some message for syslog</Message>
        <Host>localhost</Host>
        <Port>514</Port>
    </Syslog>
    <CloudLogging>
<!-- Note: You cannot use both the <CloudLogging> and the <Syslog> element in the same policy. -->
        <LogName>projects/{organization.name}/logs/{log.id}</LogName>
        <Message contentType="application/json">{"{message.queryparam.key}": "{message.queryparam.value}"}</Message>
        <Labels>
            <Label>
                <Key>key1</Key>
                <Value>value1</Value>
            </Label>
            <Label>
                <Key>key2</Key>
                <Value>value2</Value>
            </Label>
        </Labels>
        <ResourceType>gce_instance</ResourceType>
    </CloudLogging>
    <logLevel>ALERT</logLevel>
</MessageLogging>

This element has the following attributes that are common to all policies:

Attribute Default Required? Description
name N/A Required

The internal name of the policy. The value of the name attribute can contain letters, numbers, spaces, hyphens, underscores, and periods. This value cannot exceed 255 characters.

Optionally, use the <DisplayName> element to label the policy in the management UI proxy editor with a different, natural-language name.

continueOnError false Optional Set to false to return an error when a policy fails. This is expected behavior for most policies. Set to true to have flow execution continue even after a policy fails. See also:
enabled true Optional Set to true to enforce the policy. Set to false to turn off the policy. The policy will not be enforced even if it remains attached to a flow.
async   false Deprecated This attribute is deprecated.

The following table provides a high-level description of the child elements of <MessageLogging>:

Field Name Field Description
CloudLogging

Configure messages to be logged to Cloud logging.

Syslog

Configure messages to be logged to syslog.

Samples

CloudLogging

<MessageLogging name="LogToCloudLogging">
    <CloudLogging>
        <LogName>projects/{organization.name}/logs/{log.id}</LogName>
        <Message contentType="application/json">{"{message.queryparam.key}": "{message.queryparam.value}"}</Message>
        <Labels>
            <Label>
                <Key>key1</Key>
                <Value>value1</Value>
            </Label>
            <Label>
                <Key>key2</Key>
                <Value>value2</Value>
            </Label>
        </Labels>
        <ResourceType>gce_instance</ResourceType>
    </CloudLogging>
</MessageLogging>

This example illustrates the use of message templates. Since the Message element contains the flow variables

{"{message.queryparam.key}": "{message.queryparam.value}"}

when someone calls the proxy with the values message.queryparam.key = "fruit" and message.queryparam.value = "apple", the resulting log entry would be {"fruit": "apple"}.

Syslog

<MessageLogging name="LogToSyslog">
  <Syslog>
    <Message>[3f509b58 tag="{organization.name}.{apiproxy.name}.{environment.name}"] Weather request for WOEID {request.queryparam.w}.</Message>
    <Host>logs-01.loggly.com</Host>
    <Port>514</Port>
    <Protocol>TCP</Protocol>
    <FormatMessage>true</FormatMessage>
  </Syslog>
  <logLevel>ALERT</logLevel>
</MessageLogging>

In this example, suppose that you need to log information about each request message that your API receives from consumer apps. The value 3f509b58 represents a key value specific to the loggly service. If you have a loggly account, substitute your loggly key. The log message that is generated will be populated with four values: the organization, API proxy, and environment name associated with the transaction, along with the value for a query parameter on the request message.

Syslog over TLS/SSL

<MessageLogging name="LogToSyslog">
  <Syslog>
    <Message>[3f509b58 tag="{organization.name}.{apiproxy.name}.{environment.name}"] Weather request for WOEID {request.queryparam.w}.</Message>
    <Host>logs-01.loggly.com</Host>
    <Port>6514</Port>
    <Protocol>TCP</Protocol>
    <FormatMessage>true</FormatMessage>
    <SSLInfo>
        <Enabled>true</Enabled>
    </SSLInfo>
  </Syslog>
  <logLevel>WARN</logLevel>
</MessageLogging>

You can send messages to third-party message logging providers over TLS/SSL by adding the <SSLInfo> block.

File rotation: size

<MessageLogging name="LogPolicy">
  <File>
    <Message>This is a test message. Message id : {request.header.messageid}</Message>
      <FileName>test.log</FileName>
      <FileRotationOptions rotateFileOnStartup="true">
        <FileRotationType>SIZE</FileRotationType>
        <MaxFileSizeInMB>10</MaxFileSizeInMB>
        <MaxFilesToRetain>10</MaxFilesToRetain>
      </FileRotationOptions>
  </File>
  <logLevel>ERROR</logLevel>
</MessageLogging>

File rotation based on file size.

File rotation: time

<MessageLogging name="LogPolicy">
  <File>
    <Message>This is a test message. Message id : {request.header.messageid}</Message>
    <FileName>test.log</FileName>
    <FileRotationOptions rotateFileOnStartup="true">
      <FileRotationType>TIME</FileRotationType>
      <RotationFrequency unit="minute">10</RotationFrequency>
      <MaxFilesToRetain>10</MaxFilesToRetain>
    </FileRotationOptions>
  </File>
  <logLevel>ERROR</logLevel>
</MessageLogging>

File rotation based on time.

File rotation: time & size

<MessageLogging name="LogPolicy">
  <File>
    <Message>This is a test message. Message id : {request.header.messageid}</Message>
    <FileName>test.log</FileName>
    <FileRotationOptions rotateFileOnStartup="true">
      <FileRotationType>TIME_SIZE</FileRotationType>
      <MaxFileSizeInMB>10</MaxFileSizeInMB>
      <MaxFilesToRetain>10</MaxFilesToRetain>
      <RotationFrequency unit="minute">10</RotationFrequency>
    </FileRotationOptions>
  </File>
  <logLevel>ERROR</logLevel>
</MessageLogging>

File rotation based on time and size.

Stream-enabled

<MessageLogging name="LogPolicy">
  <File>
  ....
  ....
  </File>
  <BufferMessage>true</BufferMessage>
</MessageLogging>

Stream-enabled message logging

Child element reference

The following sections describe the child elements of <MessageLogging>

<CloudLogging>

Use the <CloudLogging> element to log messages to Cloud Logging.

Field Name Required? Description
LogName Yes Name of the log. The log name should be in the format projects/{PROJECT_ID}/logs/{LOG_ID}. You can use variables in place of {PROJECT_ID} and {LOG_ID}.
Message Yes

The message to be logged. The message has an attribute contentType, whose value can either be text/plain or application/json for text and JSON messages respectively. See the Samples.

Label No Label to be attached to the log message, if any. These will be in the form of a key-value pair like the following:
<Label>
    <Key>key1</Key>
    <Value>value1</Value>
</Label>
ResourceType No (defaults to global) Represents the monitored resource that is generating the logs.

Authentication for Cloud Logging

To use the <CloudLogging> element, you must deploy your API proxy to use Google authentication. Apigee will use credentials corresponding to the identity of the service account you specify in the outbound requests to Cloud Logging. For more details, see Using Google Authentication. The service account that you attach to your API proxy at deployment time must have a role with the logging.logEntries.create permission. For more information about Identity and Access Management (IAM) roles for <CloudLogging>, see the Access control guide.

<Syslog>

Use the <Syslog> element to configure messages to be logged to syslog. When you use the <Syslog>, an API proxy forwards log messages from Apigee to a remote syslog server. To use this method, you must have a syslog server available. If you don't, public log management services, such a Splunk, Sumo Logic, and Loggly, are available. see Configuring third-party log management services.

Field Name Required? Field Description
Message Yes

The message to be logged. The message has an attribute contentType, whose value can either be text/plain or application/json for text and JSON messages respectively. See the Samples.

Host No The hostname or IP address of the server where the syslog should be sent. If you don't include this element, the default is localhost.
Port No Port where the syslog is running. If you don't include this element, the default is 514.
Protocol No TCP or UDP (default). While UDP is more performant, the TCP protocol guarantees message log delivery to the syslog server. For sending syslog messages over TLS/SSL, only TCP is supported.
FormatMessage No, but <FormatMessage>true</FormatMessage> is required for use with Loggly.

true or false (default)

This element lets you control the format of Apigee-generated content prepended to the message. If set to true, the syslog message is prepended by a fixed number of characters, which lets you filter out that information from messages. Here's an example for the fixed format:

<14>1 2016-02-23T09:24:39.039+0000 e49cd3a9-4cf6-48a7-abb9-7ftfe4d97d00 Apigee - - - Message starts here

The Apigee-generated information includes:

  • <14> - A priority score (see the Syslog Protocol) based on the log level and facility level of the message.
  • 1 - The current syslog version.
  • Date with a UTC offset (UTC = +0000).
  • Message processor UUID.
  • "Apigee - - - "

If set to false (default), the message is not prepended with those fixed characters.

PayloadOnly No

true or false (default)

This element sets the format of Apigee-generated messages to contain only the body of the syslog message, without the prepended characters specified by FormatMessage.

If you don't include this element or leave it empty, the default value is false.

See FormatMessage.

SSLInfo No

Lets you log messages through SSL/TLS. Use with sub-element <Enabled>true</Enabled>.

If you don't include this element or leave it empty, the default value is false (no TLS/SSL).

<SSLInfo>
    <Enabled>true</Enabled>
</SSLInfo>

You can configure the <SSLInfo> tag the same as you can on a TargetEndpoint, including enabling two-way TLS/SSL, as described in API proxy configuration reference. Only the TCP protocol is supported.

logLevel No

Valid values: INFO (default), ALERT, WARN, ERROR.

Sets a specific level of information to be included in the message log.

If you're using the FormatMessage element (setting it to true), your logLevel setting affects the calculated priority score (the number inside the angle brackets) in the Apigee-generated information prepended to the message.


Usage notes

When attaching a MessageLogging policy to an API proxy flow, consider placing it in the ProxyEndpoint response, in a special flow called PostClientFlow. The PostClientFlow executes after the response is sent to the requesting client, which ensures that all metrics are available for logging. For details on using PostClientFlow, see API proxy configuration reference.

The PostClientFlow is special in two ways:

  1. It only executed as part of the response flow.
  2. It is the only flow executed after the proxy enters the error state.

Because it is executed regardless of whether the proxy succeeded or failed, you can put MessageLogging policies in the PostClientFlow and be guaranteed that they always execute.

The following Debug image shows a MessageLogging policy executing as part of the PostClientFlow, after the DefaultFaultRule executes:

In this example, the Verify API Key policy caused the fault because of an invalid key.

Shown below is the ProxyEndpoint definition that includes the PostClientFlow:

<ProxyEndpoint name="default">
  ...
  <PostClientFlow>
    <Response>
      <Step>
        <Name>Message-Logging-1</Name>
      </Step>
    </Response>
  </PostClientFlow>
  ...
</ProxyEndpoint>

Apigee logs messages as simple text, and you can configure logging to include variables, such as the date and time when the request or response was received, the user identity on the request, the source IP address from which the request was sent, and so on.

Apigee logs message asynchronously: the response is returned while the logs are still being written. As a result, no latency is introduced to your API by blocking callouts. There may might be occasions when a log is not written without an error being returned, but these events are rare.

The MessageLogging policy writes logged messages in memory to a buffer. The message logger reads messages from the buffer and then writes to the destination that you configure. Each destination has its own buffer.

If the write rate to the buffer increases beyond the read rate, the buffer overflows and logging will fail. If this happens, you might find one of the following messages in the log file:

  • Using <CloudLoggingg>:
    steps.messagelogging.TooManyPendingLoggingRequest
  • Using <Syslog>:
    Log message size exceeded. Increase the max message size setting

Increase the max.log.message.size.in.kb property (default value = 128 KB) in the message-logging.properties file.

Default values for variables in message template

Default values can be specified for each variable in message template separately. For example, if the variable request.header.id cannot be resolved, then its value is replaced with the value unknown.

<Message>This is a test message. id = {request.header.id:unknown}</Message>

A common default value can be specified for all the unresolved variables by setting the defaultVariableValue attribute on the Message element:

<Message defaultVariableValue="unknown">This is a test message. id = {request.header.id}</Message>

Configuring third-party log management services

The MessageLogging policy lets you send syslog messages to third-party log management services, such as Splunk, Sumo Logic, and Loggly. If you want to send syslog to one of those services, see that service's documentation to configure the service's host, port, and protocol, then set the Syslog element on this policy accordingly.

See the following documentation for third-party log management configuration:

Error reference

This section describes the fault codes and error messages that are returned and fault variables that are set by Apigee when this policy triggers an error. This information is important to know if you are developing fault rules to handle faults. To learn more, see What you need to know about policy errors and Handling faults.

Runtime errors

These errors can occur when the policy executes.

Fault code HTTP status Cause
steps.messagelogging.StepDefinitionExecutionFailed 500 See fault string.
steps.messagelogging.InvalidGoogleCloudLogName 500 This error is thrown when the LogName does not evaluate to the valid format of projects/{project}/logs/{logid}.
steps.messagelogging.InvalidJsonMessage 500 This error is thrown when the contentType attributes value has been chosen as application/json but the actual message value is not a valid JSON string,
steps.messagelogging.TooManyPendingLoggingRequest 500 This error is thrown when there are more than 2500 pending requests that are yet to be written to Cloud Logging. The 2500 limit is for each Apigee runtime pod. For example, if the traffic is distributed over two instances of Apigee runtime pods, the effective limit is 5000 requests.
-

Deployment errors

These errors can occur when you deploy a proxy containing this policy.

Error name Cause Fix
InvalidProtocol The deployment of the MessageLogging policy can fail with this error if the protocol specified within the <Protocol> element is not valid. The valid protocols are TCP and UDP. For sending syslog messages over TLS/SSL, only TCP is supported.
InvalidPort The deployment of the MessageLogging policy can fail with this error if the port number is not specified within the <Port> element or if it is not valid. The port number must be an integer greater than zero.

Fault variables

These variables are set when a runtime error occurs. For more information, see What you need to know about policy errors.

Variables Where Example
fault.name="fault_name" fault_name is the name of the fault, as listed in the Runtime errors table above. The fault name is the last part of the fault code. fault.name Matches "StepDefinitionExecutionFailed"
messagelogging.policy_name.failed policy_name is the user-specified name of the policy that threw the fault. messagelogging.ML-LogMessages.failed = true

Example error response

{  
   "fault":{
      "detail":{
         "errorcode":"steps.messagelogging.StepDefinitionExecutionFailed"
      },
      "faultstring":"Execution failed"
   }
}

Example fault rule

<FaultRule name="MessageLogging">
    <Step>
        <Name>ML-LogMessages</Name>
        <Condition>(fault.name Matches "StepDefinitionExecutionFailed") </Condition>
    </Step>
    <Condition>(messagelogging.ML-LogMessages.failed = true) </Condition>
</FaultRule>


Flow variables

The following variables are populated on policy failure.

  • messagelogging.failed
  • messagelogging.{stepdefinition-name}.failed

Related topics