Enable southbound mTLS for configurable proxies

This page applies to Apigee and Apigee hybrid.

View Apigee Edge documentation.

As of August 17, 2023, Configurable API proxies (Preview) is deprecated. Configurable proxies will no longer be available after November 10, 2023. For more information, see Feature deprecations.

This topic describes how to enable southbound Mutual TLS (mTLS) to secure traffic from your Apigee Configurable API Proxy to mTLS-enabled proxy targets and backends. Mutual TLS (mTLS) enables authentication of both client and server endpoints in a TLS handshake, allowing users to ensure that both the client and the server are trusted entities. Many business enterprises reply on mTLS to ensure that all traffic within a network is trusted and secure. By adding mTLS functionality to configurable proxies, Apigee customers can seamlessly maintain their current usage of mTLS when transitioning to the use of configurable proxies, or increase security for communications between existing configurable proxies and their backends.

The following sections outline the steps required to set up mTLS support for southbound (i.e. gateway -> backend) traffic to Apigee configurable proxies.

Configurable^PREVIEW API proxy development is available only to customers with Apigee paid Subscription orgs. Apigee customers with Pay-as-you-go orgs can create programmable API proxies.

Overview of mTLS concepts

The following diagram shows how the mTLS handshake works between a client and a server, in this case, a configurable proxy and a target backend:

In the diagram above, both the server and the client have a keystore containing their own certificate/public key and private key, indicating the possibility of a "mutual" TLS handshake. During verification, the configurable proxy client will compare the certificate in its truststore with the certificate sent by the target server backend.

When performing the TLS handshake:

1. The TLS server presents its certificate to the TLS client to authenticate itself.
2. The client verifies the identity of the server and then presents its own certificate to authenticate itself to the server.
3. Once mutually authentication, the client and server calculate a shared secret key used to encrypt requests and responses between them.

Deploy an mTLS-enabled configurable proxy

The following section outlines the steps required to deploy a configurable proxy with mTLS enabled for traffic between the gateway and its target backend server.

Step 1: Configure the source files

Before you can deploy an mTLS-enabled configurable proxy, you must set up a few new files, including the following:

• A truststore (a certificate-only alias), key, and certificate alias used by your configurable proxy gateway to communicate with the target backend server.
• An Apigee keystore that contains the truststore, key, and certificate described above.
• An archive deployment containing the keystore details in the target server configuration.

Note: The following example steps will use a self-signed certificate, which is not recommended for production use.

To create the required source files:

1. Create the private certificate authority (CA) private and public keys using the following command:

   openssl req \
     -new \
     -x509 \
     -nodes \
     -days 365 \
     -subj '/CN=my-ca' \
     -keyout RootCA.key \
     -out RootCA.pem

2. Create the configurable proxy gateway certificate, private key, and certificate signing request (CSR) using the following commands:

   openssl genrsa -out my_gateway.key 2048
   openssl req -new -key my_gateway.key -out my_gateway.csr
   

3. Sign the newly created client cert by using the certificate authority you created previously:

   openssl x509 -req -in my_gateway.csr -CA RootCA.pem \
     -CAkey RootCA.key -set_serial 01 -out my_gateway.pem -days 365 -sha256

4. Create a server certificate, private key, and certificate signing request (CSR) using the following commands:

   openssl genrsa -out my_server.key 2048
   openssl req -new -key my_server.key -subj '/CN=${SERVER_HOST}' -out my_server.csr
   

5. Sign the newly created server cert by using the certificate authority you created previously:

   openssl x509 -req -in my_server.csr -CA RootCA.pem \
     -CAkey RootCA.key -set_serial 01 -out my_server.pem -days 365 -sha256

Upon successful completion, you should have eight new files in your directory, as shown in the example below:

ls \
  RootCA.key
  RootCA.pem // Certificate used for the Apigee truststore
  my_gateway.csr
  my_gateway.key  // Used for the Apigee key
  my_gateway.pem // Used for the client cert alias
  my_server.csr
  my_server.key // Used for the server key
  my_server.pem // Used for the server cert


  

Step 2: Configure the Keystore

Create and configure an Apigee keystore using the following steps:

1. Open Apigee UI in a browser.
2. Select Admin > Environments > TLS Keystores.
3. Click the + Keystore button to create a new keystore.
4. Enter a name in the New Keystore dialog, as shown in the figure below:
5. Click Add Keystore.
6. From the TLS Keystores landing page, select the row for the keystore you just created, then click + to create a new alias.
7. On the Create new alias screen:
   • Enter the Alias Name.
   • Select Certificate and Key from the Type dropdown of the Certificate Details panel.
   • Use the file picker to choose the Keystore Certificate File created in a previous step.
   • Enter a Key Password.
   • Use the file picker to choose the Key File created in a previous step.
8. Once the fields are complete, as shown in the image below, click Save to save the alias and certificate details.
9. On the TLS Keystores landing page, select the row for the keystore you just created, then click + to create a new alias for the truststore.
10. On the Create new alias screen:
    • Enter the Alias Name.
    • Select Certificate Only from the Type dropdown of the Certificate Details panel.
    • Use the file picker to choose the Truststore Certificate File created in a previous step.
    • Once the fields are complete, as shown in the image below, click Save to save the alias and certificate details.

Now you have a keystore and truststore within Apigee that contains the TLS details required when deploying the configurable proxy archive with mTLS.

Step 3: Format the Archive

Now that you have a keystore holding your certificates and key, you can create the archive used to deploy the configurable proxy with mTLS.

Note: If you have not already set up an archive, follow the steps in Deploying and managing archives in an Apigee environment before proceeding.

Once complete, your archive file structure should look like this:

src/
|  main/apigee
|  |  apiproxies/helloworld
|  |  |  config.yaml
|  |  environments/dev
|  |  |  deployments.json
|  |  |  targetservers.json
…    

The targetservers.json file is used to hold the references to the keystore configured in the previous step. The file should contain references to the reference the keystore, alias, and truststore created in Step 2. For example:

#targetservers.json
[
    {
        "name": "mymtlstarget",
        "description": "An example use mTLS",
        "host": "foo",
        "port": 8080,
        "isEnabled": true,
        "sSLInfo": {
                "enabled": true,
                "clientAuthEnabled": true,
                "keyStore": "mtls_keystore",
                "keyAlias": "mtls_alias",
                "trustStore": "mtls_truststore",
                "ignoreValidationErrors": false,
                "protocols": [
                "1.2"
                ],
                }
        },
        "protocol": "HTTP"
    },
    …
]

The only fields that are required in the SSLInfo section of the targetservers.json file when enabling mTLS are the following:

• enabled
• clientAuthEnabled
• keyStore
• keyAlias
• trustStore

Optional fields include:

• protocols
• ciphers

You can now reference this target server within your archive proxy. For example, apiproxies/helloworld/config.yaml could contain an operation like the following:

...
operations:
  - id: mTLSOperation
   target:
      target_server_id: "mymtlstarget"
    http_match:
      - path_template: "/foo"
        method: GET
...

Step 4: Deploy the proxy

Once the archive setup is complete, deploy the configurable proxy using the following command:

$ gcloud alpha apigee archives deploy --env=ENV_NAME --org=ORG_NAME --source=PATH_TO_SOURCE

Where:

• ENV_NAME is the Apigee environment where your proxy should be deployed
• ORG_NAME is the name of your Apigee organization
• PATH_TO_SOURCE is the relative path to the source directory containing your archive configuration

For example:

$ gcloud alpha apigee archives deploy --env=dev --org=myorg --source=src

When the deployment operation completes, you should see the following message:

Waiting for operation [b64c2665-b5ac-43cc-9e2d-232e8895c2ed] to complete...done.

If the deployment fails, see Archive Deployment Errors for troubleshooting issues related to this step.

Step 5: Verify the proxy

Once your proxy successfully deploys, you can use a command like the following to ensure it works as expected:

$ curl https://ORG_NAME-ENV_NAME.PROXY_URL

Where:

• ENV_NAME is the Apigee environment where your proxy should be deployed
• ORG_NAME is the name of your Apigee organization
• PROXY_URL in the URL of your deployed proxy

For example, if you are calling a helloworld proxy that should return a message on success:

$ curl https://myorg-dev.apigee.net/helloworld

If your proxy is working as expected, you should see the following:

{
        "message": "Hello world!"
}

Update a certificate for your proxy

Apigee does not auto-renew certificates on your behalf. To keep your proxies working as expected, you may need to update the certificates used by your environments from time to time. The following steps outline how to create a new alias or truststore under the same keystore used for your configurable proxy.

Note: You cannot currently perform an in-place update on an in-use alias, truststore, or reference for Configurable Proxies. See Update a keystore alias and Update a keystore reference for more information.

To update an alias certificate:

1. Open Apigee UI in a browser.
2. Select Admin > Environments > TLS Keystores and select the keystore you wish to update.
3. Create a new alias object and select the new Keystore certificate file as described in Step 2: Configure the Keystore above.
4. Update the targetservers.json file to reference your new keystore object. For example,

   $ cat src/main/apigee/apiproxies/helloworld/apiproxy/environments/dev/targetservers.json
   [
       {
           "name": "mymtlstarget",
           "description": "An example use mTLS",
           "host": "foo",
           "port": 8080,
           "isEnabled": true,
           "sSLInfo": {
                   "enabled": true,
                   "clientAuthEnabled": true,
                   "keyStore": "mtls_keystore",
                   "keyAlias": "new_mtls_alias",
                   "trustStore": "mtls_truststore",
                   "ignoreValidationErrors": false,
                   "protocols": [
                   "1.2"
                   ],
                   }
           },
           "protocol": "HTTP"
       },
       …
   ]

5. Deploy the new archive. For example:

   $ gcloud alpha apigee archives deploy --env=ENV_NAME --org=ORG_NAME --source=PATH_TO_SOURCE

   Where:

   • ENV_NAME is the Apigee environment where your proxy should be deployed
   • ORG_NAME is the name of your Apigee organization
   • PATH_TO_SOURCE is the relative path to the source directory containing your archive configuration

   For example:

   $ gcloud alpha apigee archives deploy --env=dev --org=myorg --source=src
   

   When the deployment operation completes, you should see the following message:

   Waiting for operation [b64c2665-b5ac-43cc-9e2d-232e8895c2ed] to complete...done.

Your proxy should now be using the updated keystore certificate file in the alias object. Confirm that your proxy is working as expected using the steps outlined in Step 4: Verify the proxy above.

To update a truststore certificate:

1. Open Apigee UI in a browser.
2. Select Admin > Environments > TLS Keystores and select the keystore you wish to update.
3. Create a new alias object and select the new Keystore certificate file as described in Step 2: Configure the Keystore above.
4. Update the targetservers.json file to reference your new keystore object. For example,

   $ cat src/main/apigee/apiproxies/helloworld/apiproxy/environments/dev/targetservers.json
   [
       {
           "name": "mymtlstarget",
           "description": "An example use mTLS",
           "host": "foo",
           "port": 8080,
           "isEnabled": true,
           "sSLInfo": {
                   "enabled": true,
                   "clientAuthEnabled": true,
                   "keyStore": "mtls_keystore",
                   "keyAlias": "mtls_alias",
                   "trustStore": "new_mtls_truststore",
                   "ignoreValidationErrors": false,
                   "protocols": [
                   "1.2"
                   ],
                   }
           },
           "protocol": "HTTP"
       },
       …
   ]

5. Deploy the new archive. For example:

   $ gcloud alpha apigee archives deploy --env=ENV_NAME --org=ORG_NAME --source=PATH_TO_SOURCE

   Where:

   • ENV_NAME is the Apigee environment where your proxy should be deployed
   • ORG_NAME is the name of your Apigee organization
   • PATH_TO_SOURCE is the relative path to the source directory containing your archive configuration

   For example:

   $ gcloud alpha apigee archives deploy --env=dev --org=myorg --source=src
   

   When the deployment operation completes, you should see the following message:

   Waiting for operation [b64c2665-b5ac-43cc-9e2d-232e8895c2ed] to complete...done.

Your proxy should now be using the updated truststore certificate file in the alias object. Confirm that your proxy is working as expected using the steps outlined in Step 4: Verify the proxy above.

Troubleshooting and error guide

This section outlines a number of common configuration issues and error messages you may encounter when configuring mTLS for your configurable proxy. Where possible, potential solutions to these issues are suggested.

Keystore update errors

This section describes errors you may encounter when making updates to keystore files.

Update a keystore alias

If you attempt to update a truststore or keystore alias with a new certificate for a configurable proxy environment without deploying a new archive, you will see the following error:

Alias updates are currently disabled for configurable proxy environments. To update the certificate,
create a new alias and reference it within a new archive deployment.

Update a keystore reference

If you attempt to update a keystore reference for a configurable proxy environment without deploying a new archive, you will see the following error:

Reference updates are currently disabled for configurable proxy environments.
To update the reference, create a new reference and reference it within a new archive deployment.

Archive Deployment Errors

This section describes error cases you may encounter when deploying the configurable proxy archive.

Missing Keystore

Description

In this scenario, the specified archive deployment references a non-existent keystore within its targetservers.json.

Example

$ gcloud alpha apigee operations describe b64c2665-b5ac-43cc-9e2d-232e8895c2ed

Using Apigee organization 'myorg'
done: true
error:
  code: 3
  message: 'generic::invalid_argument: proxy "organizations/{org}/apis/{api}/revisions/1"
    contains reference(s) to invalid target server(s): target server "mtlstarget-docs"
    contains invalid TLS settings: missing required private key for "alias \"organizations/{org}/environments/{env}/keystores/does-not-exist/aliases/my_alias\"";
    missing required certificate(s) for "alias \"organizations/{org}/environments/{env}/keystores/does-not-exist/aliases/my_alias\""'
metadata:
  '@type': type.googleapis.com/google.cloud.apigee.v1.OperationMetadata
  operationType: INSERT
  state: FINISHED
  targetResourceName: organizations/{org}/environments/{env}/archiveDeployments/afjq46noszlmmgt3uh
name: organizations/{org}/operations/6b7dbb09-fa01-4cb1-8c3f-9393027b3d92
organization: {org}
uuid: 6b7dbb09-fa01-4cb1-8c3f-9393027b3d92

Resolution

To resolve this error, you can either:

1. Create a keystore with the same name that is referenced within the targetservers.json file.
2. Update the targetservers.json file to reference an existing, valid Apigee keystore.

Missing Alias

Description

In this scenario, the specified archive deployment references a non-existent alias within its targetservers.json file:

Example

$ gcloud alpha apigee operations describe b64c2665-b5ac-43cc-9e2d-232e8895c2ed

Using Apigee organization 'myorg'
done: true
error:
  code: 3
  message: 'generic::invalid_argument: proxy "organizations/{org}/apis/{api}/revisions/1"
    contains reference(s) to invalid target server(s): target server "mtlstarget-docs"
    contains invalid TLS settings: missing required private key for "alias \"organizations/{org}/environments/{env}/keystores/docs-keystore/aliases/does-not-exist\"";
    missing required certificate(s) for "alias \"organizations/{org}/environments/{env}/keystores/docs-keystore/aliases/does-not-exist\""'
metadata:
  '@type': type.googleapis.com/google.cloud.apigee.v1.OperationMetadata
  operationType: INSERT
  state: FINISHED
  targetResourceName: organizations/{org}/environments/{env}/archiveDeployments/knvztvpqxbpojcdi9o
name: organizations/{org}/operations/2917bb4a-ebcb-4c59-8f48-187f6423da5d
organization: {org}
uuid: 2917bb4a-ebcb-4c59-8f48-187f6423da5d

Resolution

To resolve this error, you can either:

1. Create an alias within the specified keystore with the same name that is referenced in the targetservers.json file
2. Update the targetservers.json file to reference an existing, valid Apigee keystore and alias combination.

Missing Truststore

Description

In this scenario, the specified archive deployment references a non-existent alias within its targetservers.json file:

Example

$ gcloud alpha apigee operations describe b64c2665-b5ac-43cc-9e2d-232e8895c2ed

Using Apigee organization 'myorg'
done: true
error:
  code: 3
  message: 'generic::invalid_argument: proxy "organizations/{org}/apis/{api}/revisions/1"
    contains reference(s) to invalid target server(s): target server "mtlstarget-docs"
    contains invalid TLS settings: missing required certificate(s) for truststore
    "organizations/{org}/environments/{env}/keystores/does-not-exist"'
metadata:
  '@type': type.googleapis.com/google.cloud.apigee.v1.OperationMetadata
  operationType: INSERT
  state: FINISHED
  targetResourceName: organizations/{org}/environments/{env}/archiveDeployments/{archive}
name: organizations/{org}/operations/5bb2ab24-e4ba-4fb9-ab7e-d620a0672557
organization: {org}
uuid: 5bb2ab24-e4ba-4fb9-ab7e-d620a0672557

Resolution

To resolve this error, you can either:

1. Create a truststore within the specified keystore with the same name that is referenced in the targetservers.json file
2. Update the targetservers.json file to reference an existing, valid Apigee keystore and truststore combination

Invalid certificate for alias or truststore

Description

In this scenario, the specified archive deployment references an invalid or expired certificate for the specified alias or truststore in its targetservers.json file:

Example

gcloud alpha apigee operations describe b64c2665-b5ac-43cc-9e2d-232e8895c2ed

done: true
error:
  code: 3
  message: 'generic::invalid_argument: proxy "organizations/{org}/apis/{api}/revisions/1"
    contains reference(s) to invalid target server(s): target server "mtlstarget-docs"
    contains invalid TLS settings: truststore "organizations/{org}/environments/{env}/keystores/expired-keystore"
    contains an invalid certificate within alias "organizations/{org}/environments/{env}/keystores/expired-keystore/aliases/expired":
    certificate has expired'
metadata:
  '@type': type.googleapis.com/google.cloud.apigee.v1.OperationMetadata
  operationType: INSERT
  state: FINISHED
  targetResourceName: organizations/{org}/environments/{env}/archiveDeployments/{archive}
name: organizations/{org}/operations/3e28f818-8e1f-4afc-aedd-4aed29cc5519
organization: {org}
uuid: 3e28f818-8e1f-4afc-aedd-4aed29cc5519

Resolution

To resolve this error, you can either:

1. Update the specified alias with a new and valid certificate
2. Update the targetservers.json file to reference an existing, valid Apigee keystore and alias combination

Runtime Traffic Errors

This section describes error cases you may encounter at runtime.

Expired or Invalid Certificate(s)

Description

If a certificate is expired or invalid, you may see a 503 error when making a request to your configurable proxy:

Example

$ curl https://myorg-dev.apigee.net/helloworld

upstream connect error or disconnect/reset before headers. reset reason: connection failure,
transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Resolution

To determine the cause of this error, ensure that:

• The backend certificates and keys are valid for the proxy certificates and keys present in the referenced keystore.
• The backend certificates are not expired. To confirm:
  1. Open Apigee UI in a browser.
  2. Select Admin > Environments > TLS Keystores and select your keystore.
  3. Check the Expiration Status of your certificates.
• The referenced keystore was not updated in the middle of an archive deployment.

To update an expired certificate, follow the steps instructions for Update a certificate for your proxy.