Create an Apache Kafka connection

You're viewing Apigee X documentation.
View Apigee Edge documentation.

The Apache Kafka connection lets you write (publish) to topics in Apache Kafka. This page provides a reference for creating a Apache Kafka connection and using it with Apigee Integration.

Before you begin

Before creating a Apache Kafka connection, perform the following tasks:

  • In your Google Cloud project:
    • Grant the following roles to the service account that you want to use to create the connection:
      • roles/secretmanager.viewer
      • roles/secretmanager.secretAccessor
    • Enable the following services:
      • apigee.googleapis.com (Apigee API)
      • secretmanager.googleapis.com (Secret Manager API)
      • connectors.googleapis.com (Connector Platform API)

    If these services or permissions have not been enabled for your project previously, you are prompted to enable them when creating the connection in the Create Connection page.

  • Create topics in Apache Kafka for which you want to write to. With the Apache Kafka connection, you can write only key-value messages to topics and the supported message formats are AVRO, CSV, JSON, and XML.

Create a connection

  1. In the Cloud console, go to the Apigee > Connectors page and then select or create a Google Cloud project.

    Go to the Connections page

  2. Click + CREATE NEW to open the Create Connection page.
  3. Configure the connection:
    1. In the Create Connection section, complete the following:
      • Connector: Select Apache Kafka from the drop down list of available Connectors.
      • Connector version: Select the Connector version from the drop down list of available versions.
      • In the Connection Name field, enter a name for the Connection instance.

        Connection names must meet the following criteria:

        • Connection names can use letters, numbers, or hyphens.
        • Letters must be lower-case.
        • Connection names must begin with a letter and end with a letter or number.
        • Connection names cannot exceed 63 characters.
      • Optionally, enter a Description for the connection instance.
      • Service Account: Select a service account that has the required roles.
      • BootStrapServers: Apache Kafka bootstrap server address. You can specify multiple servers as comma separated addresses.
      • Topic: Enter the name of the Kafka topic you want to write to.
      • Serialization Format: Enter the serialization format of messages to be published. The Apache Kafka connection supports AVRO, CSV, JSON, and XML message formats.
      • Optionally, click + ADD LABEL to add a label to the Connection in the form of a key/value pair.
      • Click NEXT.
    2. Location: Choose the location for the connection.
      • Select a location where Apigee is available from the Region drop-down list.

        Supported regions for connectors include:

        • asia-east1
        • asia-northeast1
        • asia-south1
        • asia-southeast1
        • australia-southeast1
        • europe-west1
        • europe-west2
        • europe-west3
        • europe-west4
        • europe-west6
        • northamerica-northeast1
        • southamerica-east1
        • us-central1
        • us-east1
        • us-east4
        • us-west1
        • us-west2
      • Click NEXT.
    3. Authentication: Provide authentication credentials for your connection instance, if required.

      The Apache Kafka connection supports both anonymous login and credentials based login in the form of a User and Password combination.

      If you want to use anonymous login, select Not Available, and then click NEXT.

      If you want to authenticate using a User and Password combination, select User Password, and enter the following information:

      • Username: Enter the Apache Kafka username for the connection.
      • Password: Enter the Secret Manager secret containing the password associated with the Apache Kafka username.
        • If you have previously created a secret, and it is not available in the drop down list, select DON'T SEE YOUR SECRET? ENTER SECRET RESOURCE ID. In the Add a secret by resource ID dialog, copy and paste the resource ID from the Secret Manager.
          • To use the latest version, copy and paste the resource ID from the parent secret, in the format: "projects/project-number/secrets/secret-name"
          • To select a specific version, copy and paste the resource ID for that specific version, in the format "projects/project-number/secrets/secret-name/versions/1"

          Click ADD SECRET to add the secret and close the dialog.

        • If you have not created a secret for use with Apache Kafka, click ADD A NEW SECRET. In the Create Secret dialog enter the following details:
          • Name: Enter the secret name.
          • Secret value: Enter the contents of the secret.
          • Optionally, click + ADD LABEL to add a label in the form of a key/value pair.
      • Secret version: Select the version of the Password secret from the list of available versions in the drop-down.
      • Click NEXT.
    4. Review: Review your connection and authentication details.
  4. Click Create.

Create a Connectors task

After you create an Apache Kafka connection, it becomes available in Apigee integrations. You can then use the connection to configure the Connectors tasks in your integrations. For more information, see Configure the Connectors task.