JMS to Pub/Sub template

The JMS to Pub/Sub template is a streaming pipeline that reads messages from Active MQ JMS Server (Queue/Topic) and writes them to Pub/Sub.

Pipeline Requirements

  • The Pub/Sub output topic name must exist.
  • The JMS host IP must exist and have the proper network configuration for Dataflow worker VMs to reach the JMS host.
  • The JMS topic/queue that data is extracted from must have a name.

Template parameters

Required parameters

  • inputName : The name of the JMS topic or queue that data is read from. (Example: queue).
  • inputType : The JMS destination type to read data from. Can be a queue or a topic. (Example: queue).
  • outputTopic : The name of the Pub/Sub topic to publish data to, in the format projects/<PROJECT_ID>/topics/<TOPIC_NAME>. (Example: projects/your-project-id/topics/your-topic-name).
  • username : The username to use for authentication on the JMS server. (Example: sampleusername).
  • password : The password associated with the provided username. (Example: samplepassword).

Optional parameters

  • jmsServer : The JMS (ActiveMQ) Server IP. (Example: tcp://10.0.0.1:61616).

Run the template

Console

  1. Go to the Dataflow Create job from template page.
  2. Go to Create job from template
  3. In the Job name field, enter a unique job name.
  4. Optional: For Regional endpoint, select a value from the drop-down menu. The default region is us-central1.

    For a list of regions where you can run a Dataflow job, see Dataflow locations.

  5. From the Dataflow template drop-down menu, select JMS to Pub/Sub template.
  6. In the provided parameter fields, enter your parameter values.
  7. Click Run job.

gcloud

In your shell or terminal, run the template:

gcloud dataflow flex-template run JOB_NAME \
    --project=YOUR_PROJECT_ID \
    --region=REGION_NAME \
    --template-file-gcs-location=gs://dataflow-templates-REGION_NAME/VERSION/flex/JMS_to_Cloud_PubSub \
    --parameters \
jmsServer=JMS_SERVER,\
inputName=INPUT_NAME,\
inputType=INPUT_TYPE,\
outputTopic=OUTPUT_TOPIC,\
username=USERNAME,\
password=PASSWORD
  

You must replace the following values in this example:

  • Replace YOUR_PROJECT_ID with your project ID.
  • Replace with Dataflow region name. For example: us-central1.
  • Replace JOB_NAME with a job name of your choice. The job name must match the regular expression [a-z]([-a-z0-9]{0,38}[a-z0-9])? to be valid.
  • Replace JMS_SERVER with the JMS server addresses. For example: tcp://10.0.0.0:61616
  • Replace INPUT_NAME with the name of the JMS server input topic/queue. For example: testtopic.
  • Replace INPUT_TYPE with the JMS server Destination Type(queue/topic). For example: topic
  • Replace OUTPUT_TOPIC with the name of Pub/Sub output topic. For example: projects/myproject/topics/testoutput.
  • Replace USERNAME with the username for the JMS server. For example: testuser.
  • Replace PASSWORD with the password that corresponds to the username used with the JMS server.

API

To run the template using the REST API, send an HTTP POST request. For more information on the API and its authorization scopes, see projects.templates.launch.

POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/flexTemplates:launch
{
   "launch_parameter": {
      "jobName": "JOB_NAME",
      "parameters": {
          "jmsServer": "JMS_SERVER",
          "inputName": "INPUT_NAME",
          "inputType": "INPUT_TYPE",
          "outputTopic": "OUTPUT_TOPIC",
          "username": "USERNAME",
          "password": "PASSWORD"
      },
      "containerSpecGcsPath": "gs://dataflow-templates-REGION_NAME/VERSION/flex/JMS_to_Cloud_PubSub",
   }
}
  

You must replace the following values in this example:

  • Replace YOUR_PROJECT_ID with your project ID.
  • Replace with Dataflow region name. For example: us-central1.
  • Replace JOB_NAME with a job name of your choice. The job name must match the regular expression [a-z]([-a-z0-9]{0,38}[a-z0-9])? to be valid.
  • Replace JMS_SERVER with the JMS server addresses. For example: tcp://10.0.0.0:61616
  • Replace INPUT_NAME with the name of the JMS server input topic/queue. For example: testtopic.
  • Replace INPUT_TYPE with the JMS server Destination Type(queue/topic). For example: topic
  • Replace OUTPUT_TOPIC with the name of Pub/Sub output topic. For example: projects/myproject/topics/testoutput.
  • Replace USERNAME with the username for the JMS server. For example: testuser.
  • Replace PASSWORD with the password that corresponds to the username used with the JMS server.

What's next