The JMS to Pub/Sub template is a streaming pipeline that reads messages from Active MQ JMS Server (Queue/Topic) and writes them to Pub/Sub.
Pipeline Requirements
- The Pub/Sub output topic name must exist.
- The JMS host IP must exist and have the proper network configuration for Dataflow worker VMs to reach the JMS host.
- The JMS topic/queue that data is extracted from must have a name.
Template parameters
Required parameters
- inputName : The name of the JMS topic or queue that data is read from. (Example: queue).
- inputType : The JMS destination type to read data from. Can be a queue or a topic. (Example: queue).
- outputTopic : The name of the Pub/Sub topic to publish data to, in the format
projects/<PROJECT_ID>/topics/<TOPIC_NAME>
. (Example: projects/your-project-id/topics/your-topic-name). - username : The username to use for authentication on the JMS server. (Example: sampleusername).
- password : The password associated with the provided username. (Example: samplepassword).
Optional parameters
- jmsServer : The JMS (ActiveMQ) Server IP. (Example: tcp://10.0.0.1:61616).
Run the template
Console
- Go to the Dataflow Create job from template page. Go to Create job from template
- In the Job name field, enter a unique job name.
- Optional: For Regional endpoint, select a value from the drop-down menu. The default
region is
us-central1
.For a list of regions where you can run a Dataflow job, see Dataflow locations.
- From the Dataflow template drop-down menu, select JMS to Pub/Sub template.
- In the provided parameter fields, enter your parameter values.
- Click Run job.
gcloud
In your shell or terminal, run the template:
gcloud dataflow flex-template run JOB_NAME \ --project=YOUR_PROJECT_ID \ --region=REGION_NAME \ --template-file-gcs-location=gs://dataflow-templates-REGION_NAME/VERSION/flex/JMS_to_Cloud_PubSub \ --parameters \ jmsServer=JMS_SERVER,\ inputName=INPUT_NAME,\ inputType=INPUT_TYPE,\ outputTopic=OUTPUT_TOPIC,\ username=USERNAME,\ password=PASSWORD
You must replace the following values in this example:
- Replace YOUR_PROJECT_ID with your project ID.
- Replace with Dataflow region name. For example:
us-central1
. - Replace JOB_NAME with a job name of your choice. The job name must match the regular expression
[a-z]([-a-z0-9]{0,38}[a-z0-9])?
to be valid. - Replace JMS_SERVER with the JMS server addresses. For example:
tcp://10.0.0.0:61616
- Replace INPUT_NAME with the name of the JMS server input topic/queue. For example:
testtopic
. - Replace INPUT_TYPE with the JMS server Destination Type(queue/topic). For example:
topic
- Replace OUTPUT_TOPIC with the name of Pub/Sub output topic. For example:
projects/myproject/topics/testoutput
. - Replace USERNAME with the username for the JMS server. For example:
testuser
. - Replace PASSWORD with the password that corresponds to the username used with the JMS server.
API
To run the template using the REST API, send an HTTP POST request. For more information on the
API and its authorization scopes, see
projects.templates.launch
.
POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/flexTemplates:launch { "launch_parameter": { "jobName": "JOB_NAME", "parameters": { "jmsServer": "JMS_SERVER", "inputName": "INPUT_NAME", "inputType": "INPUT_TYPE", "outputTopic": "OUTPUT_TOPIC", "username": "USERNAME", "password": "PASSWORD" }, "containerSpecGcsPath": "gs://dataflow-templates-REGION_NAME/VERSION/flex/JMS_to_Cloud_PubSub", } }
You must replace the following values in this example:
- Replace YOUR_PROJECT_ID with your project ID.
- Replace with Dataflow region name. For example:
us-central1
. - Replace JOB_NAME with a job name of your choice. The job name must match the regular expression
[a-z]([-a-z0-9]{0,38}[a-z0-9])?
to be valid. - Replace JMS_SERVER with the JMS server addresses. For example:
tcp://10.0.0.0:61616
- Replace INPUT_NAME with the name of the JMS server input topic/queue. For example:
testtopic
. - Replace INPUT_TYPE with the JMS server Destination Type(queue/topic). For example:
topic
- Replace OUTPUT_TOPIC with the name of Pub/Sub output topic. For example:
projects/myproject/topics/testoutput
. - Replace USERNAME with the username for the JMS server. For example:
testuser
. - Replace PASSWORD with the password that corresponds to the username used with the JMS server.
What's next