This page shows you how to set up your Google Cloud project, create a Maven project with the Apache Beam SDK, and run an example pipeline on the Dataflow service.
Before you begin
-
Sign in to your Google Account.
If you don't already have one, sign up for a new account.
-
In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.
- Enable the Cloud Dataflow, Compute Engine, Stackdriver Logging, Cloud Storage, Cloud Storage JSON, BigQuery, Cloud Pub/Sub, Cloud Datastore, and Cloud Resource Manager APIs.
-
Set up authentication:
-
In the Cloud Console, go to the Create service account key page.
Go to the Create Service Account Key page - From the Service account list, select New service account.
- In the Service account name field, enter a name.
From the Role list, select Project > Owner.
- Click Create. A JSON file that contains your key downloads to your computer.
-
-
Set the environment variable
GOOGLE_APPLICATION_CREDENTIALS
to the path of the JSON file that contains your service account key. This variable only applies to your current shell session, so if you open a new session, set the variable again. - Create a Cloud Storage bucket:
- In the Cloud Console, go to the Cloud Storage Browser page.
- Click Create bucket.
- In the Create bucket dialog, specify the following attributes:
- Name: A unique bucket name. Do not include sensitive information in the bucket name, because the bucket namespace is global and publicly visible.
- Default storage class: Standard
- A location where bucket data will be stored.
- Click Create.
- Download and install the Java Development Kit (JDK) version 11 (Note: Dataflow continues to support version 8.) Verify that the JAVA_HOME environment variable is set and points to your JDK installation.
- Download and install Apache Maven by following Maven's installation guide for your specific operating system.
Get the WordCount code
The Apache Beam SDK is an open source programming model for data pipelines. You define these pipelines with an Apache Beam program and can choose a runner, such as Dataflow, to execute your pipeline.
Create a Maven project containing the Apache Beam SDK's WordCount examples, using the Maven Archetype
Plugin. From a directory on your computer, run the mvn archetype:generate
command in your shell or terminal as
follows:
$ mvn archetype:generate \ -DarchetypeGroupId=org.apache.beam \ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ -DarchetypeVersion=2.27.0 \ -DgroupId=org.example \ -DartifactId=word-count-beam \ -Dversion="0.1" \ -Dpackage=org.apache.beam.examples \ -DinteractiveMode=false
After running the command, you should see a new directory called
word-count-beam
under your current directory. word-count-beam
contains
a simple pom.xml
file and a series of example pipelines that count words in text
files.
$ cd word-count-beam/ $ ls pom.xml src $ ls src/main/java/org/apache/beam/examples/ DebuggingWordCount.java WindowedWordCount.java common MinimalWordCount.java WordCount.java
For a detailed introduction to the Apache Beam concepts used in these examples, see the
WordCount Example Walkthrough.
The following example executes
WordCount.java
.
Run WordCount locally
Run WordCount locally by running the following command from your word-count-beam
directory:
$ mvn compile exec:java \ -Dexec.mainClass=org.apache.beam.examples.WordCount \ -Dexec.args="--output=counts"The output files have the prefix
counts
and are written to the word-count-beam
directory.
They contain unique words from the input text and the number of occurences of each word.
Run WordCount on the Dataflow service
Build and run WordCount on the Dataflow service:
- For the
--project
argument, specify the Project ID for the Google Cloud project you created. For the
--stagingLocation
and--output
arguments, specify the name of the Cloud Storage bucket you created as part of the path.- For the
--region
argument, specify a Dataflow regional endpoint.
$ mvn -Pdataflow-runner compile exec:java \ -Dexec.mainClass=org.apache.beam.examples.WordCount \ -Dexec.args="--project=<PROJECT_ID> \ --stagingLocation=gs://<STORAGE_BUCKET>/staging/ \ --output=gs://<STORAGE_BUCKET>/output \ --runner=DataflowRunner \ --region=<REGION>"
View your results
- Open the Dataflow Web UI.
Go to the Dataflow Web UIYou should see your wordcount job with a status of Running at first, and then Succeeded:
- Open the Cloud Storage Browser in the Google Cloud Console.
Go to the Cloud Storage browserIn your bucket, you should see the output files and staging files that your job created:
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this quickstart, follow these steps.
- In the Cloud Console, go to the Cloud Storage Browser page.
- Click the checkbox for the bucket you want to delete.
- To delete the bucket, click Delete delete.