Read with projection and filtering
Stay organized with collections
Save and categorize content based on your preferences.
Use the BigQueryIO connector in DIRECT_READ mode with column projection and filtering.
Explore further
For detailed documentation that includes this code sample, see the following:
Code sample
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eThis code sample demonstrates how to use the BigQueryIO connector in DIRECT_READ mode.\u003c/p\u003e\n"],["\u003cp\u003eColumn projection and filtering are implemented within the BigQuery read operation.\u003c/p\u003e\n"],["\u003cp\u003eThe code reads data from a specified BigQuery table, filters rows based on the condition "age > 18", and selects only the "user_name" and "age" fields.\u003c/p\u003e\n"],["\u003cp\u003eAuthentication for Dataflow is required, and Application Default Credentials (ADC) should be set up for local development.\u003c/p\u003e\n"],["\u003cp\u003eThe provided code is in Java, and is ran on Dataflow, to process the results of the table operation.\u003c/p\u003e\n"]]],[],null,["# Read with projection and filtering\n\nUse the BigQueryIO connector in DIRECT_READ mode with column projection and filtering.\n\nExplore further\n---------------\n\n\nFor detailed documentation that includes this code sample, see the following:\n\n- [Read from BigQuery to Dataflow](/dataflow/docs/guides/read-from-bigquery)\n\nCode sample\n-----------\n\n### Java\n\n\nTo authenticate to Dataflow, set up Application Default Credentials.\nFor more information, see\n\n[Set up authentication for a local development environment](/docs/authentication/set-up-adc-local-dev-environment).\n\n import com.google.common.collect.ImmutableMap;\n import java.util.List;\n import org.apache.beam.sdk.Pipeline;\n import org.apache.beam.sdk.managed.Managed;\n import org.apache.beam.sdk.options.PipelineOptionsFactory;\n import org.apache.beam.sdk.transforms.MapElements;\n import org.apache.beam.sdk.values.Row;\n import org.apache.beam.sdk.values.TypeDescriptors;\n\n public class BigQueryReadWithProjectionAndFiltering {\n public static void main(String[] args) {\n // Parse the pipeline options passed into the application. Example:\n // --projectId=$PROJECT_ID --datasetName=$DATASET_NAME --tableName=$TABLE_NAME\n // For more information, see https://beam.apache.org/documentation/programming-guide/#configuring-pipeline-options\n PipelineOptionsFactory.register(ExamplePipelineOptions.class);\n ExamplePipelineOptions options = PipelineOptionsFactory.fromArgs(args)\n .withValidation()\n .as(ExamplePipelineOptions.class);\n\n String tableSpec = String.format(\"%s:%s.%s\",\n options.getProjectId(),\n options.getDatasetName(),\n options.getTableName());\n\n ImmutableMap\u003cString, Object\u003e config = ImmutableMap.\u003cString, Object\u003ebuilder()\n .put(\"table\", tableSpec)\n .put(\"row_restriction\", \"age \u003e 18\")\n .put(\"fields\", List.of(\"user_name\", \"age\"))\n .build();\n\n // Create a pipeline and apply transforms.\n Pipeline pipeline = Pipeline.create(options);\n pipeline\n .apply(Managed.read(Managed.BIGQUERY).withConfig(config)).getSinglePCollection()\n .apply(MapElements\n .into(TypeDescriptors.strings())\n // Access individual fields in the row.\n .via((Row row) -\u003e {\n String output = String.format(\"Name: %s, Age: %s%n\",\n row.getString(\"user_name\"),\n row.getInt64(\"age\"));\n System.out.println(output);\n return output;\n }));\n pipeline.run().waitUntilFinish();\n }\n }\n\nWhat's next\n-----------\n\n\nTo search and filter code samples for other Google Cloud products, see the\n[Google Cloud sample browser](/docs/samples?product=dataflow)."]]