이 샘플은 Dataflow 파이프라인으로 고객이 관리하는 암호화 키를 사용하는 방법을 보여줍니다.
코드 샘플
Java
Dataflow에 인증하려면 애플리케이션 기본 사용자 인증 정보를 설정합니다. 자세한 내용은 로컬 개발 환경의 인증 설정을 참조하세요.
// Query from the NASA wildfires public dataset:
// https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=nasa_wildfire&t=past_week&page=table
String query =
"SELECT latitude,longitude,acq_date,acq_time,bright_ti4,confidence "
+ "FROM `bigquery-public-data.nasa_wildfire.past_week` "
+ "LIMIT 10";
// Schema for the output BigQuery table.
final TableSchema outputSchema = new TableSchema().setFields(Arrays.asList(
new TableFieldSchema().setName("latitude").setType("FLOAT"),
new TableFieldSchema().setName("longitude").setType("FLOAT"),
new TableFieldSchema().setName("acq_date").setType("DATE"),
new TableFieldSchema().setName("acq_time").setType("TIME"),
new TableFieldSchema().setName("bright_ti4").setType("FLOAT"),
new TableFieldSchema().setName("confidence").setType("STRING")));
// Create the BigQuery options from the command line arguments.
BigQueryKmsKeyOptions options = PipelineOptionsFactory.fromArgs(args)
.withValidation().as(BigQueryKmsKeyOptions.class);
// String outputBigQueryTable = "<project>:<dataset>.<table>";
String outputBigQueryTable = options.getOutputBigQueryTable();
// String kmsKey =
// "projects/<project>/locations/<kms-location>/keyRings/<kms-keyring>/cryptoKeys/<kms-key>";
String kmsKey = options.getKmsKey();
// Create and run an Apache Beam pipeline.
Pipeline pipeline = Pipeline.create(options);
pipeline
.apply("Read from BigQuery with KMS key",
BigQueryIO.readTableRows()
.fromQuery(query)
.usingStandardSql()
.withKmsKey(kmsKey))
.apply("Write to BigQuery with KMS key",
BigQueryIO.writeTableRows()
.to(outputBigQueryTable)
.withSchema(outputSchema)
.withWriteDisposition(WriteDisposition.WRITE_TRUNCATE)
.withKmsKey(kmsKey));
pipeline.run().waitUntilFinish();
Python
Dataflow에 인증하려면 애플리케이션 기본 사용자 인증 정보를 설정합니다. 자세한 내용은 로컬 개발 환경의 인증 설정을 참조하세요.
import apache_beam as beam
# output_bigquery_table = '<project>:<dataset>.<table>'
# kms_key = 'projects/<project>/locations/<kms-location>/keyRings/<kms-keyring>/cryptoKeys/<kms-key>' # noqa
# beam_args = [
# '--project', 'your-project-id',
# '--runner', 'DataflowRunner',
# '--temp_location', 'gs://your-bucket/samples/dataflow/kms/tmp',
# '--region', 'us-central1',
# ]
# Query from the NASA wildfires public dataset:
# https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=nasa_wildfire&t=past_week&page=table
query = """
SELECT latitude,longitude,acq_date,acq_time,bright_ti4,confidence
FROM `bigquery-public-data.nasa_wildfire.past_week`
LIMIT 10
"""
# Schema for the output BigQuery table.
schema = {
"fields": [
{"name": "latitude", "type": "FLOAT"},
{"name": "longitude", "type": "FLOAT"},
{"name": "acq_date", "type": "DATE"},
{"name": "acq_time", "type": "TIME"},
{"name": "bright_ti4", "type": "FLOAT"},
{"name": "confidence", "type": "STRING"},
],
}
options = beam.options.pipeline_options.PipelineOptions(beam_args)
with beam.Pipeline(options=options) as pipeline:
(
pipeline
| "Read from BigQuery with KMS key"
>> beam.io.Read(
beam.io.BigQuerySource(
query=query,
use_standard_sql=True,
kms_key=kms_key,
)
)
| "Write to BigQuery with KMS key"
>> beam.io.WriteToBigQuery(
output_bigquery_table,
schema=schema,
write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,
kms_key=kms_key,
)
)
다음 단계
다른 Google Cloud 제품의 코드 샘플을 검색하고 필터링하려면 Google Cloud 샘플 브라우저를 참조하세요.