BigQueryConnectionConnection

Property Value
Google Cloud Service Name BigQuery Connection
Google Cloud Service Documentation /bigquery/docs/
Google Cloud REST Resource Name v1beta1.projects.locations.connections
Google Cloud REST Resource Documentation bigqueryconnection/rest/v1/projects.locations.connections
Config Connector Resource Short Names gcpbigqueryconnectionconnection
gcpbigqueryconnectionconnections
bigqueryconnectionconnection
Config Connector Service Name bigqueryconnection.googleapis.com
Config Connector Resource Fully Qualified Name bigqueryconnectionconnections.bigqueryconnection.cnrm.cloud.google.com
Can Be Referenced by IAMPolicy/IAMPolicyMember No
Config Connector Default Average Reconcile Interval In Seconds 600

Custom Resource Definition Properties

Spec

Schema

aws:
  accessRole:
    iamRoleID: string
azure:
  customerTenantID: string
  federatedApplicationClientID: string
cloudResource: {}
cloudSQL:
  credential:
    secretRef:
      name: string
      namespace: string
  databaseRef:
    external: string
    name: string
    namespace: string
  instanceRef:
    external: string
    name: string
    namespace: string
  type: string
cloudSpanner:
  databaseRef:
    external: string
    name: string
    namespace: string
  databaseRole: string
  maxParallelism: integer
  useDataBoost: boolean
  useParallelism: boolean
  useServerlessAnalytics: boolean
description: string
friendlyName: string
location: string
projectRef:
  external: string
  kind: string
  name: string
  namespace: string
resourceID: string
spark:
  metastoreService:
    metastoreServiceRef:
      external: string
  sparkHistoryServer:
    dataprocClusterRef:
      external: string
      name: string
      namespace: string
Fields

aws

Optional

object

Amazon Web Services (AWS) properties.

aws.accessRole

Required*

object

Authentication using Google owned service account to assume into customer's AWS IAM Role.

aws.accessRole.iamRoleID

Required*

string

The user’s AWS IAM Role that trusts the Google-owned AWS IAM user Connection.

azure

Optional

object

Azure properties.

azure.customerTenantID

Required*

string

The id of customer's directory that host the data.

azure.federatedApplicationClientID

Optional

string

The client ID of the user's Azure Active Directory Application used for a federated connection.

cloudResource

Optional

object

Use Cloud Resource properties.

cloudSQL

Optional

object

Cloud SQL properties.

cloudSQL.credential

Required*

object

Cloud SQL credential.

cloudSQL.credential.secretRef

Optional

object

The Kubernetes Secret object that stores the "username" and "password" information. The Secret type has to be `kubernetes.io/basic-auth`.

cloudSQL.credential.secretRef.name

Required*

string

The `metadata.name` field of a Kubernetes `Secret`

cloudSQL.credential.secretRef.namespace

Optional

string

The `metadata.namespace` field of a Kubernetes `Secret`.

cloudSQL.databaseRef

Required*

object

Reference to the SQL Database.

cloudSQL.databaseRef.external

Optional

string

The SQL Database name, when not managed by Config Connector.

cloudSQL.databaseRef.name

Optional

string

The `name` field of a `SQLDatabase` resource.

cloudSQL.databaseRef.namespace

Optional

string

The `namespace` field of a `SQLDatabase` resource.

cloudSQL.instanceRef

Required*

object

Reference to the Cloud SQL instance ID.

cloudSQL.instanceRef.external

Optional

string

The SQLInstance selfLink, when not managed by Config Connector.

cloudSQL.instanceRef.name

Optional

string

The `name` field of a `SQLInstance` resource.

cloudSQL.instanceRef.namespace

Optional

string

The `namespace` field of a `SQLInstance` resource.

cloudSQL.type

Required*

string

Type of the Cloud SQL database.

cloudSpanner

Optional

object

Cloud Spanner properties.

cloudSpanner.databaseRef

Required*

object

Reference to a spanner database ID.

cloudSpanner.databaseRef.external

Optional

string

The Spanner Database selfLink, when not managed by Config Connector.

cloudSpanner.databaseRef.name

Optional

string

The `name` field of a `SpannerDatabase` resource.

cloudSpanner.databaseRef.namespace

Optional

string

The `namespace` field of a `SpannerDatabase` resource.

cloudSpanner.databaseRole

Optional

string

Optional. Cloud Spanner database role for fine-grained access control. The Cloud Spanner admin should have provisioned the database role with appropriate permissions, such as `SELECT` and `INSERT`. Other users should only use roles provided by their Cloud Spanner admins. For more details, see [About fine-grained access control] (https://cloud.google.com/spanner/docs/fgac-about). REQUIRES: The database role name must start with a letter, and can only contain letters, numbers, and underscores.

cloudSpanner.maxParallelism

Optional

integer

Allows setting max parallelism per query when executing on Spanner independent compute resources. If unspecified, default values of parallelism are chosen that are dependent on the Cloud Spanner instance configuration. REQUIRES: `use_parallelism` must be set. REQUIRES: Either `use_data_boost` or `use_serverless_analytics` must be set.

cloudSpanner.useDataBoost

Optional

boolean

If set, the request will be executed via Spanner independent compute resources. REQUIRES: `use_parallelism` must be set. NOTE: `use_serverless_analytics` will be deprecated. Prefer `use_data_boost` over `use_serverless_analytics`.

cloudSpanner.useParallelism

Optional

boolean

If parallelism should be used when reading from Cloud Spanner

cloudSpanner.useServerlessAnalytics

Optional

boolean

If the serverless analytics service should be used to read data from Cloud Spanner. Note: `use_parallelism` must be set when using serverless analytics.

description

Optional

string

User provided description.

friendlyName

Optional

string

User provided display name for the connection.

location

Required*

string

Immutable.

projectRef

Required*

object

The Project that this resource belongs to.

projectRef.external

Optional

string

The `projectID` field of a project, when not managed by Config Connector.

projectRef.kind

Optional

string

The kind of the Project resource; optional but must be `Project` if provided.

projectRef.name

Optional

string

The `name` field of a `Project` resource.

projectRef.namespace

Optional

string

The `namespace` field of a `Project` resource.

resourceID

Optional

string

Immutable. Optional. The BigQuery Connection ID used for resource creation or acquisition. For creation: If specified, this value is used as the connection ID. If not provided, a UUID will be generated and assigned as the connection ID. For acquisition: This field must be provided to identify the connection resource to acquire.

spark

Optional

object

Spark properties.

spark.metastoreService

Optional

object

Optional. Dataproc Metastore Service configuration for the connection.

spark.metastoreService.metastoreServiceRef

Optional

object

Optional. Resource name of an existing Dataproc Metastore service. Example: * `projects/[project_id]/locations/[region]/services/[service_id]`

spark.metastoreService.metastoreServiceRef.external

Required*

string

The self-link of an existing Dataproc Metastore service , when not managed by Config Connector.

spark.sparkHistoryServer

Optional

object

Optional. Spark History Server configuration for the connection.

spark.sparkHistoryServer.dataprocClusterRef

Optional

object

Optional. Resource name of an existing Dataproc Cluster to act as a Spark History Server for the connection. Example: * `projects/[project_id]/regions/[region]/clusters/[cluster_name]`

spark.sparkHistoryServer.dataprocClusterRef.external

Optional

string

The self-link of an existing Dataproc Cluster to act as a Spark History Server for the connection , when not managed by Config Connector.

spark.sparkHistoryServer.dataprocClusterRef.name

Optional

string

The `name` field of a Dataproc Cluster.

spark.sparkHistoryServer.dataprocClusterRef.namespace

Optional

string

The `namespace` field of a Dataproc Cluster.

* Field is required when parent field is specified

Status

Schema

conditions:
- lastTransitionTime: string
  message: string
  reason: string
  status: string
  type: string
externalRef: string
observedGeneration: integer
observedState:
  aws:
    accessRole:
      identity: string
  azure:
    application: string
    clientID: string
    identity: string
    objectID: string
    redirectUri: string
  cloudResource:
    serviceAccountID: string
  cloudSQL:
    serviceAccountID: string
  description: string
  friendlyName: string
  hasCredential: boolean
  spark:
    serviceAccountID: string
Fields
conditions

list (object)

Conditions represent the latest available observations of the object's current state.
conditions[]

object
conditions[].lastTransitionTime

string

Last time the condition transitioned from one status to another.
conditions[].message

string

Human-readable message indicating details about last transition.
conditions[].reason

string

Unique, one-word, CamelCase reason for the condition's last transition.
conditions[].status

string

Status is the status of the condition. Can be True, False, Unknown.
conditions[].type

string

Type is the type of the condition.
externalRef

string

A unique specifier for the BigQueryConnectionConnection resource in GCP.
observedGeneration

integer

ObservedGeneration is the generation of the resource that was most recently observed by the Config Connector controller. If this is equal to metadata.generation, then that means that the current reported status reflects the most recent desired state of the resource.
observedState

object

ObservedState is the state of the resource as most recently observed in GCP.
observedState.aws

object
observedState.aws.accessRole

object
observedState.aws.accessRole.identity

string

A unique Google-owned and Google-generated identity for the Connection. This identity will be used to access the user's AWS IAM Role.
observedState.azure

object
observedState.azure.application

string

The name of the Azure Active Directory Application.
observedState.azure.clientID

string

The client id of the Azure Active Directory Application.
observedState.azure.identity

string

A unique Google-owned and Google-generated identity for the Connection. This identity will be used to access the user's Azure Active Directory Application.
observedState.azure.objectID

string

The object id of the Azure Active Directory Application.
observedState.azure.redirectUri

string

The URL user will be redirected to after granting consent during connection setup.
observedState.cloudResource

object
observedState.cloudResource.serviceAccountID

string

The account ID of the service created for the purpose of this connection. The service account does not have any permissions associated with it when it is created. After creation, customers delegate permissions to the service account. When the connection is used in the context of an operation in BigQuery, the service account will be used to connect to the desired resources in GCP. The account ID is in the form of: @gcp-sa-bigquery-cloudresource.iam.gserviceaccount.com
observedState.cloudSQL

object
observedState.cloudSQL.serviceAccountID

string

The account ID of the service used for the purpose of this connection. When the connection is used in the context of an operation in BigQuery, this service account will serve as the identity being used for connecting to the CloudSQL instance specified in this connection.
observedState.description

string

The description for the connection.
observedState.friendlyName

string

The display name for the connection.
observedState.hasCredential

boolean

Output only. True, if credential is configured for this connection.
observedState.spark

object
observedState.spark.serviceAccountID

string

The account ID of the service created for the purpose of this connection. The service account does not have any permissions associated with it when it is created. After creation, customers delegate permissions to the service account. When the connection is used in the context of a stored procedure for Apache Spark in BigQuery, the service account is used to connect to the desired resources in Google Cloud. The account ID is in the form of: bqcx--@gcp-sa-bigquery-consp.iam.gserviceaccount.com

Sample YAML(s)

Aws Connection

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: bigqueryconnection.cnrm.cloud.google.com/v1beta1
kind: BigQueryConnectionConnection
metadata:
  name: bigqueryconnectionconnection-sample
spec:
  location: aws-us-east-1
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID
    external: ${PROJECT_ID?}
  aws:
    accessRole:
      iamRoleID: "arn:aws:iam::999999999999:role/omnirole"

Azure Connection

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: bigqueryconnection.cnrm.cloud.google.com/v1beta1
kind: BigQueryConnectionConnection
metadata:
  name: bigqueryconnectionconnection-sample
spec:
  location: "azure-eastus2"
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID
    external: ${PROJECT_ID?}
  azure:
    customerTenantID: "customer-tenant-id-1111111"
    federatedApplicationClientID: "b43eeeee-eeee-eeee-eeee-a480155501ce"

Cloudresource Connection

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: bigqueryconnection.cnrm.cloud.google.com/v1alpha1
kind: BigQueryConnectionConnection
metadata:
  name: bigqueryconnectionconnection-sample
spec:
  location: us-central1
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID
    external: ${PROJECT_ID?}
  cloudResource: {}

Cloudspanner Connection

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: bigqueryconnection.cnrm.cloud.google.com/v1beta1
kind: BigQueryConnectionConnection
metadata:
  name: bigqueryconnectionconnection-sample
spec:
  location: us-central1
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID
    external: ${PROJECT_ID?}
  cloudSpanner:
    databaseRef:
      name: bigqueryconnection-dep
    maxParallelism: 100
    useDataBoost: True
    useParallelism: True
    databaseRole: "admin"
---
apiVersion: spanner.cnrm.cloud.google.com/v1beta1
kind: SpannerDatabase
metadata:
  name: bigqueryconnection-dep
spec:
  instanceRef:
    name: bigqueryconnection-dep
  ddl:
  - "CREATE TABLE t1 (t1 INT64 NOT NULL,) PRIMARY KEY(t1)"
---
apiVersion: spanner.cnrm.cloud.google.com/v1beta1
kind: SpannerInstance
metadata:
  name: bigqueryconnection-dep
spec:
  config: regional-us-west1
  displayName: BigQueryConnection Dependency

Cloudsql Connection

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: bigqueryconnection.cnrm.cloud.google.com/v1beta1
kind: BigQueryConnectionConnection
metadata:
  name: bigqueryconnectionconnection-sample
spec:
  location: us-central1
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID
    external: ${PROJECT_ID?}
  cloudSQL:
    instanceRef:
      name: bigqueryconnection-dep
    databaseRef: 
      name: bigqueryconnection-dep
    type: "MYSQL"
    credential:
      secretRef: 
        name:  bigqueryconnection-dep
---
apiVersion: v1
kind: Secret
metadata:
  name: bigqueryconnection-dep
type: kubernetes.io/basic-auth
stringData:
  username: sqluser-${uniqueId}
  password: cGFzc3dvcmQ=
---
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLDatabase
metadata:
  name: bigqueryconnection-dep
spec:
  charset: utf8
  instanceRef:
    name: bigqueryconnection-dep
---
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
  name: bigqueryconnection-dep
spec:
  region: us-central1
  databaseVersion: MYSQL_5_7
  settings:
    locationPreference:
      zone: us-central1-a
    tier: db-custom-1-3840
---
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLUser
metadata:
  labels:
    label-one: "value-one"
  name: bigqueryconnection-dep
spec:
  instanceRef:
    name: bigqueryconnection-dep
  host: foo
  password:
    valueFrom:
      secretKeyRef:
        name: bigqueryconnection-dep
        key: password

Spark Connection

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: bigqueryconnection.cnrm.cloud.google.com/v1beta1
kind: BigQueryConnectionConnection
metadata:
  name: bigqueryconnectionconnection-sample
spec:
  location: us-central1
  projectRef:
    # Replace ${PROJECT_ID?} with your project ID
    external: ${PROJECT_ID?}
  spark: {}