Streaming FHIR resource changes to BigQuery

This page explains how to configure the Cloud Healthcare API to export resource changes to BigQuery each time a FHIR resource is created, updated, patched, or deleted in a FHIR store.

Setting BigQuery permissions

To stream FHIR resource changes to BigQuery, you must grant additional permissions to the Cloud Healthcare Service Agent service account. For more information, see FHIR store BigQuery permissions.

Streaming FHIR resource changes

The following samples show how to stream FHIR resource changes to a BigQuery dataset.

For more information, see the streamConfigs field in the FHIR store resource.

curl

To enable BigQuery streaming, you must update the FHIR store's streamConfigs field. To update the FHIR store, make a PATCH request with the following information:

  • The parent dataset
  • The FHIR store
  • The BigQuery dataset
  • The BigQuery project
  • An update mask
  • An access token

The following sample shows a PATCH request using curl.

curl -X PATCH \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    --data "{
      'streamConfigs': [
        {
          'bigqueryDestination': {
            'datasetUri': 'bq://BIGQUERY_PROJECT_ID.BIGQUERY_DATASET_ID',
            'schemaConfig': {
              'schemaType': 'ANALYTICS'
            }
          }
        }
      ]
    }" "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/fhirStores/FHIR_STORE_ID?updateMask=streamConfigs"

If the request is successful, the server returns the response in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/fhirStores/FHIR_STORE_ID",
  'streamConfigs': [
   {
     'bigqueryDestination': {
       'datasetUri': 'bq://BIGQUERY_PROJECT_ID.BIGQUERY_DATASET_ID',
       'schemaConfig': {
         'schemaType': 'ANALYTICS'
       }
     }
   }
  ]
}

After you configure BigQuery streaming, the Cloud Healthcare API inserts a new row into a BigQuery table each time a FHIR resource is created, updated, patched, or deleted.

PowerShell

To enable BigQuery streaming, you must update the FHIR store's streamConfigs field. To update the FHIR store, make a PATCH request with the following information:

  • The parent dataset
  • The FHIR store
  • The BigQuery dataset
  • The BigQuery project
  • An update mask
  • An access token

The following sample shows a PATCH request using Windows PowerShell.

$cred = gcloud auth application-default print-access-token
$headers = @{ Authorization = "Bearer $cred" }

Invoke-WebRequest `
  -Method Patch `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body "{
      'streamConfigs': [
        {
          'bigqueryDestination': {
            'datasetUri': 'bq://BIGQUERY_PROJECT_ID.BIGQUERY_DATASET_ID',
            'schemaConfig': {
              'schemaType': 'ANALYTICS'
            }
          }
        }
      ]
  }" `
  -Uri "https://healthcare.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/fhirStores/FHIR_STORE_ID?updateMask=streamConfigs" | Select-Object -Expand Content

If the request is successful, the server returns the response in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/fhirStores/FHIR_STORE_ID",
  'streamConfigs': [
   {
     'bigqueryDestination': {
       'datasetUri': 'bq://BIGQUERY_PROJECT_ID.BIGQUERY_DATASET_ID',
       'schemaConfig': {
         'schemaType': 'ANALYTICS'
       }
     }
   }
  ]
}

After you configure BigQuery streaming, the Cloud Healthcare API inserts a new row into a BigQuery table each time a FHIR resource is created, updated, patched, or deleted.

Troubleshooting FHIR streaming

If errors occur when resource changes are sent to BigQuery, the errors are logged to Cloud Logging. For more information, see Viewing error logs in Cloud Logging.