Importing and exporting HL7v2 messages using Cloud Storage

This page describes how to export and import HL7v2 messages to and from Cloud Storage using the projects.locations.datasets.hl7V2Stores.import and projects.locations.datasets.hl7V2Stores.export methods.

You can import HL7v2 messages from Cloud Storage to make it easier to bulk load many HL7v2 messages into an HL7v2 store. You can export many HL7v2 messages to Cloud Storage at once, rather than having to store them in Cloud Storage individually.

Setting Cloud Storage permissions

Before exporting and importing HL7v2 messages to and from Cloud Storage, you must grant additional permissions to the Cloud Healthcare Service Agent service account. For more information, see HL7v2 store Cloud Storage permissions.

Importing HL7v2 messages

To import HL7v2 messages, you must first create one or more newline-delimited JSON (.ndjson) files in Cloud Storage containing one or more messages. Each line of the file is a single Message resource which contains a base64-encoded HL7v2 message. The Message resource can also include optional labels.

For example, the following file, named messages.ndjson, contains two HL7v2 messages. A label is defined in the second message.

{"data" :"TVNIfF5+XCZ8QXxTRU5EX0ZBQ0lMSVRZXzF8QXxBfDIwMTgwMTAxMDAwMDAwfHxUWVBFXkF8MjAxODAxMDEwMDAwMDB8VHwwLjB8fHxBQXx8MDB8QVNDSUkNRVZOfEEwMHwyMDE4MDEwMTA0MDAwMA1QSUR8fDE0ATExMV5eXl5NUk58MTExMTExMTFeXl5eTVJOfjExMTExMTExMTFeXl5eT1JHTk1CUg=="}
{"data" :"TVNIfF5+XCZ8QXxTRU5EX0ZBQ0lMSVRZXzJ8QXxBfDIwMTgwMTAxMDAwMDAwfHxUWVBFXkF8MjAxODAxMDEwMDAwMDB8VHwwLjB8fHxBQXx8MDB8QVNDSUkNRVZOfEEwMHwyMDE4MDEwMTA0MDAwMA1QSUR8fDE0ATExMV5eXl5NUk58MTExMTExMTFeXl5eTVJOfjExMTExMTExMTFeXl5eT1JHTk1CUg==","labels":{"foo":"bar"}}

Console

To import HL7v2 messages from a Cloud Storage bucket, complete the following steps:

  1. In the Google Cloud console, go to the Datasets page.

    Go to Datasets

  2. Click the dataset that contains the HL7v2 store to which you are importing HL7v2 messages.

  3. In the list of data stores, choose Import from the Actions list for the HL7v2 store.

    The Import to HL7v2 store page appears.

  4. In the Project list, select a Cloud Storage project.

  5. In the Location list, select a Cloud Storage bucket.

  6. To set a specific location for importing files, do the following:

    1. Expand Advanced Options.
    2. Select Override Cloud Storage Path.
    3. To set a specific source for importing files, define the path in the Location text box. You can use wildcards to import multiple files from one or more directories. For more information on object naming, see Object naming guidelines.

      The following wildcards are supported:
      • Use * to match 0 or more non-separator characters. For example, gs://BUCKET/DIRECTORY/Example*.ndjson matches Example.ndjson and Example22.ndjson in DIRECTORY.
      • Use ** to match 0 or more characters (including separators). Must be used at the end of a path and with no other wildcards in the path. Can also be used with a filename extension (such as .ndjson), which imports all files with the filename extension in the specified directory and its subdirectories. For example, gs://BUCKET/DIRECTORY/**.ndjson imports all files with the .ndjson filename extension in DIRECTORY and its subdirectories.
      • Use ? to match 1 character. For example, gs://BUCKET/DIRECTORY/Example?.ndjson matches Example1.ndjson but does not match Example.ndjson or Example01.ndjson.
  7. Click Import to import HL7v2 messages from the defined source.

  8. To track the status of the operation, click the Operations tab. After the operation completes, the following indications appear:
    • The Long-running operation status section has a green check mark under the OK heading.
    • The Overview section has a green check mark and an OK indicator in the same row as the operation ID.
    If you encounter any errors, click Actions, and then click View details in Cloud Logging.

API

The following samples show how to import HL7v2 messages from Cloud Storage using the projects.locations.datasets.hl7V2Stores.import method.

Note the following when calling the import operation:

  • The location of the file within the bucket is arbitrary and does not have to adhere exactly to the format specified in the following samples.
  • When specifying the location of the HL7v2 messages in Cloud Storage, you can use wildcards to import multiple files from one or more directories. The following wildcards are supported:
    • Use * to match 0 or more non-separator characters. For example, gs://BUCKET/DIRECTORY/Example*.ndjson matches Example.ndjson and Example22.ndjson in DIRECTORY.
    • Use ** to match 0 or more characters (including separators). Must be used at the end of a path and with no other wildcards in the path. Can also be used with a filename extension (such as .ndjson), which imports all files with the filename extension in the specified directory and its subdirectories. For example, gs://BUCKET/DIRECTORY/**.ndjson imports all files with the .ndjson filename extension in DIRECTORY and its subdirectories.
    • Use ? to match 1 character. For example, gs://BUCKET/DIRECTORY/Example?.ndjson matches Example1.ndjson but does not match Example.ndjson or Example01.ndjson.

curl

To import HL7v2 messages into an HL7v2 store, make a POST request and specify the following information:

  • The name of the parent dataset
  • The name of the HL7v2 store
  • The location of the object in a Cloud Storage bucket
  • An access token

The following sample shows how to import a single file using a POST request using curl.

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    --data "{
      'gcsSource': {
        'uri': 'gs://BUCKET/DIRECTORY/HL7V2_MESSAGE_FILE'
      }
    }" "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/hl7V2Stores/HL7V2_STORE_ID:import"

If the request is successful, the server returns the response in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"
}

The response contains an operation name. To track the status of the operation, you can use the Operation get method:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"

If the request is successful, the server returns a response with the status of the operation in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.OperationMetadata",
    "apiMethodName": "google.cloud.healthcare.v1.hl7v2.Hl7V2Service.ImportMessages",
    "createTime": "CREATE_TIME",
    "endTime": "END_TIME"
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.hl7v2.ImportMessagesResponse"
  }
}

PowerShell

To import HL7v2 messages into an HL7v2 store, make a POST request and specify the following information:

  • The name of the parent dataset
  • The name of the HL7v2 store
  • The location of the object in a Cloud Storage bucket
  • An access token

The following sample shows how to import a single file using a POST request using Windows PowerShell.

$cred = gcloud auth application-default print-access-token
$headers = @{ Authorization = "Bearer $cred" }

Invoke-WebRequest `
  -Method Post `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body "{
    'gcsSource': {
      'uri': 'gs://BUCKET/DIRECTORY/HL7V2_MESSAGE_FILE'
    }
  }" `
  -Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/hl7V2Stores/HL7V2_STORE_ID:import" | Select-Object -Expand Content

If the request is successful, the server returns the response in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"
}

The response contains an operation name. To track the status of the operation, you can use the Operation get method:

$cred = gcloud auth application-default print-access-token
$headers = @{ Authorization = "Bearer $cred" }

Invoke-WebRequest `
  -Method Get `
  -Headers $headers `
  -Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID" | Select-Object -Expand Content

If the request is successful, the server returns a response with the status of the operation in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.OperationMetadata",
    "apiMethodName": "google.cloud.healthcare.v1.hl7v2.Hl7V2Service.ImportMessages",
    "createTime": "CREATE_TIME",
    "endTime": "END_TIME"
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.hl7v2.ImportMessagesResponse"
  }
}

Exporting HL7v2 messages

When you export HL7v2 messages from an HL7v2 store, all messages in the HL7v2 store are exported. To export only a subset of these messages, you can define startTime and endTime parameters to include only messages sent within the defined time span. To export only specific parts of the Message resource, set the MessageView.

When messages are exported, the Cloud Healthcare API creates an object for each HL7v2 message. Each object consists of an .ndjson file containing a Message resource on each line. Messages are exported in ascending order based on the sendTime (MSH.7).

Console

To export HL7v2 messages to Cloud Storage, complete the following steps:

  1. In the Google Cloud console, go to the Datasets page.

    Go to Datasets

  2. Click the dataset that contains the HL7v2 store from which you are exporting HL7v2 messages.

  3. In the list of data stores, choose Export from the Actions list for the HL7v2 store.

    The Export HL7v2 messages page appears.

  4. In the Project list, select a Cloud Storage project.

  5. In the Location list, select a Cloud Storage bucket.

  6. Click Export to export HL7v2 instances to the defined location in Cloud Storage.

  7. To track the status of the operation, click the Operations tab. After the operation completes, the following indications appear:
    • The Long-running operation status section has a green check mark under the OK heading.
    • The Overview section has a green check mark and an OK indicator in the same row as the operation ID.
    If you encounter any errors, click Actions, and then click View details in Cloud Logging.

API

The following samples show how to export HL7v2 messages to Cloud Storage using the projects.locations.datasets.hl7V2Stores.messages.export method.

Note the following when calling the export operation:

  • Write to a Cloud Storage bucket or directory, rather than an object, because the Cloud Healthcare API might create multiple newline-delimited JSON files when there are many messages. In each JSON file, each line is an HL7v2 message.
  • If the command specifies a directory that does not exist, the directory is created.

curl

To export HL7v2 messages, make a POST request and specify the following information:

  • The name of the parent dataset
  • The name of the HL7v2 store
  • The destination Cloud Storage bucket or directory
  • An access token

The following sample shows a POST request using curl.

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    --data "{
      'gcsDestination': {
        'uriPrefix': 'gs://BUCKET/DIRECTORY'
      }
    }" "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/hl7V2Stores/HL7V2_STORE_ID:export"

If the request is successful, the server returns the response in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"
}

The response contains an operation name. To track the status of the operation, you can use the Operation get method:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"

If the request is successful, the server returns a response with the status of the operation in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.OperationMetadata",
    "apiMethodName": "google.cloud.healthcare.v1.hl7v2.Hl7V2Service.ExportMessages",
    "createTime": "CREATE_TIME",
    "endTime": "END_TIME",
    "counter": {
      "success": "RESOURCE_COUNT"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.hl7v2.ExportMessagesResponse"
  }
}

PowerShell

To export HL7v2 messages, make a POST request and specify the following information:

  • The name of the parent dataset
  • The name of the HL7v2 store
  • The destination Cloud Storage bucket or directory
  • An access token

The following sample shows a POST request using Windows PowerShell.

$cred = gcloud auth application-default print-access-token
$headers = @{ Authorization = "Bearer $cred" }

Invoke-WebRequest `
  -Method Post `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body "{
    'gcsDestination': {
      'uriPrefix': 'gs://BUCKET/DIRECTORY'
    }
  }" `
  -Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/hl7V2Stores/HL7V2_STORE_ID:export" | Select-Object -Expand Content

If the request is successful, the server returns the response in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID"
}

The response contains an operation name. To track the status of the operation, you can use the Operation get method:

$cred = gcloud auth application-default print-access-token
$headers = @{ Authorization = "Bearer $cred" }

Invoke-WebRequest `
  -Method Get `
  -Headers $headers `
  -Uri "https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID" | Select-Object -Expand Content

If the request is successful, the server returns a response with the status of the operation in JSON format:

{
  "name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.OperationMetadata",
    "apiMethodName": "google.cloud.healthcare.v1.hl7v2.Hl7V2Service.ExportMessages",
    "createTime": "CREATE_TIME",
    "endTime": "END_TIME",
    "counter": {
      "success": "RESOURCE_COUNT"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.cloud.healthcare.v1.hl7v2.ExportMessagesResponse"
  }
}

Troubleshooting HL7v2 import and export requests

If errors occur during an HL7v2 import or export request, the errors are logged to Cloud Logging. For more information, see Viewing error logs in Cloud Logging.

If an operation returns an error, see Troubleshooting long-running operations.