This document describes how to create custom Data Catalog entries. To integrate your custom data sources, call Data Catalog APIs that allow you to create and manage Data Catalog entries with custom data resource types. In this document, an entry for a custom data resource type is referred to as a "custom entry".
Create entry groups and custom entries
Custom entries must be placed within a user-created entry group. You create the entry group, then create the custom entry within the entry group. For more information, see Entries and entry groups.
After creating an entry, you can set IAM policies on the entry group to define who has access to the entry group and the entries inside.
Before trying this sample, follow the Java setup instructions in the Data Catalog quickstart using client libraries. For more information, see the Data Catalog Java API reference documentation.
To authenticate to Data Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Before trying this sample, follow the Node.js setup instructions in the Data Catalog quickstart using client libraries. For more information, see the Data Catalog Node.js API reference documentation.
To authenticate to Data Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Before trying this sample, follow the Python setup instructions in the Data Catalog quickstart using client libraries. For more information, see the Data Catalog Python API reference documentation.
To authenticate to Data Catalog, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
REST
See the following examples and refer to the Data Catalog REST API entryGroups.create and entryGroups.entries.create documentation.
1. Create an entry group
Before using any of the request data, make the following replacements:
- project-id: Your Google Cloud project ID
- entryGroupId: The ID must begin with a letter or underscore, contain only English letters, numbers and underscores, and be at most 64 characters.
- displayName: The textual name for the entry group.
HTTP method and URL:
POST https://datacatalog.googleapis.com/v1/projects/project-id /locations/region /entryGroups?entryGroupId=entryGroupId
Request JSON body:
{ "displayName": "Entry Group display name " }
To send your request, expand one of these options:
curl (Linux, macOS, or Cloud Shell)
Save the request body in a file named request.json
,
and execute the following command:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project:project-id " \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://datacatalog.googleapis.com/v1/projects/project-id /locations/region /entryGroups?entryGroupId=entryGroupId "
PowerShell (Windows)
Save the request body in a file named request.json
,
and execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project-id " }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://datacatalog.googleapis.com/v1/projects/project-id /locations/region /entryGroups?entryGroupId=entryGroupId " | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/my_projectid/locations/us-central1/entryGroups/my_entry_group", "displayName": "Entry Group display name", "dataCatalogTimestamps": { "createTime": "2019-10-19T16:35:50.135Z", "updateTime": "2019-10-19T16:35:50.135Z" } }
2. Create a custom entry within the entry group
Before using any of the request data, make the following replacements:
- project_id: Your Google Cloud project ID.
- entryGroupId: ID of the existing entryGroup. The entry will be created in this EntryGroup.
- entryId: ID of the new entry. The ID must begin with a letter or underscore, contain only English letters, numbers and underscores, and have at most 64 characters.
- description: Optional entry description
- displayName: Optional textual name for the entry.
- userSpecifiedType: Custom type name. The type name must begin with a letter or underscore, must only contain letters, numbers, and underscores, and must have at most 64 characters.
- userSpecifiedSystem: The entry's non-Google Cloud source system, which is not integrated with Data Catalog. The source system name must begin with a letter or underscore, must only contain letters, numbers, and underscores, and must have at most 64 characters.
- linkedResource: Optional fullname of the resource the entry refers to.
- schema:
Optional data schema.
Example JSON schema:
{ ... "schema": { "columns": [ { "column": "first_name", "description": "First name", "mode": "REQUIRED", "type": "STRING" }, { "column": "last_name", "description": "Last name", "mode": "REQUIRED", "type": "STRING" }, { "column": "address", "description": "Address", "mode": "REPEATED", "subcolumns": [ { "column": "city", "description": "City", "mode": "NULLABLE", "type": "STRING" }, { "column": "state", "description": "State", "mode": "NULLABLE", "type": "STRING" } ], "type": "RECORD" } ] } ... }
HTTP method and URL:
POST https://datacatalog.googleapis.com/v1/projects/project_id /locations/region /entryGroups/entryGroupId /entries?entryId=entryId
Request JSON body:
{ "description": "Description ", "displayName": "Display name ", "userSpecifiedType": "my_type ", "userSpecifiedSystem": "my_system ", "linkedResource": "example.com/def ", "schema": {schema } }
To send your request, expand one of these options:
curl (Linux, macOS, or Cloud Shell)
Save the request body in a file named request.json
,
and execute the following command:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "x-goog-user-project:project_id " \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://datacatalog.googleapis.com/v1/projects/project_id /locations/region /entryGroups/entryGroupId /entries?entryId=entryId "
PowerShell (Windows)
Save the request body in a file named request.json
,
and execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "project_id " }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://datacatalog.googleapis.com/v1/projects/project_id /locations/region /entryGroups/entryGroupId /entries?entryId=entryId " | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/my_project_id/locations/us-central1/entryGroups/my_entryGroup_id/entries/my_entry_id", "userSpecifiedType": "my-type", "userSpecifiedSystem": "my_system", "displayName": "On-prem entry", "description": "My entry description.", "schema": { "columns": [ { "type": "STRING", "description": "First name", "mode": "REQUIRED", "column": "first_name" }, { "type": "STRING", "description": "Last name", "mode": "REQUIRED", "column": "last_name" }, { "type": "RECORD", "description": "Address", "mode": "REPEATED", "column": "address", "subcolumns": [ { "type": "STRING", "description": "City", "mode": "NULLABLE", "column": "city" }, { "type": "STRING", "description": "State", "mode": "NULLABLE", "column": "state" } ] } ] }, "sourceSystemTimestamps": { "createTime": "2019-10-23T23:11:26.326Z", "updateTime": "2019-10-23T23:11:26.326Z" }, "linkedResource": "example.com/def" }