Quickstart: Create a lake

Quickstart: Create a lake

This document shows you how to get started with Dataplex in the Google Cloud console, by walking you through creating a lake, adding a zone, and attaching an asset.

Before you begin

Note: For Dataplex use cases, don't create or use a lake with the ID goog-dataproc-default-lake because it serves as the default lake exclusively intended for Dataproc workspaces.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Dataplex, Dataproc, Dataproc Metastore, Data Catalog, BigQuery, and Cloud Storage APIs.

   Enable the APIs

5. Make sure that you have the following role or roles on the project: `roles/dataplex.admin`, `roles/dataplex.editor`

   Check for the roles

   1. In the Google Cloud console, go to the IAM page.

      Go to IAM
   2. Select the project.

   3. In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

   4. For all rows that specify or include you, check the Role colunn to see whether the list of roles includes the required roles.

   Grant the roles

   1. In the Google Cloud console, go to the IAM page.

      Go to IAM
   2. Select the project.
   3. Click Grant access.

   4. In the New principals field, enter your user identifier. This is typically the email address for a Google Account.

   5. In the Select a role list, select a role.
   6. To grant additional roles, click Add another role and add each additional role.
   7. Click Save.

6. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

7. Make sure that billing is enabled for your Google Cloud project.

8. Enable the Dataplex, Dataproc, Dataproc Metastore, Data Catalog, BigQuery, and Cloud Storage APIs.

   Enable the APIs

9. Make sure that you have the following role or roles on the project: `roles/dataplex.admin`, `roles/dataplex.editor`

   Check for the roles

   1. In the Google Cloud console, go to the IAM page.

      Go to IAM
   2. Select the project.

   3. In the Principal column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator.

   4. For all rows that specify or include you, check the Role colunn to see whether the list of roles includes the required roles.

   Grant the roles

   1. In the Google Cloud console, go to the IAM page.

      Go to IAM
   2. Select the project.
   3. Click Grant access.

   4. In the New principals field, enter your user identifier. This is typically the email address for a Google Account.

   5. In the Select a role list, select a role.
   6. To grant additional roles, click Add another role and add each additional role.
   7. Click Save.
Create a Cloud Storage bucket:
1. In the Google Cloud console, go to the Cloud Storage Buckets page.

   Go to Buckets page

2. Click Create bucket.
3. On the Create a bucket page, enter your bucket information. To go to the next step, click Continue.
   • For Name your bucket, enter a unique bucket name. Don't include sensitive information in the bucket name, because the bucket namespace is global and publicly visible.
   • For Choose where to store your data, do the following:
     • Select a Location type option.
     • Select a Location option.
   • For Choose a default storage class for your data, select the following: Standard.
   • For Choose how to control access to objects, select an Access control option.
   • For Advanced settings (optional), specify an encryption method, a retention policy, or bucket labels.
4. Click Create.

Create a lake

A lake is a logical construct representing a data domain or business unit. For example, if you need to organize data based on group usage, you would create a lake for each department (for example, retail, sales, and finance).

The following steps show you how to create a lake using the Google Cloud console.

1. Go to Dataplex in the Google Cloud console.

   Go to Dataplex

2. Navigate to the Manage view.

3. Click Create.

4. Enter a Display name.

5. The lake ID is automatically generated for you.

6. Specify the Region in which to create the lake.

   For lakes created in a given region (for example, us-central1), both single-region (us-central1) data and multi-region (us multi-region) data can be attached depending on the zone settings.

7. Click Create.

Add a zone to your lake

After you create your lake, you can add zones to the lake. Zones are logical groupings within a lake, that are useful for categorizing structured and unstructured data.

1. In the Manage view, click the name of the lake you want to add a zone to.

2. Click Add zone.

3. Enter a Display name for your zone.

   Note: The zone ID is automatically generated for you. If you need to, you can provide your own ID.

4. Click the Type drop-down. Choose Raw Zone or Curated Zone. Learn more about the types of zones.

5. Under Data locations select either Regional or Multi-regional. What you choose cannot be changed later. Single region and multi-region data cannot be mixed in the same zone.

   Note: Discovery is enabled by default. Click to expand Discovery settings and see that Enable metadata discovery is selected. Learn more about Discovery.

6. Click Create.

It may take a few minutes for the zone to be created.

Attach an asset

Data can be stored in Cloud Storage buckets or BigQuery datasets, and can be attached as assets to data zones within a Dataplex lake.

To attach your Cloud Storage bucket as an asset, follow these steps:

1. In the Manage view, click the name of your lake to which you want to attach a Cloud Storage bucket to.

2. On the Zones tab, click the zone to add the asset to.

3. On the Assets tab, click Add Assets.

4. Click Add an asset.

5. Under Type, select Storage bucket.

6. Under Display name, enter a name for the asset.

   Note: The asset ID is automatically generated for you. If you need to, you can provide your own ID.

7. In the Bucket field, Click Browse. If you have a Cloud Storage bucket, find it and click Select. If you don't have a Cloud Storage bucket, you can create one by clicking the button.

   1. Enter a unique name for the bucket. Click Continue.

   2. Choose a Location type. Click Continue.

   3. Choose a default storage class for your data. Click Continue.

   4. Choose an access control level. Click Continue.

   5. Choose a data protection option or None. Click Continue.

   6. Click Create.

   7. Click Select

8. Click Done.

9. Click Continue.

10. Under Discovery settings, select Inherit to inherit the Discovery settings from the zone level.

11. Click Continue.

12. Under Add assets, click Submit.

Wait for the Asset creation to finish.

To use your lake, see the What's next section. Otherwise, delete the resources you created by following the steps in the Clean up section.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. If the project that you plan to delete is attached to an organization, expand the Organization list in the Name column.
3. In the project list, select the project that you want to delete, and then click Delete.
4. In the dialog, type the project ID, and then click Shut down to delete the project.

Alternatively, you can delete the resources used in this tutorial. A lake isn't deleted until you delete all of its data zone resources. Similarly, a data zone isn't deleted unless you delete all of its asset resources.

Detach the storage bucket

To detach the Dataplex asset you created, follow these steps:

1. Go to Dataplex in the Google Cloud console.

   Go to Dataplex

2. In the Manage view, click the name of the lake you created.

3. In the Zones tab, click the name of the zone you created.

4. In the Assets tab, select the asset to detach by checking the box to the left of the bucket name.

5. Click Delete Asset.

6. Click Delete to confirm the detachment.

Delete the zone

To delete the Dataplex zone you created, follow these steps:

1. Go to Dataplex in the Google Cloud console.

   Go to Dataplex

2. In the Manage view, click the lake you created.

3. On the Zones tab, select the zone to delete by checking the box to the left of the data zone name.

4. Click Delete Zone.

5. Click Delete to confirm the deletion.

Delete the lake

The following steps show you how to delete the Dataplex lake you created.

1. Go to Dataplex in the Google Cloud console.

   Go to Dataplex

2. In the Manage view, click the lake you created.

3. At the top of the page, click Delete.

4. Confirm deletion by typing "delete" in the field.

5. Click Delete Lake to confirm the deletion.

What's next

• Organize your data into lakes and zones.
• Secure your lake.
• View discovered metadata by using the Google Cloud console.
• View discovered metadata by using the API.