Deployment guide
This page guides you through the steps to deploy Manufacturing Data Engine (MDE) using Terraform, and walks you through the necessary post-deployment steps to set up MDE.
Overview
MDE is delivered as a packaged solution. A Terraform script deploys all the required components and the integration code into your Google Cloud project. This unlocks maximum flexibility for you to modify and extends the architecture based on your needs.
The deployment script automatically sets up and configures the solution within an hour, assuming prerequisites such as a Google Cloud project and permissions are in place. Some customers may have extensive, customized Google Cloud restrictions in place - if this is the case, extra work (via PSO or SI partner) may be required for the deployment to navigate around these restrictions.
Before you begin
Make sure to complete the following prerequisites before the deployment:
Read the runtime versions required for deployment documentation, as of MDE release 1.4.1, the minimum requirement for the Terraform and the Terraform providers have changed.
Your client environment has the required CLI tools installed in their most updated version:
Google Cloud CLI with the following additional components installed:
- kubectl
- cbt
- Terraform CLI ( v1.9.x or later)
- Helm CLI (v3.9.x or later)
You can use any client environment to deploy MDE, but you can save time by deploying from Cloud Shell since it has most of the required tools already installed.
You have a Google Cloud project with the following characteristics:
- An active Cloud Billing account.
- In an organization with an active Cloud Identity or Workspace account.
- Set up the default project to the MDE deployment project. You can set the default project using the following command:
gcloud config set project PROJECT_ID
Replace the following:
PROJECT_ID
with the MDE deployment project ID.
Access the MDE solution:
- Submit a MDE access request.
- Download the MDE solution release package.
- Grant your mde-imgs service account permissions to read images from the MDE Artifact Registry.
- Grant your mde-tf service account permissions to read Dataflow Flex templates from the MDE Cloud Storage bucket.
- Grant your mde-df-worker service account permissions to read images from the MDE Artifact Registry and read Dataflow templates from the MDE Cloud Storage bucket.
Name your service accounts as following:
mde-imgs@<PROJECT_ID>.iam.gserviceaccount.com
mde-df-worker@<PROJECT_ID>.iam.gserviceaccount.com
mde-tf@<PROJECT_ID>.iam.gserviceaccount.com
.
Replace the following:
PROJECT_ID
with the MDE deployment project ID.
Optional. If you have not yet requested access to MDE, a user with the
roles/iam.serviceAccountCreator
role can create the mde-imgs, mde-df-worker and mde-tf service accounts with the following commands:gcloud iam service-accounts create mde-imgs \ --description="Manufacturing Data Engine GKE Image Service Account" \ --display-name="Manufacturing Data Engine GKE Image Service Account" gcloud iam service-accounts create mde-df-worker \ --description="Manufacturing Data Engine Dataflow Worker Service Account" \ --display-name="Manufacturing Data Engine Dataflow Worker Service Account" gcloud iam service-accounts create mde-tf \ --description="Manufacturing Data Engine Terraform Service Account" \ --display-name="Manufacturing Data Engine Terraform Service Account"
- After you created the service accounts, request access to MDE using the MDE access form.
You have a service account key as a JSON file for the mde-imgs:
In order to pull container images from the MDE Artifact Registry to your GKE cluster, GKE cluster requires the mde-imgs service account key.
A user with the
roles/iam.serviceAccountKeyAdmin
role or equivalent permissions can create the service account key with the following commands:export PROJECT_ID=$(gcloud config get-value project) gcloud iam service-accounts keys create ~/mde-projects/${PROJECT_ID}/mde-imgs-service-account-key.json \ --iam-account="mde-imgs@${PROJECT_ID}.iam.gserviceaccount.com"
Grant the mde-tf service account, used by Terraform for deployments, the following roles:
roles/bigquery.admin
roles/bigtable.admin
roles/cloudsql.admin
roles/compute.instanceAdmin
roles/compute.loadBalancerAdmin
roles/compute.networkAdmin
roles/compute.securityAdmin
roles/container.admin
roles/container.developer
roles/dns.admin
roles/iam.serviceAccountAdmin
roles/iam.serviceAccountUser
roles/pubsub.admin
roles/resourcemanager.projectIamAdmin
roles/secretmanager.admin
roles/secretmanager.secretVersionManager
roles/serviceusage.serviceUsageAdmin
roles/storage.admin
roles/monitoring.admin
roles/redis.admin
roles/file.editor
A user with the
roles/iam.securityAdmin
role or equivalent permissions can grant the required roles to the mde-tf service account with the following commands:export PROJECT_ID=$(gcloud config get-value project) export SA_TERRAFORM="mde-tf" gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/bigquery.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/dataflow.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/bigtable.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/cloudsql.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/compute.instanceAdmin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/compute.loadBalancerAdmin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/compute.networkAdmin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/compute.securityAdmin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/container.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/container.developer' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/dns.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/iam.serviceAccountAdmin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/iam.serviceAccountUser' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/pubsub.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/resourcemanager.projectIamAdmin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/secretmanager.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/secretmanager.secretVersionManager' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/serviceusage.serviceUsageAdmin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/storage.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/monitoring.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/redis.admin' gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${SA_TERRAFORM}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role='roles/file.editor'
Allow your user account to impersonate the mde-tf service account if you have the
roles/iam.serviceAccountTokenCreator
role.A user with the
roles/iam.serviceAccountAdmin
role or equivalent permissions can grant your user account the role on the mde-tf service account using the following command:export USER_EMAIL=$(gcloud auth list --filter=status:ACTIVE --format="value(account)") export PROJECT_ID=$(gcloud config get-value project) gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.serviceAccountTokenCreator \ --member "user:${USER_EMAIL}" \ "mde-tf"@"${PROJECT_ID}".iam.gserviceaccount.com
Make sure Google Cloud APIs are enabled.
The following Google Cloud APIs must be enabled in order to execute Terraform scripts:
compute.googleapis.com
iamcredentials.googleapis.com
cloudresourcemanager.googleapis.com
A user with the
roles/servicemanagement.serviceConsumer
role or equivalent permissions can enable the APIs using the following command:gcloud services enable \ compute.googleapis.com \ iamcredentials.googleapis.com \ cloudresourcemanager.googleapis.com
Prepare a Cloud Storage Bucket to store Terraform state:
Terraform requires a Cloud Storage bucket to save Terraform state (
PROJECT_ID-tf
), and the mde-tf service account must have read and write permissions on this bucket.A user with the
roles/storage.admin
role or equivalent permissions can create the bucket and grant the mde-tf the requisite permissions with the following command:export PROJECT_ID=$(gcloud config get-value project) gsutil mb "gs://${PROJECT_ID}-tf" gsutil iam ch \ "serviceAccount:mde-tf@${PROJECT_ID}.iam.gserviceaccount.com:roles/storage.objectViewer" \ gs://"${PROJECT_ID}-tf" gsutil iam ch \ "serviceAccount:mde-tf@${PROJECT_ID}.iam.gserviceaccount.com:roles/storage.objectCreator" \ gs://"${PROJECT_ID}-tf"
Deployment
This section provides guidance on how to provision infrastructure and deploy MDE services using Terraform.
Configure gcloud CLI:
Login with the
--update-adc
option which saves the access credentials to the well-known location for Application Default Credentials (ADC). Configuring ADC is necessary so that Terraform can create resources as themde-tf
service account with your access credentials. Use the following command:#skip this if you're deploying using Cloud Shell gcloud auth login --update-adc
Get the MDE solution:
- Download the solution package.
- Extract the package to your client environment.
From the MDE release package root, navigate from the deployment to the Terraform directory using the following command:
cd deployment/terraform
Create a Terraform backend configuration:
To enable Terraform to store state in the
PROJECT_ID-tf
Cloud Storage bucket, you must edit thebackend.conf
file in theterraform
directory.Open the file in a text editor or use
sed
command on Linux and substitute the placeholderPROJECT_ID
for the project ID of MDE deployment project.Initialize Terraform with the backend configuration using the following command:
# Execute from deployment/terraform directory terraform init -backend-config=backend.conf -reconfigure
Edit the
input.tfvars
file and provide values for required variables:Open the
input.tfvars
in a text editor, and review the values. You must provide values for the required variables.Deployment size
Pay particular attention when selecting the appropriate deployment size. The size option changes what is deployed as well as scale parameters such as max replicas, max workers for Dataflow, machine types, and others. For a detailed explanation see the Sizes section.
MDE web interface
By default, MDE deploys with a web interface application, but it is only accessible over a private IP. See the SSH tunneling over IAP section on how to access the MDE web interface over a private IP from your local workstation using SSH tunneling with IAP after deployment.
To expose the MDE web interface to a wider group of users, you can optionally deploy an External HTTP Load Balancer for the MDE web interface by setting in
input.tfvars
:mde_ui_ext_http_lb = { enabled = true domain = "MDE_UI_DOMAIN_NAME" }
If you enable the External HTTP Load Balancer option for the MDE web interface, you must also supply a domain name. Select a domain name for which you are authorized to create DNS records.
Replace the following:
MDE_UI_DOMAIN_NAME:
with the domain name for the MDE web interface. A Google-managed SSL certificate is created for this domain.Create a Terraform plan:
Once the input parameters are ready, you must create a Terraform plan with the following command. You can use the plan to verify what artifacts configurations will be created to the project.
terraform plan -var-file=./input.tfvars -out=./tfplan
Apply the Terraform plan:
terraform apply ./tfplan
Verify that the deployment succeeded:
After the
terraform apply
command finishes executing, you should see a success message that looks similar to the following (the actual number will depend on the specific deployment options you have chosen):Apply complete! Resources: 251 added, 0 changed, 0 destroyed.
Post-deployment
This section provides guidance on required post-deployment steps to complete your MDE installation.
Enable GKE cluster filestore driver
As of 1.4.0, the MDE GKE cluster uses a
filestore instance to store JVM heap dumps which enables the engineering team to
troubleshoot customer issues more effectively. Due to Terraform restrictions,
this add-on might not be enabled for the autopilot clusters of earlier versions.
If you experience that the pods are not starting up and they are in the
ContainerCreating
state, then you need to enable the CSI driver for the
Filestore using the following command:
gcloud container clusters update CLUSTER_NAME \
--update-addons=GcpFilestoreCsiDriver=ENABLED --region REGION_NAME
Configure Identity-Aware Proxy
Identity-Aware Proxy (IAP) lets you securely connect to the internal IPs of the MDE API and web interface, as well as to restrict external access the MDE web interface to only authorized users if you deployed it with the External Load Balancer option.
To use IAP, you must first configure the OAuth screen and enable the IAP service API:
Configure the OAuth consent screen for IAP:
In the Google Cloud console, go to the Security > Identity-Aware Proxy page and select the MDE deployment project.
If you haven't configured your project's OAuth consent screen, you'll be prompted to do so:
Go to OAuth consent.
Select External user type.
Enter the Application name you want to display.
Under Support email, select the email address you want to display as a public contact. The email address must belong to the logged in user account or to a Google Group for which the logged in user is a manager or owner.
In Developer email, enter the email addresses you want Google to use to notify you about any changes to your project.
Add any optional details.
Click Save.
To change information on the OAuth consent screen later, such as the product name or email address, repeat the preceding steps to configure the consent screen.
In the Google Cloud console, go to the IAP and click Enable API. This operation may take several minutes to complete.
SSH tunneling over IAP
To connect to the private IPs of the MDE services, you can tunnel them through the proxy over SSH, and it lets you use local tools like Postman for interacting with the MDE APIs.
MDE provides a Postman environment and collection that contains example requests for the configuration manager, metadata manager, and federation APIs.
Before you begin
Make sure you completed the general IAP configuration steps.
The standard MDE deployment creates a proxy VM called mde-proxy
. This machine can proxy
incoming requests to the MDE API gateway or
MDE web interface. This VM
only has a private IP, but IAP lets you to create a secure SSH
tunnel to this machine using
Identity-Aware Proxy (IAP) for TCP forwarding.
The rest of this guide explains how to configure IAP for
tunneling traffic to the mde-proxy
, and how to create a tunnel to it from your
workstation:
In the Google Cloud console, go to the IAP and click SSH AND TCP RESOURCES
Grant users who are authorized to use IAP to connect to
mde-proxy
over IAP:Before users can use IAP to connect to
mde-proxy
, grant them theroles/iap.tunnelResourceAccessor
role with the following instructions:Select mde-proxy from the list.
Click Add Principal in the right side panel.
Type the email of the principal you want to grant access.
Select the
IAP-secured Tunnel User
role.
Create a tunnel to the MDE API using the following command:
export MDE_PROXY_ZONE=$(gcloud compute instances list --filter="mde-proxy" --format="value(zone)") gcloud compute ssh mde-proxy \ --zone "$MDE_PROXY_ZONE" --tunnel-through-iap \ -- -N -L 8080:api.mde.cloud.google.com:80
After running the command, the MDE APIs are accessible under
http://localhost:8080
.Create a tunnel to the MDE web interface.
If you created a tunnel to the MDE API in Step 3, you can open a new terminal to execute the commands later. The SSH tunnels to the MDE API and MDE web interface can run in parallel.
export MDE_PROXY_ZONE=$(gcloud compute instances list --filter="mde-proxy" --format="value(zone)") gcloud compute ssh mde-proxy \ --zone "$MDE_PROXY_ZONE" --tunnel-through-iap \ -- -N -L 3000:ui.mde.cloud.google.com:80
After running the command, the MDE web interface is accessible under
http://localhost:3000
.
Configure access to the web interface External HTTP Load Balancer
In this section, you configure access to the MDE web interface External HTTP Load Balancer using IAP.
If you enabled the External HTTP Load balancer for the MDE web interface in Step 6, you must use IAP to restrict access to the application to only authorized users.
Before you begin
Make sure you complete the following prerequisites:
- You deployed MDE with the MDE web interface.
- You deployed the MDE web interface with an External HTTP Load Balancer.
- You completed the general IAP configuration steps.
Steps
- In the Google Cloud console, go to the IAP.
- Click Applications.
Select the
mde/mde-ui-ext-service
service.Click the Enable toggle.
Read and accept the configuration requirements.
Click Turn On. This operation might take several minutes.
Grant users who are authorized to use IAP to access the MDE web interface:
- Click Add Principals in the right side panel.
- Type the email of the principal you want to grant access.
- Select the
IAP-secured Web App User
role.
Configure DNS for MDE web interface
If you enabled the External HTTP Load balancer for the MDE
web interface in Step 6, you must set the A record of the domain name you assigned to the
variable MDE_UI_DOMAIN_NAME
in input.tfvars
to the IP
address of the external HTTP load balancer that was deployed to finish
provisioning the Google-managed SSL certificate.
You can look up the IP address of the external HTTP balancer with the following command:
gcloud compute addresses list --filter="name~'.*mde-ui.*'" --format="value(address)" --global
Consult your DNS host for details on how to create an A record.
Apply the default configuration package
After establishing connectivity to the MDE APIs by
configuring SSH tunneling over IAP, you can apply the default MDE
configuration package using curl
.
The standard MDE deployment doesn't apply any configuration settings by default. Applying the default configuration package lets you to start quickly ingesting data Manufacturing Connect edge (MCe) and other data sources. See the Quickstart guide for more information about the default configuration package.
Before you begin
The curl
deployment option requires that you perform the following:
- Install the
curl
utility on your workstation. - Configure SSH tunneling over IAP or otherwise have established connectivity between your client and the MDE APIs.
Steps
Open a tunnel to the
mde-proxy
by executing using the following command:export MDE_PROXY_ZONE=$(gcloud compute instances list --filter="mde-proxy" --format="value(zone)") gcloud compute ssh mde-proxy \ --zone "$MDE_PROXY_ZONE" --tunnel-through-iap \ -- -N -L 8080:api.mde.cloud.google.com:80
Download the default configuration package shell script.
Execute the shell script
sh default-package.sh
.
Set up an MDE Looker project
MDE provides an MDE Looker Project Template that helps you set up a new Looker project that uses the MDE LookML Library. The library contains LookML components to start exploring and visualizing your data quickly. Follow Looker setup guide on how to set up a new Looker project using the MDE Looker Project Template.
Install MCe
For installation instructions, see the documentation on the MCe Google Cloud Marketplace page.
Optional: Deploy an External HTTP Load Balancer for MC
MCe can be deployed with a TCP Load Balancer, in which case you must upload an SSL certificate to the MC application to secure traffic between MC and clients. However, MDE provides a Helm chart to provision an External HTTP Load Balancer with a Google-managed SSL certificate.
Before you begin
Make sure to complete the following prerequisites:
- You deployed MCe from the Google Cloud Marketplace.
- MC is deployed in the
default
GKE cluster namespace. If MC is deployed in a different namespace, append the--namespace
option tohelm
andkubectl
commands and provide the name of the namespace in which MC is deployed. - You are authorized to deploy Kubernetes resources to the GKE cluster in which MCe is deployed.
- You have downloaded and un-archived the MDE release package.
Steps
From the MDE release package root, navigate to the optional/charts/mc/helm-mc-lb directory using the following command:
cd optional/charts/mc/helm-mc-lb
Obtain the cluster credentials using the following commands:
export CLUSTER_NAME="mc-cluster" export CLUSTER_LOCATION=$(gcloud container clusters list \ --filter="name:${CLUSTER_NAME}" \ --format="value(LOCATION)" ) gcloud container clusters get-credentials ${CLUSTER_NAME} \ --region ${CLUSTER_LOCATION} export KUBE_CONFIG_PATH=~/.kube/config
Update the
values.yaml
:Open the
values.yaml
in a text editor and set the value toingress.domain
to a domain name under which you want MC to be accessible. You must have permissions to create DNS records for this domain name.Install the Helm chart using the following command:
helm install mc-http-lb .
Obtain the IP address of the External HTTP Load Balancer:
After the Load Balancer finishes provisioning (it may take up to 5 minutes), look up the external IP address of the Load Balancer using the following command:
gcloud compute addresses list --filter="name~'.*mc-http-lb.*'" --format="value(address)" --global
Create a DNS entry:
Create an A record for the domain you specified in Step 3. Consult your DNS host for details on how to create an A record.
Change the Base Domain Name in MC:
Use
mc-nginx
Service IP to access the MC administration console by navigating tohttps://<IP>/admin-ui/settings/domain
, and change the Base Domain Name to the domain name you selected in the step earlier. You can obtain the Service IP address using the following command:kubectl get service/mc-nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'