Some of the default open source components included with Google Cloud Dataproc clusters, such as Apache Hadoop and Apache Spark, provide web interfaces. These interfaces can be used to manage and monitor cluster resources and facilities, such as the YARN resource manager, the Hadoop Distributed File System (HDFS), MapReduce, and Spark. Component Gateway provides secure access to web endpoints for Cloud Dataproc default and optional components.
Clusters created with Cloud Dataproc image version 1.3.29 and later can enable access to component web interfaces without relying on SSH tunnels or modifying firewall rules to allow inbound traffic.
Considerations
- Component web interfaces can be accessed by users who have dataproc.clusters.use IAM permission. By default, only project members have this permission (see Cloud Dataproc Roles).
- Component Gateway cannot be used to access REST APIs, such as Apache Hadoop YARN and Apache Livy, and history servers.
- Component Gateway is not available with Kerberized clusters.
- When Component Gateway is enabled, the first master node in the cluster will have the following additional services:
- Component gateway does not enable direct access to
node:portinterfaces, but proxies a specific subset of services automatically. If you want to access services on nodes (node:port), use an SSH SOCKS proxy.
Creating a cluster with Component Gateway
Console
To enable Component Gateway from the GCP Console, check the Component Gateway checkbox on the Cloud Dataproc Create a cluster form.
gcloud command
Run the Cloud SDK gcloud beta dataproc clusters create command locally in a terminal window or in Cloud Shell.
gcloud beta dataproc clusters create cluster-name \ --enable-component-gateway \ other args ...
REST API
Set the EndpointConfig.enableHttpPortAccess property totrue as part of a
clusters.create
request.
Viewing and Accessing Component Gateway URLs
When Component Gateway is enabled on a cluster, you can connect to component
web interfaces running on the cluster's first master node by clicking links
provided on the GCP Console. Component Gateway also sets
endpointConfig.httpPorts
with a map of port names to URLs. As an alternative to using the console,
you can use the gcloud command-line
tool or the Cloud Dataproc REST API to view this mapping information,
then copy and paste the URL into your browser to connect with the component's UI.
Console
Navigate to the Cloud Dataproc Clusters form on Google Cloud Platform Console, then select your cluster to open the Cluster details form. Click the Web Interfaces tab to display a list of Component Gateway links to the web interfaces of default and optional components installed on the cluster. Click a link to open the web interface running on the master node of the cluster in your local browser.
gcloud command
Run the Cloud SDK gcloud beta dataproc clusters describe command locally in a terminal window or in Cloud Shell.
gcloud beta dataproc clusters describe cluster-name
Sample Output
...
config:
endpointConfig:
enableHttpPortAccess: true
httpPorts:
HDFS NameNode:
https://584bbf70-7a12-4120-b25c-31784c94dbb4-dot-dataproc.google.com/hdfs/
MapReduce Job History:
https://584bbf70-7a12-4120-b25c-31784c94dbb4-dot-dataproc.google.com/jobhistory/
Spark HistoryServer:
https://584bbf70-7a12-4120-b25c-31784c94dbb4-dot-dataproc.google.com/sparkhistory/
YARN ResourceManager:
https://584bbf70-7a12-4120-b25c-31784c94dbb4-dot-dataproc.google.com/yarn/
YARN Application Timeline:
https://584bbf70-7a12-4120-b25c-31784c94dbb4-dot-dataproc.google.com/apphistory/
...
REST API
Call clusters.get to get the endpointConfig.httpPorts map of port names to URLs.What's Next
- Create a cluster with Cloud Dataproc components.
Har du brug for hjælp? Besøg vores