Configuring Cloud Service Mesh user authentication
If you have the TRAFFIC_DIRECTOR
control plane implementation then
this feature is only supported by allow-list. Contact support to request access.
Cloud Service Mesh user authentication is an integrated solution for browser-based end-user authentication and access control to your deployed workloads. It lets you integrate with existing Identity Providers (IDP) for user authentication and uses Istio APIs and authorization policies for access management. It is a user-friendly alternative to Istio JSON Web Token (JWT) authentication.
A typical use-case is when an organization uses Cloud Service Mesh to host a web application for its workforce to access via a web browser. In addition, the organization needs to use their existing identity provider to manage user identities. Cloud Service Mesh user authentication makes it easy for users to authenticate using a standard web-based OpenID Connect (OIDC) login and consent flow. When the user authenticates, Cloud Service Mesh enforces Istio authorization policies, and on successful authorization, it transmits the identity to workloads in a secure credential format.
How it works
Cloud Service Mesh user authentication introduces a new component, authservice
.
This component integrates with the Envoy-based ingress as an external
authorization service that intercepts all the incoming requests for
authentication. authservice
implements the client-side of the OIDC protocol
and enables user access to applications via a browser, where users complete an
interactive authentication and consent flow to establish a short-lived session.
authservice
implements industry standard protocols to integrate with any
identity provider that can act as a OIDC authorization server. When the user is
authenticated, the principal information is encapsulated in an RCToken
in JWT
format, signed by authservice
which it forwards to the Istio authorization
layer in the ingress. This model provides perimeter access control for traffic
into the mesh. If the user is authorized to access a resource, this RCToken is
also forwarded to the microservices to obtain principal information and enforce
fine-grained access control.
The following diagram shows the location of authservice
in the mesh and how it
relates to the other parts of the mesh, such as the ingress, workloads, user's
browser, and any existing IDP.
Administrators can install authservice
as an add-on over an Cloud Service Mesh
installation. When installed, authservice
reads the OIDC endpoint
configuration and other associated settings defined in the UserAuth
custom
resource. The administrator can use Cloud Service Mesh ExternalAuthorization
APIs
to configure auth_server
as a filter on the ingress.
Install the user authentication service
The following steps explain how to configure the authservice
.
Prerequisites
Follow the steps in Install dependent tools and validate cluster to:- Install required tools
- Download
asmcli
- Grant cluster admin permissions
- Validate your project and cluster
- If you are using managed Cloud Service Mesh on a private cluster, then ensure the cluster is capable of sending egress traffic to the IDP.
Additionally, ensure that you meet the prerequisites by using the following steps.
Customize the installation user authentication overlay
To install the user authentication service, you must customize the Cloud Service Mesh installation to add a mesh-level external authorization provider. The steps required depend on whether you are using managed or in-cluster Cloud Service Mesh.
Managed
Update the ConfigMap to include the user authentication MeshConfig. In the following command, use the same
REVISION_LABEL
you used when provisioning managed Cloud Service Mesh (such as,asm-managed
,asm-managed-rapid
, orasm-managed-stable
):kubectl edit configmap istio-REVISION_LABEL -n istio-system
Add the following text under
mesh
field in the MeshConfig:mesh: |- ... extensionProviders: - name: "asm-userauth-grpc" envoyExtAuthzGrpc: service: "authservice.asm-user-auth.svc.cluster.local" port: "10003"
Create
asm-user-auth
namespace.kubectl create namespace asm-user-auth
Enable the namespace for injection. The steps depend on your control plane implementation.
Managed (TD)
Apply the default injection label to the namespace:
kubectl label namespace asm-user-auth \ istio.io/rev- istio-injection=enabled --overwrite
Managed (Istiod)
Recommended: Run the following command to apply the default injection label to the namespace:
```sh kubectl label namespace asm-user-auth \ istio.io/rev- istio-injection=enabled --overwrite ```
If you are an existing user with the Managed Istiod control plane: We recommend that you use default injection, but revision-based injection is supported. Use the following instructions:
- Run the following command to locate the available release channels:
kubectl -n istio-system get controlplanerevision
The output is similar to the following:
NAME AGE asm-managed-rapid 6d7h
In the output, the value under the
NAME
column is the revision label that corresponds to the available release channel for the Cloud Service Mesh version.Apply the revision label to the namespace:
kubectl label namespace asm-user-auth \ istio-injection- istio.io/rev=REVISION_LABEL --overwrite
Install the Istio gateway in the
asm-user-auth
namespace.kubectl apply -n asm-user-auth -f DIR_PATH/samples/gateways/istio-ingressgateway
In-cluster
Get the example user auth overlay and update it if there are any customizations in your mesh. It is a recommended best practice to maintain this overlay file in your source control.
curl https://raw.githubusercontent.com/GoogleCloudPlatform/asm-user-auth/v1.2.3/overlay/user-auth-overlay.yaml > user-auth-overlay.yaml
Follow the install Cloud Service Mesh with overlay to use a Google-provided script to install Cloud Service Mesh with the user authentication overlay. For example:
./asmcli install \ --project_id PROJECT_ID \ --cluster_name CLUSTER_NAME \ --cluster_location CLUSTER_LOCATION \ --fleet_id FLEET_PROJECT_ID \ --output_dir DIR_PATH \ --enable_all \ --custom_overlay user-auth-overlay.yaml
The user authentication
kpt
packages creates anAuthorizationPolicy
to reference the external authorization provider specified bypkg/ext-authz.yaml
.Create
asm-user-auth
namespace.kubectl create namespace asm-user-auth
Enable the namespace for injection.
Recommended: Run the following command to apply the default injection label to the namespace:
kubectl label namespace asm-user-auth \ istio.io/rev- istio-injection=enabled --overwrite
We recommend that you use default injection, but revision-based injection is supported: Use the following instructions:
Use the following command to locate the revision label on
istiod
:kubectl get deploy -n istio-system -l app=istiod -o \ jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}'
Apply the revision label to the namespace. In the following command,
REVISION_LABEL
is the value of theistiod
revision label that you noted in the previous step.kubectl label namespace asm-user-auth \ istio-injection- istio.io/rev=REVISION_LABEL --overwrite
Install the Istio gateway in the
asm-user-auth
namespace.kubectl apply -n asm-user-auth -f DIR_PATH/samples/gateways/istio-ingressgateway
Prepare the OIDC client configuration
Set your OIDC client configuration by using the following steps. This guide uses Google as an IDP, but you can use any IDP that supports OIDC authentication.
In the Google Cloud console, go to API & Services > Credentials.
Go to Create Credentials, then choose OAuth client ID. If required, set your OAuth consent screen options, then configure the following options:
- Set Application type to Web application.
- Set Authorized redirect URI to
https://REDIRECT_HOST/REDIRECT_PATH
. For example, for localhost you could set tohttps://localhost:8443/_gcp_asm_authenticate
.
Then, click Save.
In addition, save your OIDC client configuration to use later.
export OIDC_CLIENT_ID=CLIENT_ID export OIDC_CLIENT_SECRET=CLIENT_SECRET export OIDC_ISSUER_URI=ISSUER_URI export OIDC_REDIRECT_HOST=REDIRECT_HOST export OIDC_REDIRECT_PATH=REDIRECT_PATH
Get the kpt
packages
Use the following steps to install the recommended authservice
configuration
from the
public repository. These
commands retrieve the latest authservice
container and start it as a Pod in
the asm-user-auth
namespace. It also configures the ingress to intercept all
requests.
Get the kpt package:
kpt pkg get https://github.com/GoogleCloudPlatform/asm-user-auth.git/@v1.2.3 .
cd asm-user-auth/
Set the redirection URL and secret for ingress gateway
OAuth2
requires a redirection URL hosted on an HTTPS-protected endpoint. These
commands are for example purposes and simplify setup by generating a self-signed
certificate for the Istio ingress gateway.
Generate a self-signed certificate:
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \ -days 365 -nodes -subj '/CN=localhost'
Create a secret for the ingress gateway to host HTTPS traffic:
kubectl create -n asm-user-auth secret tls userauth-tls-cert --key=key.pem \ --cert=cert.pem
Apply the encryption and signing keys
The authservice
needs two sets of keys to operate successfully. The first is a
symmetric key for encryption and decryption. This key is used for encrypting the
session state before setting that as a cookie.
The second set of keys are a public/private key pair. This key is used to sign the authenticated user information in JWT format as an RCToken. The public key from this pair is published at a predefined endpoint that the sidecars can use to validate the JWT.
The user authentication kpt
package contains two sample keys for quick setup.
However, you can use your preferred key management system to generate these keys
instead.
Prepare the session encryption key with following format or use the sample from the pkg, which you can view by
cat ./samples/cookie_encryption_key.json
.{ "keys":[ { "kty":"oct", "kid":"key-0", "K":"YOUR_KEY", "useAfter": 1612813735 } ] }
You can generate a test AES key with following command:
openssl enc -aes-256-cbc -k mycustomkey -P -md sha1 | grep key
Prepare the RCToken signing key with following format or use the sample from the pkg, which you can view by
cat ./samples/rctoken_signing_key.json
.{ "keys":[ { "kty":"RSA", "kid":"rsa-signing-key", "K":"YOUR_KEY", # k contains a Base64 encoded PEM format RSA signing key. "useAfter": 1612813735 # unix timestamp } ] }
You can generate a test 256-bit RSA private key with following command:
openssl genpkey -algorithm RSA -out rsa_private.pem -pkeyopt rsa_keygen_bits:256
Create the kubernetes secret, which
authservice
will mount into its own file system.kubectl create secret generic secret-key \ --from-file="session_cookie.key"="./samples/cookie_encryption_key.json" \ --from-file="rctoken.key"="./samples/rctoken_signing_key.json" \ --namespace=asm-user-auth
Deploy the user authentication service
The following commands create the user authentication service and deployment in
the asm-user-auth
namespace.
Set the necessary values for User Auth configuration. The client ID and secret are stored as Kubernetes secrets, so we use Base64 to encode them. Please visit public repository to view all available setters.
kpt fn eval pkg --image gcr.io/kpt-fn/apply-setters:v0.2 --truncate-output=false -- \
client-id="$(echo -n ${OIDC_CLIENT_ID} | base64 -w0)" \
client-secret="$(echo -n ${OIDC_CLIENT_SECRET} | base64 -w0)" \
issuer-uri="${OIDC_ISSUER_URI}" \
redirect-host="${OIDC_REDIRECT_HOST}" \
redirect-path="${OIDC_REDIRECT_PATH}"
Apply the kpt
package:
# Remove the potential alpha version CRD if exists.
kubectl delete crd userauthconfigs.security.anthos.io
kubectl apply -f ./pkg/asm_user_auth_config_v1beta1.yaml
kubectl apply -f ./pkg
The authservice
consumes the UserAuthConfig
CRD to provide end user
authentication. UserAuthConfig
is configurable in the run time, and you can
update it to change the authservice
behavior and configure it with endpoints
for any OIDC authorization server.
You can view the file by cat pkg/user_auth_config.yaml
, it contains these fields:
apiVersion: security.anthos.io/v1beta1
kind: UserAuthConfig
metadata:
name: user-auth-config
namespace: asm-user-auth
spec:
authentication:
oidc:
certificateAuthorityData: "" # kpt-set: ${ca-cert}
issuerURI: "<your issuer uri>" # kpt-set: ${issuer-uri}
proxy: "" # kpt-set: ${proxy}
oauthCredentialsSecret:
name: "oauth-secret" # kpt-set: ${secret-name}
namespace: "asm-user-auth" # kpt-set: ${secret-namespace}
redirectURIHost: "" # kpt-set: ${redirect-host}
redirectURIPath: "/_gcp_asm_authenticate" # kpt-set: ${redirect-path}
scopes: "" # kpt-set: ${scopes}
groupsClaim: "" # kpt-set: ${groups}
outputJWTAudience: "test_audience" # kpt-set: ${jwt-audience}
See
user authentication configuration details
for detailed descriptions of the user_auth_config.yaml
fields.
Perform post-install tasks
The following tasks are required after you finish the previous installation steps.
Enable user authentication for your applications
This section demonstrates how to enable user authentication, by using the
httpbin
as an example.
Cloud Service Mesh user authentication uses a
CUSTOM
typed authorization policy to trigger the OIDC flow.
After you have
installed the Istio gateway,
configure it to serve HTTPS traffic using the TLS certificate userauth-tls-cert
you created above. Below is the pkg/gateway.yaml
configuration that you just
installed.
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: userauth
namespace: asm-user-auth
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: userauth-tls-cert
---
# This ensures the OIDC endpoint has at least some route defined.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: userauth-oidc
namespace: asm-user-auth
spec:
gateways:
- userauth
hosts:
- '*'
http:
- match:
- uri:
prefix: /status
- uri:
prefix: "your-oidc-redirect-path"
name: user-auth-route
route:
- destination:
host: authservice
port:
number: 10004
Label
default
namespace to enableistio-proxy
auto injection for deployments.kubectl label namespace default istio.io/rev=REVISION --overwrite
Deploy
httpbin
todefault
namespace.kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml -n default
Update
httpbin
to use this gateway to serve HTTPS traffic, and use port forwarding to access the application locally:kubectl apply -f./samples/httpbin-route.yaml -n default kubectl port-forward service/istio-ingressgateway 8443:443 -n asm-user-auth
The ingress gateway on port 8443 will be forwarded to
localhost
to make the application accessible locally.Deploy the
samples/rctoken-authz.yaml
to enable RequestAuthentication and AuthorizationPolicy to verify the RCToken for the requests.kubectl apply -f ./samples/rctoken-authz.yaml -n asm-user-auth
Example
samples/rctoken-authz.yaml
:apiVersion: security.istio.io/v1beta1 kind: RequestAuthentication metadata: name: require-rc-token spec: selector: matchLabels: istio: ingressgateway jwtRules: - issuer: "authservice.asm-user-auth.svc.cluster.local" audiences: - "test_audience" jwksUri: "http://authservice.asm-user-auth.svc.cluster.local:10004/_gcp_user_auth/jwks" fromHeaders: - name: X-ASM-RCTOKEN forwardOriginalToken: true --- apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: require-rc-token spec: selector: matchLabels: istio: ingressgateway action: ALLOW rules: - when: - key: request.auth.claims[iss] values: - authservice.asm-user-auth.svc.cluster.local - key: request.auth.claims[aud] values: - test_audience
Verify user authentication
The httpbin
serves two paths, /ip
is publicly accessible and /headers
requires the end user to login via their configured IDP.
Verify that you are able to access
/ip
directly by visitinghttps://localhost:8443/ip
.Verify that you see the OIDC login page by visiting
https://localhost:8443/headers
.After you login, click Next and verify that it redirects you to the
/headers
page.
Configure authorization policies
After you finish the configuration in the previous steps, each user will be
redirected through a web-based authentication flow. When the flow completes, the
authservice
will generate an RCToken
in JWT format, which it uses to
transmit the authenticated user information.
Add Istio authorization policies at the ingress to ensure that an authorization check occurs for each authenticated user:
kubectl apply -f ./samples/httpbin-authz.yaml -n asm-user-auth
The
httpbin-authz.yaml
file configures the ingress gateway to validate the RC token issued by authservice, and only authorize when the JWT contains the desired fields, such as audiences and issuers.See the following example authorization policy:
apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: require-rc-token spec: selector: matchLabels: istio: ingressgateway action: ALLOW rules: - to: - operation: paths: ["/ip"] - to: when: - key: request.auth.claims[iss] values: - authservice.asm-user-auth.svc.cluster.local - key: request.auth.claims[aud] values: - test_audience - key: request.auth.claims[sub] values: - allowed_user_sub_1 # Change this with the "sub" claim in the RC token. Wildcard '*' will match everything.
Configure environment-specific settings
The previous steps use localhost
and a self-signed HTTPS certificate for quick
setup. For real production use, use your own domain, such as example.com
.
In addition, ensure the certificateAuthorityData
has the intended root cert
content. For example, if the IDP is trusted with system's root certs, you can
leave it empty. If there is a HTTPS proxy terminating the HTTPS connection, it
should be set to the proxy's root cert.
Manage and rotate keys
There are two sets of keys used by authservice
. You can rotate each key
independently. However, before you rotate the keys, it is important to
understand how the rotation works.
Both keys are in JSON format. The useAfter
field specifies the timestamp since
when the key will be considered to use. During a key rotation, you should
include both old and new keys in the JSON. For example, in the following
example, new-key
will only be used after timestamp 1712813735
.
{
"keys":[
{
"kty":"RSA",
"kid":"old-key",
"K":"...", # k contains a Base64 encoded PEM format RSA signing key.
"useAfter": 1612813735, # unix timestamp
}
{
"kty":"RSA",
"kid":"new-key",
"K":"...", # k contains a Base64 encoded PEM format RSA signing key.
"useAfter": 1712813735, # unix timestamp
}
]
}
Cloud Service Mesh uses the symmetric key for encrypting session data that is stored
in browser cookies. To ensure validity of existing sessions, authservice
attempts decryption with all keys in the key set. On rotation, the authservice
will use the new key for encrypting new sessions, and will continue to attempt
decryption with the old keys.
The public/private key pair is used to sign RCToken
. The public key is
transmitted to the sidecars by istiod
for JWT verification. It is crucial for
sidecars to receive the new public key before authservice
starts using the new
private key to sign the RCToken
. To that end, authservice
starts publishing
the public key immediately after the key is added, but waits a significant
amount of time before starting to use that to sign RCToken
.
To summarize, when performing key rotations we recommend:
- Perform regular key rotations or on demand as you need.
- In the JSON format, include both the current and the new keys. The new keys should be associated with a timestamp in the future. We recommend that you specify a timestamp at least a couple of hours ahead of the current time.
- Monitor and confirm that the services are still healthy after the new key is in use. Wait at least one day after the new key is being used before moving to next step.
- Remove the old keys from the JSON entries. They are no longer needed.
Multi Cluster Deployment
Cloud Service Mesh User Auth supports multi cluster deployment. You need to deploy user auth in each cluster as described above. The user auth configuration such as UserAuth custom resource, OIDC client secret, encryption keys, all need to be replicated in each clusters.
By default ingress gateway will load balance the authentication requests to any
one of authservice
instances. You can use destination rule to configure the
ingress gateway to send requests to the authservice
in the same cluster, and
only fail over to other clusters' authservice
.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: authservice-fail-over
namespace: asm-user-auth
spec:
host: authservice.asm-user-auth.svc.cluster.local
trafficPolicy:
loadBalancer:
localityLbSetting:
enabled: true
failover:
- from: us-east
to: us-west
- from: us-west
to: us-east
Same as other configuration, this needs to be configured in each cluster.
Custom claims mapping
To configure custom claims mapping, configure spec.authentication.oidc.attributeMapping
to define any mappings from original identity provider's IDToken. The key will be the claim
name in the RCToken and value is a CEL expression on how to parse the claim from IDToken,
use assertion
to reference the IDToken.
Example:
spec:
authentication:
oidc:
attributeMapping:
aud_copy: assertion.aud
decision: 'assertion.sub.startsWith("123") ? "success" : "fail"'
In the RCToken, a nested claim attributes
contains the claims that been configured:
"attributes": {
"aud_copy": "foo.googleusercontent.com",
"decision": "success"
}
If the CEL expression fails to parse the value from IDToken, it will ignore the claim without failing the authentication flow.
User Auth upgrades
Install the user-auth packages again as it contains the updated binary for new user-auth version:
kpt pkg get https://github.com/GoogleCloudPlatform/asm-user-auth.git/@v1.2.3 . cd asm-user-auth/
Save your OIDC client configuration:
export OIDC_CLIENT_ID=CLIENT_ID export OIDC_CLIENT_SECRET=CLIENT_SECRET export OIDC_ISSUER_URI=ISSUER_URI export OIDC_REDIRECT_HOST=REDIRECT_HOST export OIDC_REDIRECT_PATH=REDIRECT_PATH
Deploy the user authentication service to upgrade to a new version.
User authentication configuration details
The following table describes each field in the CRD:
Field name | Description |
---|---|
authentication.oidc |
This section holds the OIDC endpoint configuration and the parameters used in OIDC flow. |
authentication.oidc.certificateAuthorityData |
This is the SSL root certificate of the domain of the OIDC authorization server or HTTPS proxy if there is any. |
authentication.oidc.oauthCredentialsSecret |
Secret references to the Kubernetes Opaque type secret which contains OAuth2 OIDC client_id and client_secret in JSON payload. |
authentication.oidc.issuerURI |
The URI to use as the issuer in the output RCToken. |
authentication.oidc.proxy |
Proxy server to the OIDC IDP, if applicable. With format http://user:password@10.10.10.10:8888. |
authentication.oidc.redirectURIHost |
The host to be used for OAuth termination URI. If you leave this empty, the host from the
target URL will be used and the redirect URI will be assembled dynamically. This value can be used when a user auth SSO session is desired at a higher level domain. For example, to enable SSO between profile.example.com/ and admin.example.com/, this value can be set to example.com. It will enable a user auth session to be established at example.com that'll be shared amongst all subdomains. Note: If multiple domains are served from the same mesh, example1.com and example2.com, the feature cannot be used, and is recommended to be left empty. |
authentication.oidc.redirectURIPath |
The endpoint path where authservice will terminate the OAuth flow. You
should register this URI path plus the host as an authorized redirect URI in the
authorization server for the authentication.oidc.clientID .In addition, this URI should be served from the same service mesh and ingress where authservice is enabled. |
authentication.oidc.scopes |
The OAuth scope that should be requested in the authentication request. Comma-separated list of identifiers used to specify what access privileges are being requested in addition to "openid" scope, eg. "groups,allatclaim". |
authentication.oidc.groupsClaim |
If the idtoken contains a groups claim, use this field to indicate
its name. If specified, the service will pass on the data in this
claim into the groups claim in the output RCToken. This claim should contain a
comma-separated list of strings, eg. ["group1", "group2"]. |
authentication.oidc.attributeMapping |
Contains one or more claim mappings from idtoken followed CEL expressions. All claims
should be referenced by assertion.X , assertion is referenced
to the original IDToken, for example aud_copy: assertion.aud |
authentication.outputJWTAudience |
The audience of the RCToken generated by authservice . The sidecars can
validate the incoming RCToken against this audience value. |
Troubleshoot
Network accessibility to IDP.
Possible log:
error: TLS handshake failed.
.Verify by executing
curl
from the ephemeral container attached to theistio-proxy
to call IDP issuer URI. For example, see Collecting Cloud Service Mesh logs.If you're unable to connect, check the firewall rules or other network configurations for the cluster.
Root CA certificate.
Possible log:
error: The server's TLS certificate did not match expectations.
orerror: TLS handshake failed.
.Make sure the
certificateAuthorityData
holds the correct root CA certificate. When there is no HTTPS proxy terminating HTTPS traffic, this should hold the root CA certificate for the IDP. If there is one, this should hold the proxy's instead.Redirect path configuration.
Possible observation: receive 404 error page during OIDC authentication flow.
User Auth returns the "Set-Cookie" header without using the path attribute, which by default the browser uses the directory of the request url as the cookie path (scope of the cookie related to path). So we recommend to not include "/" in the redirect path unless you are intended.
The sidecar is not able to fetch jwksUri.
In some scenarios, a sidecar restriction could lead to failure of fetching jwksUri. If the namespace is not present using a wildcard (for example,
./*
oristio-system/*
) then this will not work. You must manually add their namespace in the egress sidecar.
FAQs
How do I upgrade Cloud Service Mesh with User Auth enabled?
Follow the Cloud Service Mesh upgrade process and specify the overlay file by adding
--custom_overlay user-auth-overlay.yaml
on the command line toasmcli install
.How much resources should we provision for the
authservice
? And how many requests per second can it handle?By default,
authservice
is configured with 2.0 vCPU, 256Mi memory. Under such configuration,authservice
is able to handle 500 requests per second. To handle larger amounts of requests, you should provision more CPU, which is roughly proportionally to its requests handling capacity. You can also configure multiple replicas of the authservice to increase the horizontal scalability.