About the Cloud SQL Proxy

MySQL   |  PostgreSQL

Beta

This is a beta release of Cloud SQL for PostgreSQL. This product might be changed in backward-incompatible ways and is not subject to any SLA or deprecation policy.

This page provides a basic introduction to the Cloud SQL Proxy, and describes the proxy options.

For step-by-step instructions on using the Cloud SQL Proxy, follow the link for your environment:

• How to connect using the proxy from Google Compute Engine
• How to connect using the proxy from an external application
• How to connect a psql client using the proxy from a client machine
• How to connect a psql client using the proxy Docker client
• How to connect using the proxy from Google Container Engine

What the proxy provides

The Cloud SQL Proxy provides secure access to your Cloud SQL Second Generation instances without having to whitelist IP addresses or configure SSL.

Accessing your Cloud SQL instance using the Cloud SQL Proxy offers these advantages:

• Secure connections: The proxy automatically encrypts traffic to and from the database; SSL certificates are used to verify client and server identities.
• Easier connection management: The proxy handles authentication with Google Cloud SQL, removing the need to provide static IP addresses.

The proxy uses the Cloud SQL API to authenticate with the Google Cloud Platform; you must enable the API before using the proxy. You must also provide the proxy with a valid user account.

The following diagram shows how the proxy connects to Google Cloud SQL:

How the Cloud SQL Proxy works

The Cloud SQL Proxy works by having a local client, called the proxy, running in the local environment. Your application communicates with the proxy with the standard database protocol used by your database. The proxy uses a secure tunnel to communicate with its companion process running on the server.

When you start the proxy, you provide it with the following sets of information:

• What Cloud SQL instances it should establish connections to
• Where it will listen for data coming from your application to be sent to Cloud SQL
• Where it will find the credentials it will use to authenticate your application to Cloud SQL

The proxy startup options you provide determine whether it will listen on a TCP port or on a Unix socket. If it is listening on a Unix socket, it creates the socket at the location you choose; usually, the /cloudsql/ directory. For TCP, the proxy listens on localhost by default.

You can install the proxy anywhere in your local environment. The location of the proxy binaries does not impact where it listens for data from your application.

Installing the Cloud SQL Proxy

Linux 64-bit

1. Download the proxy:

   wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
   

2. Make the proxy executable:

   chmod +x cloud_sql_proxy
   

Linux 32-bit

1. Download the proxy:

   wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.386 -O cloud_sql_proxy
   

2. Make the proxy executable:

   chmod +x cloud_sql_proxy
   

OS X 64-bit

1. Download the proxy:

   curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.amd64
   

2. Make the proxy executable:

   chmod +x cloud_sql_proxy
   

OS X 32-bit

1. Download the proxy:

   curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.386
   

2. Make the proxy executable:

   chmod +x cloud_sql_proxy
   

Windows 64-bit

Right-click https://dl.google.com/cloudsql/cloud_sql_proxy_x64.exe and select "Save link as..." to download the proxy, renaming it to cloud_sql_proxy.exe.

Windows 32-bit

Right-click https://dl.google.com/cloudsql/cloud_sql_proxy_x86.exe and select "Save link as..." to download the proxy, renaming it to cloud_sql_proxy.exe.

If your operating system is not included here, you can also compile the proxy from source.

Options for authenticating the Cloud SQL Proxy

Note: When you authenticate the Cloud SQL Proxy, you enable it to access the Google Cloud Platform on behalf of your application, using a set of Google credentials. This is separate from database user authentication.

The Cloud SQL Proxy provides several alternatives for authentication, depending on your environment. The proxy checks for each of the following items, in this order, using the first one it finds to attempt to authenticate:

1. Credentials supplied in the proxy invocation command
2. Credentials supplied by the local environment
3. Credentials associated with the Compute Engine instance
4. Credentials from an authenticated Cloud SDK client

See sample invocations and connection strings.

Credentials supplied in the proxy invocation command

You can create a credential file using the Google Cloud Platform Console and provide it on the command line when you start the Cloud SQL Proxy with the -credential_file parameter. The service account must have the required permissions for the Cloud SQL instance.

The advantage of this authentication method is that you can create a credential file specifically for the proxy, and it is explicitly and permanently linked to the proxy as long as it is running. For this reason, this is the recommended method for production instances not running on a Compute Engine instance.

The credential file can be duplicated in a system image if you need to invoke the Cloud SQL Proxy from multiple machines.

To use this method, you must create and manage the credential file. Only users with the resourcemanager.projects.setIamPolicy permission (such as project owners) can create the service account. If your Cloud Platform user does not have this permission, you must have someone else create the service account for you, or use another method for authenticate the proxy.

For help with creating a credential file, see Creating a service account.

Credentials supplied by the local environment

This method is identical to supplying a credential file on the command line, except that the location of the credential file is supplied using the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Credentials associated with the Compute Engine instance

If you are connecting to Cloud SQL from a Compute Engine instance, the proxy can use the service account associated with the Compute Engine instance. If the service account has the required permissions for the Cloud SQL instance, the proxy authenticates successfully.

If the Compute Engine instance is in the same project as the Cloud SQL instance, the default service account for the Compute Engine instance has the necessary permissions for authenticating the proxy. If the two instances are in different projects, you must add the Compute Engine instance's service account to the project containing the Cloud SQL instance.

Credentials from an authenticated Cloud SDK client

If you have installed the Cloud SDK and used it to authenticate to the Google Cloud Platform, the Cloud SQL Proxy can use the same credentials. This method is especially helpful for getting a development environment up and running quickly. For a production environment, you should use one of the other methods to authenticate.

Note: To enable the proxy to use your Cloud SDK credentials, you must use the gcloud auth application-default login command to authenticate the Cloud SDK.

You can determine what your current Cloud SDK credentials are by using the gcloud auth list command.

Required permissions for service accounts

When you use a service account to provide the credentials for the proxy, you must create it with sufficient permissions. If you are using the finer-grained Identity Access and Management (IAM) roles to manage your Cloud SQL permissions, you must give the service account the Cloud SQL Client role.

If you are using the legacy roles (Viewer, Editor, Owner), the service account must have at least the Editor role.

Options for specifying Cloud SQL instances

There are several ways to tell the proxy which instances you want to connect to. Some are explicit and some are implicit. In some configurations, you do not have to tell the proxy ahead of time which instances you want to connect to, because the proxy connects based on connection requests.

Your options for instance specification depend on your operating system and environment:

Option	Benefits	Caveats and Requirements	Linux/OS X (Unix sockets)	Java	Windows	Notes
Automatic instance discovery	No need to specify instances; sockets created for all instances in default project.	Proxy API usage is increased. Must have Cloud SDK installed and authenticated, with a default project set. Must restart proxy to add new instance.	Supported	No	No	Not recommended for production instances.
Project discovery	No need to specify instances; sockets created for all instances in specified projects.	Proxy API usage is increased. Must have Cloud SDK installed and authenticated. Must restart proxy to add new instance.	Supported	No	No	Use -projects parameter. Not recommended for production instances.
Instances specified on proxy invocation	Instance list known and static.	Must restart proxy to add new instance.	Supported	Supported with TCP sockets	Supported with TCP sockets	Use -instances parameter. For multiple instances, use a comma-separated list.
Instances specified using Compute Engine metadata	Instance list can be updated by changing the metadata value without restarting the proxy.	Available only on Compute Engine.	Supported	Supported with TCP sockets	Supported with TCP sockets	Use -instances_metadata flag. Learn more.

See sample invocations and connection strings.

Tips for working with Cloud SQL Proxy

Invoking the Cloud SQL Proxy

All of the example proxy invocations start the proxy in the background, so a prompt is returned. It is preferable to reserve that terminal for the proxy, to avoid having its output mixed with the output from other programs. Also, the output from the proxy can help you diagnose connection problems, so it can be helpful to capture in a log file.

You do not have to use /cloudsql as the directory for the proxy sockets. (That directory name was chosen to minimize differences with App Engine connection strings.) If you change the directory name, however, keep the overall length to a minimum; it is incorporated in a longer string that has a length limit imposed by the operating system.

Keeping the Cloud SQL Proxy up to date

Google occasionally releases new versions of the proxy. You can see what the current version is by checking the Cloud SQL Proxy Github releases page. Future proxy releases will also be noted in the Google Groups Cloud SQL announce forum.

API usage

The Cloud SQL Proxy issues requests to the Cloud SQL API. These requests count against the API quota for your project.

The highest API usage occurs when you start the proxy; this is especially true if you use automatic instance discovery or the -projects parameter. While the proxy is running, it issues 2 API calls per hour per connected instance.

Cloud SQL Proxy parameters and flags

The Cloud SQL Proxy accepts several flags and parameters when it is started. These options determine where and how the Cloud SQL Proxy creates the sockets it uses for communicating with Cloud SQL, and how it authenticates.

For help with proxy options, see the following information:

• Options for authenticating the Cloud SQL Proxy
• Options for specifying Cloud SQL instances
• Example proxy invocations
• Cloud SQL Proxy GitHub page
• The proxy help, displayed with ./cloud_sql_proxy -help

Example proxy invocations and psql client connection strings

The instance connection name is listed in the instance properties in the Google Cloud Platform Console.

Using automatic instance discovery with gcloud credentials:

./cloud_sql_proxy -dir=/cloudsql &
 psql -U [USER_NAME] -h /cloudsql/[INSTANCE_CONNECTION_NAME]

Using project discovery with gcloud credentials:

./cloud_sql_proxy -dir=/cloudsql -projects=[PROJECT-ID]&
psql -U [USER_NAME] -h /cloudsql/[INSTANCE_CONNECTION_NAME]

For a Google Compute Engine instance, with explicit instance specification:

./cloud_sql_proxy -dir=/cloudsql -instances=[INSTANCE_CONNECTION_NAME] &
psql -U [USER_NAME] -h /cloudsql/[INSTANCE_CONNECTION_NAME]

For Unix, using TCP:

./cloud_sql_proxy -instances=[INSTANCE_CONNECTION_NAME]=tcp:5432 &
psql -U [USER_NAME] -h 127.0.0.1

For Windows (at the command line prompt):

cloud_sql_proxy.exe -instances=[INSTANCE_CONNECTION_NAME]=tcp:5432
psql -U [USER_NAME] -h 127.0.0.1

Note: 5432 is the default port used for the psql client. If you use a different port number, you must specify it when you start the psql client by using the --port parameter.

For more information about Cloud SQL Proxy options and connection strings, see the Cloud SQL Proxy GitHub page.

Creating a service account

To create a service account:

Note: To create a service account with the required permissions, the user you are logged in as must have the resourcemanager.projects.setIamPolicy permission. This permission is included in the Project Owner role, as well as the Project IAM Admin and Organization Administrator roles. If your user account does not have this permission, you must ask a user with the required permission to create the service account for you.

1. Go to the Cloud SQL Service accounts page of the Google Cloud Platform Console.

   Go to the Service accounts page

2. If needed, select the project that contains your Cloud SQL instance.
3. Click Create service account.
4. In the Create service account dialog, provide a descriptive name for the service account.
5. For Role, select Cloud SQL > Cloud SQL Client.

   Alternatively, you can use the primitive Editor role by selecting Project > Editor, but the Editor role includes permissions across Google Cloud Platform.

   If you do not see these roles, your Google Cloud Platform user might not have the resourcemanager.projects.setIamPolicy permission. You can check your permissions by going to the IAM page in the Google Cloud Platform Console and searching for your user id.

6. Change the Service account ID to a unique value that you will recognize so you can easily find this service account later if needed.
7. Click Furnish a new private key.
8. The default key type is JSON, which is the correct value to use.
9. Click Create.

   The private key file is downloaded to your machine. You can move it to another location. Keep the key file secure.

For more information about service accounts, see the Authentication documentation.

Using the Cloud SQL Proxy in a production environment

When you are using the Cloud SQL Proxy in a production environment, there are some steps you can take to ensure that the proxy provides the required availability for your application.

Ensure that the Cloud SQL Proxy is run as a persistent service

If the proxy process is terminated, all existing connections through it are dropped, and your application cannot create any more connections to the Cloud SQL instance with the proxy. To prevent this scenario, be sure to run the proxy as a persistent service, so that if the proxy exits for any reason, it is automatically restarted. This can be accomplished by using a service such as systemd, upstart, or supervisor. For the Windows operating system, run the proxy as a Windows Service. In general, the proxy should have the same uptime requirements as your application process.

How many copies of the Cloud SQL Proxy your application needs

There is no need to create a proxy process for every application process; many application processes can share a single proxy process. In general, you should run one proxy client process per workstation or virtual machine.

If you are using auto-scaling for virtual machines, ensure that the proxy is included in your virtual machine configuration, so that whenever a new virtual machine is started, it has its own proxy process.

It is up to you to manage how many connections your application requires, whether by limiting or pooling the connections. The proxy does not place any limitations on new connection rates or persistent connection count.

Reducing Cloud SQL Proxy output

If you need to reduce the size of the proxy log, you can do so by setting -verbose=false when you start the proxy. Keep in mind, however, that doing so will reduce the effectiveness of the proxy output in diagnosing connection issues.

Troubleshooting Cloud SQL Proxy connections

If you are having trouble connecting to your Cloud SQL instance using the Cloud SQL Proxy, here are a few things to try to find what's causing the problem.

• Check the proxy output.

  Often, the proxy output can help you determine the source of the problem and how to solve it. Pipe the output to a file, or watch the terminal where you started the proxy.

• If you are getting a 403 notAuthorized error, and you are using a service account to authenticate the proxy, make sure the service account has the correct permissions.

  You can check the service account by searching for its ID on the IAM page. It should have the cloudsql.instances.connect permission, for example by having the Cloud SQL Client role or the Legacy Editor role.

• Make sure the Cloud SQL API is enabled.

  If it is not, you will see output like Error 403: Access Not Configured in your proxy logs.

• If you are including multiple instances in your instances list, make sure you are using a comma as a delimiter.

• If you are attempting to connect from an application, connect using an administrative client first, to eliminate any issues with your application.

• If you are connecting using UNIX sockets, confirm that the sockets were created by listing the directory you provided when you started the proxy.

What's next

• Learn more about the Cloud SQL Proxy.
• Connect using the proxy from Google Compute Engine.
• Connect using the proxy from an external application.
• Connect a psql client using the proxy from a client machine.
• Connect a psql client using the proxy Docker client.
• Connect using the proxy from Google Container Engine.