Accessing private GitHub repositories

This tutorial demonstrates how to use Secret Manager with Cloud Build to interact with private GitHub repositories. Secret Manager is a Google Cloud service that securely stores API keys, passwords, and other sensitive data.

Objectives

  • Set up a GitHub SSH key.
  • Add the public SSH key to a private repository's deploy keys.
  • Store the private SSH key in Secret Manager.
  • Submit a build that accesses the key from Secret Manager and uses it to access the private repository.

Costs

This tutorial uses billable components of Google Cloud, including:

  • Secret Manager
  • Cloud Build

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. In the Cloud Console, on the project selector page, select or create a Cloud project.

    Go to the project selector page

  3. Zorg dat facturering is ingeschakeld voor uw project.

    Meer informatie over het inschakelen van facturering

  4. Enable the Cloud Build and Secret Manager APIs.

    Enable the APIs

  5. Install and initialize the Cloud SDK.
  6. Optional. Complete the Secret Manager quickstart to become familiar with this product.

Create a SSH key

  1. Open a terminal window.

  2. Create a new directory named workingdir and navigate into it:

    mkdir workingdir
    cd workingdir
    
  3. Create a new GitHub SSH key, where github-email is your GitHub email address:

    ssh-keygen -t rsa -b 4096 -C github-email
    
  4. When you're prompted to enter a file in which to save the key, enter id_github. This creates a new SSH key workingdir/id_github.

  5. At the prompt to type a passphrase, leave it empty and press Enter. This doesn't set a passphrase for your SSH key. Cloud Build cannot use your SSH key if it is protected with a passphrase.

Store the private SSH key in Secret Manager

When you create an SSH key, an id_github file is created in your environment. Because anyone can authenticate to your account with this file, you must store the file in Secret Manager before using it in a build.

  1. To store your SSH key in Secret Manager:

    1. Go to the Secret Manager page in the Cloud Console:

      Go to the Secret Manager page

    2. On the Secret Manager page, click Create Secret.

    3. On the Create secret page, under Name, enter secret-name.

    4. In the Secret value field, click Upload and upload your workingdir/id_github file.

    5. Leave the Regions section unchanged.

    6. Click the Create secret button.

This will upload your id_github file to Secret Manager.

Add the public SSH key to your private repository's deploy keys

  1. Login to GitHub.

  2. In the upper-right corner, click your profile photo, then click Your profile.

  3. On your profile page, click Repositories, then click the name of your repository.

  4. From your repository, click Settings.

  5. In the sidebar, click Deploy Keys, then click Add deploy key.

  6. Provide a title, paste your public SSH key from workingdir/id_github.pub.

  7. Select Allow write access if you want this key to have write access to the repository. A deploy key with write access lets a deployment push to the repository.

  8. Click Add key.

  9. Delete the SSH key from your disk:

    rm id_github*
    

Grant permissions

You need to grant the Cloud Build service account permission to access Secret Manager during the build.

  1. Open the Cloud IAM page in the Cloud Console:

    Open the Cloud IAM page

  2. Select your project and click Open.

  3. In the permissions table, locate the email ending with @cloudbuild.gserviceaccount.com, and click on the pencil icon.

  4. Add Secret Manager Secret Accessor role.

  5. Click Save.

Add the public SSH key to known hosts

Most machines contain a file named known_hosts, which contains known keys for remote hosts. The keys are often collected from the remote hosts when connecting to them for the first time, but they can also be added manually. The keys in this file are used to verify the identity of the remote host and protect against impersonation.

For Cloud Build to connect to GitHub, you must add the public SSH key to the known_hosts file in Cloud Build's build environment. You can do this by adding the key to a temporary known_hosts.github file, and then copying the contents of known_hosts.github to the known_hosts file in Cloud Build's build environment.

In your workingdir directory, create a file named known_hosts.github and add the public SSH key to this file:

ssh-keyscan -t rsa github.com > known_hosts.github

In the next section when you configure the build, you'll add instructions in the Cloud Build config file to copy the contents of known_hosts.github to the known_hosts file in Cloud Build's build environment.

Configure the build

To configure the build:

  1. Create a build config file named cloudbuild.yaml with three steps: the first gcloud step accesses the SSH key in Secret Manager and saves it in a volume named ssh. The volume is used to persist files across the build steps. The second git step configures git to use the SSH key by copying the key from known_hosts.github to known_hosts. The third git step uses the key in known_hosts to connect to the repository at git@github.com:git-username/git-repository.

    # Access the id_github file from Secret Manager
    steps:
    - name: gcr.io/cloud-builders/gcloud
      entrypoint: 'bash'
      args: [ '-c', 'gcloud secrets versions access latest --secret=secret-name > /root/.ssh/id_github' ]
      volumes:
      - name: 'ssh'
        path: /root/.ssh
    
    # Set up git with key and domain
    - name: 'gcr.io/cloud-builders/git'
      entrypoint: 'bash'
      args:
      - '-c'
      - |
        chmod 600 /root/.ssh/id_github
        cat <<EOF >/root/.ssh/config
        Hostname github.com
        IdentityFile /root/.ssh/id_github
        EOF
        mv known_hosts.github /root/.ssh/known_hosts
      volumes:
      - name: 'ssh'
        path: /root/.ssh
    
    # Connect to the repository
    - name: 'gcr.io/cloud-builders/git'
      args:
      - clone
      - git@github.com:git-username/git-repository
      volumes:
      - name: 'ssh'
        path: /root/.ssh
    

Submit the build

To submit the build, run the following command:

gcloud builds submit --config=cloudbuild.yaml .

The output is similar to the following:

Creating temporary tarball archive of 3 file(s) totalling 4.1 KiB before compression.
Uploading tarball of [.] to [gs://[PROJECT-ID]_cloudbuild/source/1504288639.02---.tgz]
Created [https://cloudbuild.googleapis.com/v1/projects/[PROJECT-ID]/builds/871b68bc---].
Logs are available at [https://console.cloud.google.com/gcr/builds/871b68bc---?project=[PROJECT-ID]].
----------------------------- REMOTE BUILD OUTPUT ------------------------------
starting build "871b68bc-cefc-4411-856c-2a2b7c7d2487"

FETCHSOURCE
Fetching storage object: gs://[PROJECT-ID]_cloudbuild/source/1504288639.02---.tgz#1504288640827178
Copying gs://[PROJECT-ID]_cloudbuild/source/1504288639.02---.tgz#1504288640827178...
/ [1 files][  3.9 KiB/  3.9 KiB]
Operation completed over 1 objects/3.9 KiB.
BUILD
Step #0: Already have image (with digest): gcr.io/cloud-builders/gcloud
Starting Step #0
Finished Step #0
Step #1: Already have image (with digest): gcr.io/cloud-builders/git
Starting Step #1
Step #1: # github.com SSH-2.0-libssh_0.7.0
Finished Step #1
Step #2: Already have image (with digest): gcr.io/cloud-builders/git
Starting Step #2
Step #2: Cloning into '[REPOSITORY-NAME]'...
Step #2: Warning: Permanently added the RSA host key for IP address 'XXX.XXX.XXX.XXX' to the list of known hosts.
Finished Step #2
PUSH
DONE
-----------------------------------------------------------------------------------------------------------------

ID                                    CREATE_TIME                DURATION  SOURCE                                                                              IMAGES  STATUS
871b68bc-cefc-4411-856c-2a2b7c7d2487  XXXX-XX-XXT17:57:21+00:00  13S       gs://[PROJECT-ID]_cloudbuild/source/1504288639.02---.tgz  -                                 SUCCESS

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.

Delete the deploy key from your repository

  1. On GitHub, navigate to the main page of the repository.

  2. Under your repository name, click Settings.

  3. In the left sidebar, click Deploy keys.

  4. On the Deploy keys page, look for the deploy keys associated with your repository and click Delete.

What's next