Create virtual repositories

This page describes how to create Artifact Registry virtual repositories.

Virtual repositories act as a single access point to download, install, or deploy artifacts in the same format from one or more upstream repositories. An upstream repository can be a Artifact Registry standard or remote repository.

The other repository modes are:

Standard: The default repository mode. You upload or publish artifacts such as private packages directly to standard repositories. Although you can download directly from individual standard repositories, accessing groups of repositories with a virtual repository simplifies tool configuration.
Remote (language package repositories only): A pull through cache for artifacts in public repositories such as Maven Central or PyPI. It acts as a proxy for the public repositories so that you have more control over your external dependencies.

For more information on how virtual repositories work, see the Virtual repository overview.

Before you begin

Enable Artifact Registry, including enabling the Artifact Registry API and installing the latest version of Google Cloud CLI.

Required roles

To get the permissions that you need to manage repositories, ask your administrator to grant you the following IAM roles:

Create virtual repositories and grant access to individual repositories: Artifact Registry Repository Administrator (roles/artifactregistry.repoAdmin) on the Google Cloud project
Grant access to repositories at the project level: Project IAM Admin (roles/resourcemanager.projectIamAdmin) on the Google Cloud project
Grant access to repositories at the folder level: Folder Admin (roles/resourcemanager.folderAdmin) on the folder
Grant access to repositories at the organization level: Organization Admin (roles/resourcemanager.organizationAdmin) on the Google Cloud organization

For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

If upstream repositories are in other projects, an administrator in the projects with upstream repositories must have permissions to manage access to those repositories.

Create upstream repositories

Before setting up a virtual repository, create the upstream repositories that you want to make available with the virtual repository.

Create the upstream repositories in the same Google Cloud region where you plan to create the virtual repository.
- Create a standard repository
- Create a remote repository
Virtual repositories use the Artifact Registry Service Agent to authenticate to upstream repositories. You must grant the service agent read access to upstream repositories.

You can grant the Artifact Registry Reader role to the Artifact Registry service agent at the project level so that you do not need to grant access to individual upstream repositories.

In each project with upstream repositories, run the following command:
```
gcloud projects add-iam-policy-binding UPSTREAM_PROJECT_ID \
    --member serviceAccount:service-VIRTUAL-REPO-PROJECT_NUMBER@gcp-sa-artifactregistry.iam.gserviceaccount.com \
    --role roles/artifactregistry.reader
```
Where
- UPSTREAM_PROJECT_ID is the ID of the Google Cloud project with the upstream repository.
- VIRTUAL_REPO_PROJECT_NUMBER is the project number of the Google Cloud project for your virtual repository.
Alternatively, if you want to grant more narrowly-scoped role to the service agent, add the permission artifactregistry.repositories.readViaVirtualRepository to a custom role and then grant that role to the service agent.

For more information about granting permissions, see the access control documentation.

Create a virtual repository

When you create a repository, you must configure the following settings that cannot be changed after the repository is created:

Artifact format.
Repository mode, if multiple modes are available for the selected format.
Repository location.
Encryption with Google-managed keys or customer-managed encryption keys. Artifact Registry uses Google-managed encryption keys by default.

Artifact Registry enforces organization policy constraints that require CMEK to encrypt resources or limit which Cloud KMS keys can be used for CMEK protection.

Create a virtual repository using Google Cloud console

Open the Repositories page in the Google Cloud console.

Open the Repositories page
Click Create Repository.
Specify the repository name. For each repository location in a project, repository names must be unique.
Select the repository format.
Under Repository mode, select Virtual.
Under Virtual Repository Upstreams, click Add Upstream to add an upstream repository.
- An upstream repository can be a standard, remote, or virtual repository.
- An upstream repository must be in the same location as the virtual repository, but can be in a different Google Cloud project.
- An upstream repository must be the same artifact format as the virtual repository.
Specify a priority and policy name for each upstream repository. Higher priority values take precedence over lower values. Leave space between priority values so that there is flexibility to add upstream repositories with priorities that are in between existing values.
Under Location Type, choose the location for the repository:
1. Choose the location type: Region or Multi-Region. The list of locations changes to reflect your selection.
2. In the Region or Multi-region list, select a location.
For information about location types and supported locations, see Repository locations
Add a description for the repository. Descriptions help to identify the purpose of the repository and the kind of artifacts it contains.

Do not include sensitive data, since repository descriptions are not encrypted.
If you want to use labels to organize your repositories, click Add Label and enter the key-value pair for the label. You can add, edit, or remove labels after you create the repository.
In the Encryption section, choose the encryption mechanism for the repository.
- Google-managed key - Encrypt repository content with a Google-managed encryption key.
- Customer-managed key - Encrypt repository content with a key that you control through Cloud Key Management Service. For key setup instructions, see Setting up CMEK for repositories.
Click Create.

Artifact Registry creates the repository and adds it to the list of repositories.

After you have created the repository:

Grant access to the repository.
Configure Docker, package managers, and other third-party clients to authenticate to repositories.
- Container images: Docker, Helm
- Language packages: Java, Node.js, Python, Go
- Other: Kubeflow Pipeline templates

Create a virtual repository using gcloud CLI

To create a virtual repository, you define your upstream repositories in a policy file and then create the repository using the policy file.

Create an upstream policy file in JSON format with the following settings:
```
[{
"id" : "POLICY_ID",
"repository" : "UPSTREAM_REPOSITORY_RESOURCE_NAME",
"priority" : PRIORITY
}, {...}]
```
- POLICY_ID is a name for the policy. Each upstream repository in the policy must have a unique policy ID.
- UPSTREAM_REPO_RESOURCE_NAME is the name of the upstream repository in the format:
```
projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY
```
  - An upstream repository can be a standard, remote, or virtual repository.
  - An upstream repository must be in the same location as the virtual repository, but can be in a different Google Cloud project.
  - An upstream repository must be the same artifact format as the virtual repository.
- PRIORITY is an integer that indicates the priority for the upstream repository. Higher priority values take precedence over lower values.
The following example shows a policy file named policies.json with four upstream repositories. upstream1 has the highest value for priority, so Artifact Registry will search it first. upstream-test has the lowest priority value, so Artifact Registry will search it last. upstream2 and upstream3 have the same priority. If Artifact Registry finds a requested package in repositories with the same priority, it can serve the package from either repository.
```
[{
"id" : "upstream1",
"repository" : "projects/myproject/locations/us-west1/repositories/python-repo1",
"priority" : 100
}, {
"id" : "upstream2",
"repository" : "projects/myproject/locations/us-west1/repositories/python-repo2",
"priority" : 80
}, {
"id" : "upstream3",
"repository" : "projects/myproject/locations/us-west1/repositories/python-repo2",
"priority" : 80
}, {
"id" : "upstream-test",
"repository" : "projects/myproject/locations/us-west1/repositories/python-test-repo",
"priority" : 20
}]
```
This example also leaves space between the priority values. These gaps provide flexibility to add upstream repositories with priorities that are in between existing values. For example, you can add an upstream-3 repository with a priority of 70 so that Artifact Registry searches it before upstream-test.

Consider storing your upstream policy file in your source control system so that you can reuse it later to make changes to upstream repository settings and track historical changes.
In the directory with your upstream policy file, run the following command to create the virtual repository:
```
gcloud artifacts repositories create VIRTUAL-REPOSITORY-NAME \
    --project=PROJECT_ID \
    --repository-format=FORMAT \
    --mode=virtual-repository \
    --location=LOCATION \
    --description="DESCRIPTION" \
    --upstream-policy-file=UPSTREAM_POLICY_FILE
```
Replace the following:
- VIRTUAL-REPOSITORY-NAME is the name of the virtual repository.
- PROJECT_ID is the ID of the project where you are creating the virtual repository. If this flag is omitted, the current or default project is used.
- FORMAT is the format of the repository. The following values are available:
  - apt (private preview)
  - docker
  - maven
  - npm
  - python
  - yum (private preview)
- LOCATION is the regional or multi-regional location for the repository. You can omit this flag if you set a default. To view a list of supported locations, run the command:
```
gcloud artifacts locations list
```
- DESCRIPTION is an optional description of the repository. Do not include sensitive data, since repository descriptions are not encrypted.
- UPSTREAM_POLICY_FILE is a path to the upstream policy file.
For example the following command creates a virtual repository named virtual-repo for Python packages in the region us-west1 with settings in policies.json.
```
gcloud artifacts repositories create virtual-repo \
--repository-format=python \
--mode=virtual-repository \
--location=us-west1 \
--description="Python virtual repository" \
--upstream-policy-file=policies.json
```

Artifact Registry creates the virtual repository.

Create a virtual repository using Terraform

Use the google_artifact_registry_repository resource to create repositories. terraform-provider-google version 5.0.0 or newer is required.

If you are new to using Terraform for Google Cloud, see the Get Started - Google Cloud page on the HashiCorp website.

The following example defines the provider, a virtual repository with the Terraform resource name my-repo and an upstream repository with the Terraform resource name my-repo-upstream.

Docker

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "docker"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "docker"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

Where

PROJECT-ID is the Google Cloud project ID.
UPSTREAM_LOCATION is the upstream repository location. Note that the virtual repository and upstream must be in the same region or multi-region.
UPSTREAM_REPOSITORY_ID is the upstream repository ID.
UPSTREAM_REPOSITORY_DESCRIPTION is the optional description of the upstream repository.
LOCATION is the virtual repository location.
REPOSITORY_ID is the virtual repository ID.
DESCRIPTION is the optional description for the virtual repository. Do not include sensitive data, since repository descriptions are not encrypted.
KEY is the name of the Cloud Key Management Service key, if you are using customer-managed encryption keys (CMEK) for encryption. Omit this argument to use the default setting, Google-managed encryption keys.
PRIORITY is the optional priority value. The highest priority is 1. Entries with a greater priority value take precedence in the pull order.

Maven

If you do not specify a version policy, Artifact Registry creates a Maven repository that stores both snapshot and release versions of packages by default.

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "maven"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "maven"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

Where

PROJECT-ID is the Google Cloud project ID.
UPSTREAM_LOCATION is the upstream repository location. Note that the virtual repository and upstream must be in the same region or multi-region.
UPSTREAM_REPOSITORY_ID is the upstream repository ID.
UPSTREAM_REPOSITORY_DESCRIPTION is the optional description of the upstream repository.
LOCATION is the virtual repository location.
REPOSITORY_ID is the virtual repository ID.
DESCRIPTION is the optional description for the virtual repository. Do not include sensitive data, since repository descriptions are not encrypted.
KEY is the name of the Cloud Key Management Service key, if you are using customer-managed encryption keys (CMEK) for encryption. Omit this argument to use the default setting, Google-managed encryption keys.
PRIORITY is the optional priority value. The highest priority is 1. Entries with a greater priority value take precedence in the pull order.

To store snapshot and release versions in different repositories, specify a version policy for the repository using a maven_config block. This block supports the following settings:

version_policy sets the version policy with one of the following values:
- VERSION_POLICY_UNSPECIFIED: Store snapshot and release packages. This is the default setting.
- RELEASE: Store release packages only.
- SNAPSHOT: Store snapshot packages only.
allow_snapshot_overwrites configures a repository with a SNAPSHOT version policy to accept non-unique snapshots that overwrite existing versions in the repository.

The following example defines a Maven repository with a release version policy.

provider "google" {
project = "my-project"
}

resource "google_artifact_registry_repository" "my-repo" {
provider = google-beta

location = "us-central1"
repository_id = "my-repo"
description = "Maven repository"
format = "MAVEN"
maven_config {
  version_policy = "RELEASE"
}
}

npm

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "npm"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "npm"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

Where

PROJECT-ID is the Google Cloud project ID.
UPSTREAM_LOCATION is the upstream repository location. Note that the virtual repository and upstream must be in the same region or multi-region.
UPSTREAM_REPOSITORY_ID is the upstream repository ID.
UPSTREAM_REPOSITORY_DESCRIPTION is the optional description of the upstream repository.
LOCATION is the virtual repository location.
REPOSITORY_ID is the virtual repository ID.
DESCRIPTION is the optional description for the virtual repository. Do not include sensitive data, since repository descriptions are not encrypted.
KEY is the name of the Cloud Key Management Service key, if you are using customer-managed encryption keys (CMEK) for encryption. Omit this argument to use the default setting, Google-managed encryption keys.
PRIORITY is the optional priority value. The highest priority is 1. Entries with a greater priority value take precedence in the pull order.

Python

provider "google" {
    project = "PROJECT-ID"
}
resource "google_artifact_registry_repository" "my-repo-upstream" {
  location      = "UPSTREAM_LOCATION"
  repository_id = "UPSTREAM_REPOSITORY_ID"
  description   = "UPSTREAM_REPOSITORY_DESCRIPTION"
  format        = "python"
}

resource "google_artifact_registry_repository" "my-repo" {
  depends_on    = []
  location      = "LOCATION"
  repository_id = "REPOSITORY_ID"
  description   = "VIRTUAL_REPOSITORY_DESCRIPTION"
  format        = "python"
  kms_key_name = "KEY"
  mode          = "VIRTUAL_REPOSITORY"
  virtual_repository_config {
    upstream_policies {
      id          = "UPSTREAM_REPOSITORY_ID"
      repository  = google_artifact_registry_repository.my-repo-upstream.id
      priority    = PRIORITY
    }
  }
}

Where

PROJECT-ID is the Google Cloud project ID.
UPSTREAM_LOCATION is the upstream repository location. Note that the virtual repository and upstream must be in the same region or multi-region.
UPSTREAM_REPOSITORY_ID is the upstream repository ID.
UPSTREAM_REPOSITORY_DESCRIPTION is the optional description of the upstream repository.
LOCATION is the virtual repository location.
REPOSITORY_ID is the virtual repository ID.
DESCRIPTION is the optional description for the virtual repository. Do not include sensitive data, since repository descriptions are not encrypted.
KEY is the name of the Cloud Key Management Service key, if you are using customer-managed encryption keys (CMEK) for encryption. Omit this argument to use the default setting, Google-managed encryption keys.
PRIORITY is the optional priority value. The highest priority is 1. Entries with a greater priority value take precedence in the pull order.

Artifact Registry creates your repository. Run the following command to view a description of the repository:

gcloud artifacts repositories describe REPOSITORY \
    --location=LOCATION

After you have created the repository:

Grant access to the repository.
Configure Docker, package managers, and other third-party clients to authenticate to repositories.
- Container images: Docker, Helm
- Language packages: Java, Node.js, Python, Go
- OS packages: Debian, RPM
- Other: Kubeflow Pipeline templates

Update upstream repositories

You can change the list of upstream repositories or adjust the priority of upstream repositories.

Console

Open the Repositories page in the Google Cloud console.

Open the Repositories page
In the repository list, select the virtual repository and click Edit Repository.
In the Virtual Repository Upstream list, make the required changes. You can add and remove repositories or change the priority of repositories in the list
Click Save.

gcloud

Edit your upstream policy file with your changes. For example, you might want to add an upstream repository or adjust priority settings.

If you want to remove all upstream repositories, your policy file should contain an empty set of repositories.
```
[]
```
Run the following command to apply your changes to the virtual repository.
```
gcloud artifacts repositories update REPOSITORY \
    --upstream-policy-file=UPSTREAM_POLICY_FILE
```
Replace UPSTREAM_POLICY_FILE with the path to the upstream policy file.

For more information on the command, see the gcloud artifacts repositories update reference documentation.

Edit repository descriptions

You can change the repository description from Google Cloud console or the gcloud CLI.

Console

Open the Repositories page in the Google Cloud console.

Open the Repositories page
In the repository list, select the repository and click Edit Repository.
Edit the repository description and then click Save.

gcloud

To update the repository description, run the command:

gcloud artifacts repositories update REPOSITORY \
    --project=PROJECT] \
    --location=LOCATION \
    --description="DESCRIPTION"

Replace the following values:

REPOSITORY is the name of the repository. If you configured a default repository, you can omit this flag to use the default.
PROJECT is the Google Cloud project ID. If this flag is omitted, the current or default project is used.
LOCATION is a regional or multi-regional location. Use this flag to view repositories in a specific location. If you configured a default location, you can omit this flag to use the default.
DESCRIPTION is a description for the repository.