Download Python packages using direct repository access

Enterprise tier only: This feature is available only with the Security Command Center Enterprise tier.

After you integrate Assured OSS with Security Command Center, Assured Open Source Software packages are hosted in an Artifact Registry repository that is created in a project that you control.

This page explains how you can connect to the Artifact Registry repository for Assured OSS to directly access and download the Python packages.

This document applies to the Assured OSS paid tier only. For the free tier, see Download Python packages using direct repository access for the free tier.

Before you begin

1. Integrate Assured OSS with Security Command Center.

2. Validate connectivity to Assured OSS for the requested service accounts.

3. Install the latest version of the Google Cloud CLI.

4. If you have installed the Google Cloud CLI previously, make sure you have the latest version by running the command:

   gcloud components update
   

Set up authentication

You can authenticate to the Assured OSS Python package repository using one of the following methods:

• Authenticate with a keyring
• Authenticate with a service account key

The following sections describe how to set up these authentication methods.

Authenticate with keyring

To use the Python keyring to authenticate to the Artifact Registry, see authenticating with keyring and the information on the credential search order. We recommend that you use the Python keyring for authentication.

To set up the keyring for authentication, do the following:

1. Install the keyring library:

   pip install keyring
   

2. Install the Artifact Registry backend:

   pip install keyrings.google-artifactregistry-auth
   

3. List backends to confirm the installation:

   keyring --list-backends
   

   The list should include the following:

   • ChainerBackend(priority:10)
   • GooglePythonAuth(priority:9)

4. For information about setting up Application Default Credentials, see Set up authentication.

   This step ensures that the Assured OSS credential helper obtains your key when connecting with the repositories.

Authenticate with a service account key

Authenticate with a service account key when the application requires authentication with a username and a password.

Note: Service account keys are a security risk if not managed correctly. You should choose a more secure alternative to service account keys whenever possible. If you must authenticate with a service account key, you are responsible for the security of the private key and for other operations described by Best practices for managing service account keys. If you are prevented from creating a service account key, service account key creation might be disabled for your organization. For more information, see Managing secure-by-default organization resources.

Replace the package repository URL https://us-python.pkg.dev/PROJECT_ID/assuredoss-python/simple with the URL https://_json_key_base64:[e-mail]/PROJECT_ID/assuredoss-python/simple

Replace the following:

• Replace PROJECT_ID with the ID of the project that you selected when you set up Assured Open Source Software.

• Replace BASE64_KEY with the base64-encoding of entire service account JSON key file. To convert the entire service account JSON key file to base64-encoding use the following command:

  BASE64_KEY=$(cat KEY_FILE_LOCATION | base64)
  

  Replace KEY_FILE_LOCATION with location of service account JSON key file.

Install packages

The following instructions assume that you're using PyPI as the repository to download your dependencies. If you're using another repository, you must use different steps to download dependencies.

Install packages existing in Assured OSS

To specify the Assured OSS Python packages that you want to download, create two requirements.txt files. The sample files are the following:

• requirements-google.txt

  # Packages present in Google's Artifact Registry
  
  urllib3==1.26.11 --hash=sha256:1cffe1aa066363a75c856f261c8fce62d87f7c40ce0f46453ea12bf652b12a13
  jsonschema==4.13.0 --hash=sha256:29895bfe55b93b75552fbdd1e09aa0c82b7c1c9395d4f267e10c7d43cd31a74e
  

• requirements-pypi.txt

  # Packages present in Google's Artifact Registry are mentioned here so that pip
  # downloads their dependencies from PyPI.
  
  urllib3==1.26.11
  jsonschema==4.13.0
  
  # Below this comment, add any package version which you need but is NOT
  # present in Google's Artifact Registry and therefore needs to be downloaded from
  # PyPI.
  

Note: To generate the requirements-google.txt for your environment, see Generating the requirements-google.txt file.

To download the packages, run the following commands:

• To download the required packages which are present in the Artifact Registry for Assured OSS, run the following:

   pip install --require-hashes --requirement=requirements-google.txt --index-url https://_json_key_base64:[e-mail]/PROJECT_ID/assuredoss-python repo/simple  -v --no-deps
  

  Consider the following:

  • --require-hashes is optional. If included, hashes are specified for all packages and for all package versions in the requirements.txt file.
  • -v is optional. If specified, the command gives more output.

• To download the required packages which aren't present in the Artifact Registry for Assured OSS, run the following:

   pip install --requirement=requirements-pypi.txt --index-url https://pypi.org/simple -v
  

  This command also downloads the missing dependencies of the packages that you have downloaded using the previous command.

List all Python packages available in Assured OSS

To use an API to get a list of all the Python packages in the Artifact Registry repository, see List all Python packages available in Assured OSS.

Generating the requirements-google.txt file

This section provides additional information about generating the requirements-google.txt file that you need to download the Python packages. To download the hash and generate the requirements-google.txt file for your environment, you can use one of the following two options:

• Use a script to generate a single requirements file for all artifacts.

• Download a separate requirements file for each artifact.

The following sections provide more information about these options.

Option 1: Use a script to generate a single requirements file for all artifacts

Use the generator.sh script to generate a single requirements-google.txt file for all the Python packages (along with their hashes) available with Assured OSS for linux operating system. The packages must satisfy constraints such as Python version, machine architecture, and operating system. You can then delete the package versions which you don't require and use the resulting file.

The generator.sh script helps in the following two ways:

• It generates the most recent list of Python package versions available with Assured OSS that can be successfully installed on your system.
• It generates the requirements-google.txt file along with all the hashes.

The required script and its README.md file are available in a Cloud Storage bucket (gs://cloud-aoss/utils/python-requirements-txt/v1.0) which can be downloaded using the gsutil command line tool.

To download the script and the README.md file, follow these steps:

1. Authenticate with the service account to access the Cloud Storage bucket using the following command:

   gcloud auth activate-service-account --key-file KEY_FILE
   

   Replace KEY_FILE with the path to the file containing the credentials of the service account provided during the set up.

2. Download the generator.sh to your machine using the following command:

   gsutil cp -r gs://cloud-aoss/utils/python-requirements-txt/v1.0/generator.sh PATH_TO_LOCAL_STORE
   

   Replace PATH_TO_LOCAL_STORE with the local path where you want to save the downloaded file.

3. Download the README.md file using the following command:

   gsutil cp -r gs://cloud-aoss/utils/python-requirements-txt/v1.0/README.md PATH_TO_LOCAL_STORE
   

   Replace PATH_TO_LOCAL_STORE with the local path where you want to save the downloaded file. The README.md file contains instructions on how to use the script.

To run the script, use the following commands:

• To generate the requirements-google.txt file, run the following command:

   chmod +x generator.sh
   ./generator.sh
  

• To get the information of the package in a CSV file, run the following command:

  chmod +x generator.sh
  ./generator.sh -i
  

Option 2: Download requirements.txt for each required artifact

You can also download a separate requirements.txt (containing the hash) file for each Python artifact and then combine them into a single requirements.txt file.

Artifact hashes are available in a Cloud Storage bucket which can be downloaded using the gsutil command line tool. Hashes for each package and a version are located at the gs://cloud-aoss/python/PACKAGE_NAME/VERSION Cloud Storage bucket location.

To download the requirements.txt file, follow these steps:

1. Authenticate with the service account to access the Cloud Storage bucket using the following command:

   gcloud auth activate-service-account --key-file KEY_FILE
   

   Replace KEY_FILE with the path to the file containing the service account credentials.

2. Download the requirements.txt of a specific package and version to your local machine using the following command:

   gsutil cp -r gs://cloud-aoss/python/PACKAGE_NAME/VERSION PATH_TO_LOCAL_STORE
   

   Replace the following:

   • PACKAGE_NAME: the package name
   • VERSION: the version of the package
   • PATH_TO_LOCAL_STORE: the local path where you want to download the file

   Sample command:

   gsutil cp -r gs://cloud-aoss/python/bleach/5.0.0 /tmp/bleach

   Sample requirements.txt file:

   bleach==5.0.0 \
    --hash=sha256:6d286e765bfd3e309209cfa1d063e4d46afa966dea8cb97431c02b1e3067d812
   

   The contents of each such requirements.txt file can be combined into a single requirements-google.txt file.

What's next

• Download Java packages using direct repository access in the paid tier
• Access security metadata and verify packages in the paid tier