Download Python packages using direct repository access

Stay organized with collections Save and categorize content based on your preferences.

This page explains how you can connect to the Assured OSS repository directly to access and download the Python packages.

Set up authentication

You can authenticate to the Artifact Registry Python package repository using one of the following options:

Authenticating with keyring

For information about using the Python keyring to authenticate to the Artifact Registry, see authenticating with keyring. See also, the information on the credential search order.

To set up the keyring, do the following:

  1. Install the keyring library using the command:

    pip install keyring
    
  2. Install the Artifact Registry backend using the command:

    pip install keyrings.google-artifactregistry-auth
    
  3. List backends to confirm the installation using the command:

    keyring --list-backends`
    

    The list should include:

    • ChainerBackend(priority:10)
    • GooglePythonAuth(priority:9)
  4. After you have set up the keyring, use a service account for authentication. Assign the service account key file location to the variable GOOGLE_APPLICATION_CREDENTIALS so that the Assured OSS credential helper can obtain your key when connecting with repositories.

    export GOOGLE_APPLICATION_CREDENTIALS=`KEY_FILE_LOCATION`
    

Where KEY_FILE_LOCATION is the path to the service account json key file.

Authenticating with a service account key

This step describes how to authenticate with a service account key when the application requires authentication with a username and a password.

Replace the package repository URL:

https://us-python.pkg.dev/cloud-aoss/cloud-aoss-python/simple

with the URL

https://_json_key_base64:BASE64_ENCODED_KEY_FILE_CONTENT@us-python.pkg.dev/ cloud-aoss/cloud-aoss-python/simple

Where

  • BASE64_ENCODED_KEY_FILE_CONTENT is the base64-encoded key in your service account key file.

Install Packages

This section assumes that you're using PyPI as the repository for downloading your dependencies. If you're using some other repository, you may need to change the steps as per your requirement.

Step 1. You require two requirements.txt files to download and use the Assured OSS Python packages. Here are the sample files for both.

requirements-google.txt

# Packages present in Google's artifact registry

urllib3==1.26.8 --hash=sha256:c96c2bc833aa80529ecd50f589656f4c091631a7baa2bd3f3a32c9f6d173f8f5
jsonschema==4.4.0 --hash=sha256:6bf79e8dec0a5b48d9638e7351f714ef14a04da71aa9b9739eeb417f563a7ba2

requirements-pypi.txt

# Packages present in Google's artifact registry are mentioned here so that pip
# downloads their dependencies from PyPI.

urllib3==1.26.8
jsonschema==4.4.0

# Below this comment, please add any package version which you need but is NOT
# present in Google's artifact registry and hence needs to be downloaded from
# PyPI.

To generate the requirements-google.txt for your environment, refer to the Additional information section in this page.

Step 2. After you have the two requirements.txt files, run the following commands:

  • Command to download the required packages which are present in the Assured OSS Artifact Registry.

    pip install --require-hashes --requirement=requirements-google.txt --index-url
     https://us-python.pkg.dev/cloud-aoss/cloud-aoss-python/simple  -v --no-deps`
    

    Where

    • --require-hashes is optional. If included, hashes are specified for all packages and for all package versions in the requirements.txt file.
    • -v is optional. If specified, the command gives more output.
  • Command to download the required packages which are not present in the Assured OSS Artifact Registry. This command will also download the missing dependencies of the packages that you have downloaded using the earlier command.

    pip install --requirement=requirements-pypi.txt --index-url https://pypi.org/simple -v`
    

List all Python packages available in Assured OSS

Use the following API to list all Python packages and their versions.

GET https://artifactregistry.googleapis.com/v1/{parent=projects/*/locations/*/repositories/*}/pythonPackages

Authentication to the API is set up using the same service account that has been created to access Assured OSS. For information, see Authentication at Google. You can also provide authentication credentials to the application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS using the following command:

export GOOGLE_APPLICATION_CREDENTIALS=`KEY_FILE_LOCATION`

Where KEY_FILE_LOCATION is the path to the service account json key file.

HTTP request

GET https://artifactregistry.googleapis.com/v1/{parent=projects/*/locations/*/repositories/*}/pythonPackages\

You can also call this API using curl command. The following command is a sample curl command to connect to the Assured OSS Python repository:

curl -X GET -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) https://artifactregistry.googleapis.com/v1/projects/cloud-aoss/locations/us/repositories/cloud-aoss-python/pythonPackages

Sample response:

{
  "pythonPackages": [
    {
      "name": "projects/cloud-aoss/locations/us/repositories/cloud-aoss-python/pythonPackages/Flask:2.1.2",
      "uri": "us-python.pkg.dev/cloud-aoss/cloud-aoss-python/flask/Flask-2.1.2-py3-none-any.whl",
      "packageName": "Flask",
      "version": "2.1.2",
      "createTime": "2022-07-13T11:06:54.163313Z",
      "updateTime": "2022-07-13T11:06:54.163313Z"
    },
    {
      "name": "projects/cloud-aoss/locations/us/repositories/cloud-aoss-python/pythonPackages/ujson:5.3.0",
      "uri": "us-python.pkg.dev/cloud-aoss/cloud-aoss-python/ujson/ujson-5.3.0-cp38-cp38-linux_x86_64.whl",
      "packageName": "ujson",
      "version": "5.3.0",
      "createTime": "2022-07-13T11:06:17.263638Z",
      "updateTime": "2022-07-13T11:06:17.263638Z"
    },
    {
      "name": "projects/cloud-aoss/locations/us/repositories/cloud-aoss-python/pythonPackages/ujson:5.4.0",
      "uri": "us-python.pkg.dev/cloud-aoss/cloud-aoss-python/ujson/ujson-5.4.0-cp38-cp38-linux_x86_64.whl",
      "packageName": "ujson",
      "version": "5.4.0",
      "createTime": "2022-07-13T11:09:00.865162Z",
      "updateTime": "2022-07-13T11:09:00.865162Z"
    },
    {
      "name": "projects/cloud-aoss/locations/us/repositories/cloud-aoss-python/pythonPackages/urllib3:1.26.8",
      "uri": "us-python.pkg.dev/cloud-aoss/cloud-aoss-python/urllib3/urllib3-1.26.8-py2.py3-none-any.whl",
      "packageName": "urllib3",
      "version": "1.26.8",
      "createTime": "2022-07-13T11:05:56.529484Z",
      "updateTime": "2022-07-13T11:05:56.529484Z"
    }
  ]
}

Additional information

This section provides additional information on generating the requirements-google.txt file that you require to download and use the Python packages. To download the hash and generate the requirements-google.txt file for your environment, you can use one of the following two options:

Option 1: Use a script to generate a single requirements file for all artifacts

Use the requirements_txt_generator.sh script to generate a single requirements-google.txt file for all the Python packages (along with their hashes) available with Assured OSS. The packages must satisfy constraints such as Python version, machine architecture, and operating system. You can then delete the package versions which you don't require and use the resulting file.

The requirements_txt_generator.sh script helps in the following two ways:

  • It generates the most recent list of Python package versions available with Assured OSS that can be successfully installed on your system.
  • It also generates the requirements-google.txt file along with all the hashes.

The required script and its README.md are available in a Cloud Storage bucket (gs://cloud-aoss/utils/python-requirements-txt/v1.0) which can be downloaded using the gsutil command line tool.

To download the script and the README.md file, use the following steps:

  1. Authenticate with the service account to access the Cloud Storage bucket using the following command:

    gcloud auth activate-service-account --key-file `KEY_FILE`
    

    Where KEY_FILE is the path to the file containing the service account credentials.

  2. Download the requirements_txt_generator.sh to your machine using the following command:

    gsutil cp -r gs://cloud-aoss/utils/python-requirements-txt/v1.0/generator.sh `PATH_TO_LOCAL_STORE`
    

    Where, PATH_TO_LOCAL_STORE is the local path where you want to save the downloaded file.

  3. Download the README.md using the following command:

    gsutil cp -r gs://cloud-aoss/utils/python-requirements-txt/v1.0/README.md `PATH_TO_LOCAL_STORE`
    

    Where, PATH_TO_LOCAL_STORE is the local path where you want to save the downloaded file. The README.md file contains instructions on how to use the script.

To run the script, use the following commands:

  • To generate the requirements-google.txt file use command:

     chmod +x generator.sh
     ./generator.sh
    
  • To list the package names satisfying the given criteria, use command:

    chmod +x generator.sh
    ./generator.sh -p
    
  • To get the information of the package in a CSV file, use command:

    chmod +x generator.sh
    ./generator.sh -i
    

Option 2: Download requirements.txt for each required artifact

You can also download a separate requirements.txt (containing the hash) file for each Python artifact and then combine them into a single requirements.txt file.

Artifact hashes are available in a Cloud Storage bucket which can be downloaded using the gsutil command line tool. Hashes for each package and a version are located at the gs://cloud-aoss/python/PACKAGE_NAME/VERSION Cloud Storage bucket location.

To do this:

  1. Authenticate with the service account to access the Cloud Storage bucket using the following command:

    gcloud auth activate-service-account --key-file KEY_FILE`
    

    Where KEY_FILE is the path to the file containing the service account credentials.

  2. Download the requirements.txt of a specific package and version to your local machine using the following command:

    gsutil cp -r gs://cloud-aoss/python/PACKAGE_NAME/VERSION PATH_TO_LOCAL_STORE`
    

    Where

    • PACKAGE_NAME is the package name
    • VERSION is the version of the package
    • PATH_TO_LOCAL_STORE is the local path where you want to download the file

    Sample command:

    gsutil cp -r gs://cloud-aoss/python/sqlalchemy/1.3.24 /tmp/sqlalchemy

    Alternatively, you can download the requirements.txt for all packages using the following command:

    gsutil -m cp -r gs://cloud-aoss/python PATH_TO_LOCAL_STORE

    Where PATH_TO_LOCAL_STORE is the local store path.

    Sample requirements.txt file:

    bleach==4.1.0 --hash=sha256:5b337d703954b3d68d2e5745b83622db36bbba9b83005d12dffeeb03a2a41f35
    

    The contents of each such requirements.txt file can be combined into a single requirements-google.txt file.

Learn more

Assured Open Source Software is part of the Software Delivery Shield solution. Software Delivery Shield is a fully-managed, end-to-end software supply chain security solution that helps you to improve the security posture of developer workflows and tools, software dependencies, CI/CD systems used to build and deploy your software, and runtime environments such as Google Kubernetes Engine and Cloud Run. To learn how you can use Assured Open Source Software with other components of Software Delivery Shield to improve the security posture of your software supply chain, see Software Delivery Shield overview.

What's next?