Accelerate database performance using disk cache

This page describes how to provision, set up, and use disk caching on AlloyDB Omni to increase performance of your AlloyDB Omni installation.

In addition to the standard PostgreSQL in-memory shared buffers, AlloyDB Omni disk cache enables storing buffers on fast storage such as solid-state drives (SSDs). Disk caching accelerates data retrieval in AlloyDB Omni installations with data directories located on slower storage.

Like PostgreSQL shared buffers, AlloyDB Omni disk cache is non-persistent, which means cached data is lost on restart.

By default, AlloyDB Omni disk cache uses all storage reported by the file system. You can define the amount of storage reserved for caching data using the omni_disk_cache_file_size parameter.

Note: To use the disk cache on the AlloyDB Omni Kubernetes operator, you don't need to provision a file system or mount disks. See Enable AlloyDB Omni disk cache for details on how to modify the AlloyDB Omni Kubernetes operator cluster's manifest to include the ultraFastCache feature.

Enable AlloyDB Omni disk cache

The steps you use to enable disk cache for AlloyDB Omni depend on whether you run AlloyDB Omni in a container or on a Kubernetes cluster.

Single-server

Provision disks and create a file system

For AlloyDB Omni disk cache, you create a file system on a disk or multiple disks and mount it inside a container with AlloyDB Omni. Additionally, you can use utilities like mdadm or lvm to pool capacity together using multiple disks and use any file system.

The following steps demonstrate using lvm and ext4 on a Ubuntu Compute Engine instance using NVMe SSDs.

1. Create a volume group from all available physical devices:

     nvme_prefix="STORAGE_PREFIX"
     nvme_list=$(ls "$nvme_prefix"*)
     sudo vgcreate VOLUME_GROUP ${nvme_list}

   Replace the following:

   • STORAGE_PREFIX: the prefix of the target local disks path that are attached to a virtual machine using the nonvolatile memory express (NVMe) interface—for example, on Google Cloud, the NVMe device paths always start with /dev/nvme0n.
   • VOLUME_GROUP: the name of a volume group where your SSDs are combined—for example, omni-disk-cache-volume.

2. To create a logical volume from the free capacity of the volume group from the preceding step, use the following command:

     sudo lvcreate -n LOGICAL_VOLUME -l 100%FREE VOLUME_GROUP

   Replace LOGICAL_VOLUME with the name of a logical volume that is treated as a partition by the LVM—for example, omni_disk_cache_device.

3. Create the ext4 file system on the logical volume. If needed, you can specify other ext4 options subject to data safety.

     sudo mkfs.ext4 /dev/VOLUME_GROUP/LOGICAL_VOLUME
   

4. To create a directory that serves as a mount point on the host machine and mount the file system, use the following command:

     sudo mkdir /OMNI_DISK_CACHE_DIRECTORY
     sudo mount /dev/VOLUME_GROUP/LOGICAL_VOLUME /OMNI_DISK_CACHE_DIRECTORY

   Replace OMNI_DISK_CACHE_DIRECTORY with the name of the directory or a path to the directory serving as a mount point—for example, omni_disk_cache_directory.

Mount cache directory inside AlloyDB Omni

Before enabling disk cache for AlloyDB Omni running in a container, you must mount the cache directory inside AlloyDB Omni.

For information about installing AlloyDB Omni from a Docker image and customizing it, see Customize your AlloyDB Omni installation.

To mount the OMNI_DISK_CACHE_DIRECTORY inside your Docker container running AlloyDB Omni, use the following command:

Docker

      sudo docker run --name CONTAINER_NAME \
        -e POSTGRES_PASSWORD=PASSWORD \
        -e PGDATA=/var/lib/postgresql/data/pgdata \
        -v DATA_DIR:/var/lib/postgresql/data \
        -v /OMNI_DISK_CACHE_DIRECTORY:/CACHE_DIRECTORY_PATH_INSIDE_CONTAINER \
        -d google/alloydbomni
      

Replace the following:

• CONTAINER_NAME: the name to assign the new AlloyDB Omni container—for example, my-omni.
• PASSWORD: the password for your PostgreSQL database root administrator.
• DATA_DIR: the file system path that you want AlloyDB Omni to use for its data directory.
• CACHE_DIRECTORY_PATH_INSIDE_CONTAINER: the cache directory inside the AlloyDB Omni container that maps to the mount point on the host machine—for example, based on the value of the cache directory inside the container, either /omni_disk_cache_directory, similar to OMNI_DISK_CACHE_DIRECTORY, or /disk/cache/inside/container.

Podman

      podman run --name CONTAINER_NAME \
        -e POSTGRES_PASSWORD=PASSWORD \
        -e PGDATA=/var/lib/postgresql/data/pgdata \
        -v DATA_DIR:/var/lib/postgresql/data \
        -v /OMNI_DISK_CACHE_DIRECTORY:/CACHE_DIRECTORY_PATH_INSIDE_CONTAINER \
        -d docker.io/google/alloydbomni
      

Replace the following:

• CONTAINER_NAME: the name to assign the new AlloyDB Omni container—for example, my-omni.
• PASSWORD: the password for your PostgreSQL database root administrator.
• CACHE_DIRECTORY_PATH_INSIDE_CONTAINER: the cache directory inside the AlloyDB Omni container that maps to the mount point on the host machine—for example, based on the value of the cache directory inside the container, either /omni_disk_cache_directory, similar to OMNI_DISK_CACHE_DIRECTORY, or /disk/cache/inside/container.

To grant full access permissions to the mounted OMNI_DISK_CACHE_DIRECTORY, use the following commands:

Docker

      sudo docker exec -it CONTAINER_NAME chown postgres:postgres /CACHE_DIRECTORY_PATH_INSIDE_CONTAINER
      sudo docker exec -it CONTAINER_NAME chmod -R a+rw  /CACHE_DIRECTORY_PATH_INSIDE_CONTAINER
      

Podman

      sudo podman exec -it CONTAINER_NAME chown postgres:postgres /CACHE_DIRECTORY_PATH_INSIDE_CONTAINER
      sudo podman exec -it CONTAINER_NAME chmod -R a+rw  /CACHE_DIRECTORY_PATH_INSIDE_CONTAINER
      

Enable AlloyDB Omni disk cache for AlloyDB Omni running in a container

To enable AlloyDB Omni disk cache for your database, set the appropriate Grand Unified Configuration (GUC) parameters after ensuring that the mounted cache directory is accessible from inside the Docker container.

1. To connect to the containerized AlloyDB Omni database as a superuser, use the following command:

   Docker

         sudo docker exec -it CONTAINER_NAME psql -h localhost -U postgres
         

   Podman

         sudo podman exec -it CONTAINER_NAME psql -h localhost -U postgres
         

2. To set the value of the parameters, use the following commands inside the AlloyDB Omni database:

           alter system set omni_disk_cache_enabled=on;
           alter system set omni_disk_cache_directory='/CACHE_DIRECTORY_PATH_INSIDE_CONTAINER';
         

3. By default, AlloyDB Omni uses all available space in the file system. You can override the default value using the omni_disk_cache_file_size parameter if needed.

         alter system set omni_disk_cache_file_size=SIZE_IN_MB;
         

4. For the caching configuration parameters change to take effect, restart your running container with AlloyDB Omni:

   Docker

         sudo docker restart CONTAINER_NAME
         

   Podman

         sudo podman restart CONTAINER_NAME
         

Kubernetes

Enable disk cache on a generic volume

You can enable disk cache using a generic volume.

For enabling disk cache on a generic volume on the AlloyDB Omni Kubernetes operator, you need a persistent volume ready ahead of time and a storageClass.

For example, if you use GKE and don't have the persistent volume and storageClass ready, make sure you have performed the following before enabling disk cache on a generic volume:

1. Created a cluster with local SSD-based storage.
2. Formatted your volume to the ext4 file system using step 1 of Run the local volume static provisioner.
3. Manually created a persistent volume for each SSD in your cluster with the storageClass that defines persistent storage on a storage device.

To enable disk cache on a generic volume for your database, complete the following steps:

1. Modify your database cluster manifest to add the ultraFastCache attribute to the features section of the spec section:

       apiVersion: alloydbomni.dbadmin.goog/v1
       kind: DBCluster
       metadata:
         name: CLUSTER_NAME
       spec:
         databaseVersion: "15.7.0"
         primarySpec:
           features:
             ultraFastCache:
               cacheSize: DISK_CACHE_SIZE
               genericVolume:
                 storageClass: "STORAGE_CLASS_NAME"
        ...
         

   Replace the following:

   • DB_CLUSTER_NAME: the name of your database cluster. It is the same database cluster name you declared when you created it.
   • DISK_CACHE_SIZE: the size of the cache—for example, 100Gi. It must be greater than shared_buffers. This field is optional. If you don't specify the value of this field, AlloyDB Omni uses all space left on the disk, which applies to both AlloyDB Omni in a container and on a Kubernetes cluster.
   • STORAGE_CLASS_NAME: the name of the storage class of the ultra fast cache volume—for example, local-storage.
2. Reapply the manifest.

Enable disk cache on a local volume

If you want to use a local volume, you don't have to create a persistent volume. You can use the following optimization instead.

For example, if you use GKE and don't have the persistent volume and storageClass ready, make sure you have performed the following before enabling disk cache on a local volume:

1. Created a cluster with local SSD-based storage.
2. Formatted your volume to the ext4 file system using step 1 of Run the local volume static provisioner.

To enable disk cache on a local volume for your database, do the following:

1. Modify your database cluster manifest to add the ultraFastCache attribute to the features section of the spec section:

       apiVersion: alloydbomni.dbadmin.goog/v1
       kind: DBCluster
       metadata:
         name: CLUSTER_NAME
       spec:
         databaseVersion: "15.7.0"
         primarySpec:
           features:
             ultraFastCache:
               cacheSize: DISK_CACHE_SIZE
               localVolume:
                 path: "LOCAL_VOLUME_PATH"
                 nodeAffinity:
                   required:
                     nodeSelectorTerms:
                     - matchExpressions:
                       - key: "LABEL_KEY"
                         operator: "OPERATOR_VALUE"
                         values:
                         - "LABEL_KEY_VALUE"
       ...
         

   Replace the following:

   • CLUSTER_NAME: the name of your database cluster. It is the same database cluster name you declared when you created it.
   • DISK_CACHE_SIZE: the size of the cache—for example, 100Gi. It must be greater than shared_buffers. This field is optional. If you don't specify the value of this field, AlloyDB Omni uses all space left on the disk, which applies to both AlloyDB Omni in a container and on a Kubernetes cluster.
   • STORAGE_CLASS_NAME: the name of the storage class.
   • LOCAL_VOLUME_PATH: the path to the local volume—for example, /mnt/disks/raid/0.
   • LABEL_KEY: the node's label for the key that serves as a location indicator and facilitates even Pod distribution across the cluster—for example, cloud.google.com/gke-local-nvme-ssd.
   • OPERATOR_VALUE: the key's relationship to a set of values—for example, In. Set the parameter to one of the following:
     • In: the values array must be non-empty.
     • NotIn: the values array must be non-empty.
     • Exists: the values array must be empty.
     • DoesNotExist: the values array must be empty.
     • Gt: the values array must have a single element, which is interpreted as an integer.
     • Lt: the values array must have a single element, which is interpreted as an integer.
   • LABEL_KEY_VALUE: the value for your label key—for example, true. Set the parameter to an array of string values as follows:
     • If the operator is In or NotIn, the values array must be non-empty.
     • If the operator is Exists or DoesNotExist, the values array must be empty.
     • If the operator is Gt or Lt, the values array must have a single element, which is interpreted as an integer.
2. Reapply the manifest.

Verify the disk cache configuration

After enabling AlloyDB Omni disk cache, verify that the disk cache is accessed by monitoring read and write activity to the disks using available utilities like iotop or iostat.

Additionally, you can check if the AlloyDB Omni disk cache is open.

The steps you use to verify the disk cache configuration for AlloyDB Omni depend on whether you run AlloyDB Omni in a container or on a Kubernetes cluster.

Single-server

Docker

sudo docker logs CONTAINER_NAME 2>&1 | grep "opened omni disk cache"

Podman

sudo podman logs CONTAINER_NAME 2>&1 | grep "opened omni disk cache"

Kubernetes

  kubectl exec -i $DATABASE_POD -c database -n $DBCLUSTER_NAMESPACE -- cat /obs/diagnostic/postgresql.log | grep "opened omni disk cache"

If your disk caching is configured correctly, the Successfully opened omni disk cache ... message displays in the logs.

What's next

• About automatic memory management
• About local SSD disks
• Creating and starting a VM instance