Google Cloud Platform

Exploring container security: Digging into Grafeas container image metadata

Editor’s note: This is the third in a series of blog posts on container security at Google.

The great thing about containers is how easy they are to create, modify and share. But that also raises the question of whether or not they're safe to deploy to production. One way to answer that is to track metadata about your container, for example, who worked on it, where it's stored, and whether it has any known vulnerabilities.

Last October, Google and several partners announced Grafeas, an open-source project that provides a structured metadata API for container images. Grafeas simplifies how you handle metadata, including security concerns like package vulnerability scan results, and keeps track of lots of different kinds of information:

  • "Build" metadata can be used to certify that a container image was built according to your build policies using a trusted builder with checked-in source code. 
  • "Package Vulnerability" metadata contains information from a vulnerability scanning service and lets you create policies on deploying vulnerable containers. 
  • "Image Basis" metadata contains information about the base image from which your container was derived as well as additional layers used in building the container image. 
  • "Package Manager" metadata indicates which packages were installed in the container image. 
  • "Deployment History" metadata allows you to track which resources were deployed and when.

Tracking Grafeas’ metadata lets you know what containers are in your environment, and also to enforce restrictions on which containers get deployed. You should review Grafeas metadata for compliance with your policies before deploying. This is where one last type of Grafeas metadata comes in—the mighty "Attestation." If the metadata you pull from Grafeas is consistent with your policy, you can then write an attestation that certifies that your image complies with your deployment requirements. Then, using the Kubernetes Admission Controller, you can check for the expected attestations, and block deployment when they aren’t present.

  occurrencesResponse := grafeas.ListOccurrencesResponse{}
 
imageUrl := "https://gcr.io/myproject/name@sha256hash"
hasFixableVulnerability := false
 
// Iterate through occurrences looking for vulnerability occurrences for this image
for _, occurrence := range occurrencesResponse.Occurrences {
  resourceUrl := occurrence.ResourceUrl
  if imageUrl != resourceUrl && occurrence.Kind != grafeas.Note_PACKAGE_VULNERABILITY {
    continue
  }
  details := occurrence.GetVulnerabilityDetails()
  issues := details.GetPackageIssue()
  // This vulnerability is fixable, we should not insert an attestation
  for _, i := range issues {
    if i.FixedLocation != nil {
      hasFixableVulnerability = true
    }   
  }
}
 
// If there are no fixable vulnerabilities, create an attestation occurrence
if !hasFixableVulnerability {
  occ := &grafeas.Occurrence{
    ResourceUrl: imageUrl,
    NoteName:    "projects/vulnerabilty-certification/notes/vuln-free",
    Details: &grafeas.Occurrence_AttestationDetails{
      AttestationDetails: &grafeas.AttestationAuthority_AttestationDetails{
        Signature: &grafeas.AttestationAuthority_AttestationDetails_PgpSignedAttestation{
          PgpSignedAttestation: &grafeas.PgpSignedAttestation{
            Signature:   "MySignature",
            ContentType: grafeas.PgpSignedAttestation_SIMPLE_SIGNING_JSON,
            KeyId:       "ThisIsMyKeyId",
          },
        },
      },
    },
  }
  req := grafeas.CreateOccurrenceRequest{Occurrence: occ}
}

In fact, you can use Grafeas to enforce all kinds of security policies. Check out the tutorial by Kelsey Hightower on how to use Grafeas to allow only container images signed by a specific private key. But it doesn’t stop there. For example, you can write policies to block container images with vulnerabilities from being deployed, ensure that all your deployed images are built from a base image sanctioned by your security team, or require images to go through your build pipeline. The possibilities are endless!

How Shopify uses Grafeas to manage metadata for its 500k container images


Big changes in infrastructure, like moving to Kubernetes for container orchestration, can create opportunities to change the way you govern your workloads. Here at Shopify, we use Grafeas as the central place to store all the metadata about our deploys, helping us build six thousand containers a day, and making our lives easier.

Grafeas lets us answer fundamental questions like:

When was the container created? What code runs inside it? Who created it? Where is the running container located? Is the container vulnerable to a known exploit and do we need to pull it out of production to replace it?

Shopify’s Continuous Integration/Deployment (CI/CD) pipeline supplies information about who built the code, when it was built, what the commit hash is, and whether or not tests passed. We store this data in the “build” metadata in Grafeas, leaving a digital paper trail that can be audited to track changes.

After build, we test the container for known vulnerabilities. Since that list of vulnerabilities can change at any time, we check our images at regular intervals. The information in Grafeas’ “package vulnerability” metadata field is updated at each check and allows us to make informed decisions about when to pull an image out of production and deploy a new secure one.

Next up, we’re working on how to track information about deploys in the deployment history metadata field, so that we know when, where and by whom a container was deployed, as well as its current status. We will use that information to create a “bill of materials” of what’s running in our environment.

The metadata in Grafeas gives us an answer to all of our questions around deploys. We also use it to restrict the deployment of containers. During the creation of a workload, the admission controller makes its decision by querying Grafeas for the container’s metadata — if it cannot find anything that violates predefined policies, for example the existence of known vulnerabilities, it allows the workload to be created.

Using Grafeas in production gives us a 360-degree view of our deploys so we can protect our infrastructure and maintain a history of what’s going on in our cloud.

Get involved in Grafeas 


We’re currently working toward a beta version of the Grafeas API spec. In addition, Grafeas is also working with other projects, such as the in-toto team, to allow in-toto to create attestations based on Grafeas metadata.

If you’d like to contribute, Grafeas is always looking for more contributors as we grow this community. Join us!