Create a broken-link checker

This document describes how to configure a periodic test of the the links contained in a URI by creating a synthetic monitor. You specify the options for the test, such as the origin URI, the number of links tested, and the number of retries, and then deploy a preconfigured Cloud Function. To support your troubleshoot and debug efforts, synthetic monitors save detailed information about each test, including screenshots. Screenshots let you view the exact response seen by the customers of your application.

To learn more about synthetic monitors, see About synthetic monitors.

About broken-link checkers

Each broken-link checker tests the links serially, and there is an overall synthetic timeout, which is configurable.

By default, a broken-link checker does the following:

• Searches the origin URI for HTML anchor elements with href attributes.
• Tests the first 10 links found on the origin URI.
• For each link, the checker issues a request and then waits at most 30 seconds for a response. When a response is received, the checker verifies that the HTTP response status is 200, which indicates a successful response. The checker doesn't perform retries.

You specify the origin URI. You can configure which HTML elements that the broken-link checker searches for, the maximum number of elements tested, the per-test timeout, and whether retries are performed. You can also configure broken-link checkers to wait for a selector to appear.

Broken-link checkers use the broken-links-ok template. The configuration for a broken-link checker is specified by the options object of the index.js file. If you create your checker by using the Google Cloud console, you're prompted for each configuration option and the Cloud Function is updated for you. However, if you use the Cloud Monitoring API or Terraform, then you must populate this object.

After you create a broken-link checker, to modify the configuration, update the options object and redeploy the Cloud Function.

Before you begin

1. To get the permissions that you need to view and modify synthetic monitors by using the Google Cloud console, ask your administrator to grant you the following IAM roles on your project:

   • Monitoring Editor (roles/monitoring.editor)
   • Cloud Functions Developer (roles/cloudfunctions.developer)

   For more information about granting roles, see Manage access.

   You might also be able to get the required permissions through custom roles or other predefined roles.

2. Enable the Cloud Monitoring API, Artifact Registry API, Cloud Build API, Cloud Functions API, Cloud Logging API, Pub/Sub API, and Cloud Run Admin API APIs.

   Enable the APIs

3. Verify that your Google Cloud project contains the default Compute Engine service account. This service account is created when you enable the Compute Engine API and has a name similar to [e-mail].

   In the navigation panel of the Google Cloud console, select IAM & Admin, and then select Service Accounts:

   Go to Service Accounts

   If the default Compute Engine service account doesn't exist, then click Create service account and complete the dialog.

4. Ensure that the default Compute Engine service account, or the service account that you created, has been granted the role of Editor (roles/editor).

   To view the roles granted to your service account, do the following:

   1. In the navigation panel of the Google Cloud console, select IAM:

      Go to IAM

   2. Select Include Google-provided role grants.
   3. If the service account used by your synthetic monitor isn't listed, or if it hasn't been granted a role that includes the permissions in the role of Cloud Trace Agent (roles/cloudtrace.agent), then grant this role to your service account.
5. Configure the notification channels that you want to use to receive notifications. We recommend that you create multiple types of notification channels. For more information, see Create and manage notification channels and Create and manage notification channels by API.

Create a broken-link checker

Console

When you create a synthetic monitor by using the Google Cloud console, a new Cloud Function (2nd gen) is deployed and the monitor for that Cloud Function is created. You can't create a synthetic monitor that monitors an existing Cloud Function.

1. Ensure that you've enabled the required APIs, that your project contains a default Compute Engine service account, and that this account has been granted the role of Editor (roles/editor). For more information, see Before you begin.

2. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Synthetic monitoring:

   Go to Synthetic monitoring

3. Select Create Synthetic Monitor.
4. For the template, select Broken link checker.
5. Enter a name for the synthetic monitor.

6. Optional: Update the Response Timeout, the Check Frequency, and add user-defined labels.

7. Configure the URI and the elements to test:

   Note: The values in the URI configuration fields are copied to the Options object in the index.js file when you click Create Function. After you click Create Function, to change the configuration, edit the Options object. For details, see Edit a synthetic monitor or Cloud Function.

   1. Click Origin URI, and enter a URI that you want tested. The value that you enter must be an HTTP or HTTPS endpoint. For example, you might enter https://mywebsite.example.com.

   2. Optional: In the Number of links to follow, update the maximum number of links that are tested. The default value of this field is 10.

   3. Optional: In the HTML element selector field, enter the HTML element that you want matched, as a comma separated list. The value that you enter is converted to a string and then passed to the Document: querySelectorAll() method.

      By default, this field is set to a, which matches anchors. You can enter values such as a, img, when you want to match both anchors and images.

   4. Optional: In the HTML attributes to follow field, enter the HTML attributes that you want matched. The comma separated values that you enter are individually passed to the getAttribute() method.

      By default, this field is set to href, which specifies the URI for the link. You can enter multiple attributes, for example, you can enter href, src. In this example, the code searches for the attribute href and then searches for the attribute src.

   5. Optional: Configure wait for selector, per-URI timeout, retries, and expected status codes:

      1. Click Show more options.

      2. To configure the broken-link checker to wait for a specific selector to appear in the URI before any links are scraped, enter the CSS selectors in the Wait for element selector field. The value that you enter is converted to a string and then passed to the page.waitForSelector() method.

         If the selector doesn't appear before the timeout expires, then the failure is recorded in the logs.

      3. Update the order in which links are selected for testing.

      4. Configure retries.

         By default, one request is sent to each link and if the initial request fails for any reason, for example, the command times out or the HTTP status code isn't 200, then the link is marked as failed.

         This field specifies the number of times the broken-link checker can issue an HTTP request to a link before marking the link as failed.

      5. Configure a timeout that applies to each URI. By default, this value is set to 30 seconds.

      6. To specify the expected status code and time out for a specific URI, click Add per-link option and complete the dialog.

8. Optional: Configure whether screenshots of responses are collected and saved. If you use the default settings, then screenshots aren't saved. If you enable screenshot collection, then you can collect screenshots for all tests or only for failing tests. Cloud Monitoring uses the following convention to name the Cloud Storage bucket:

   gcm-PROJECT_ID-synthetics-LOCATION
   

   In the previous expression:

   • PROJECT_ID: The ID of your Google Cloud project.
   • LOCATION: The location of your Cloud Storage bucket.

   You have the option to use an existing Cloud Storage bucket.

9. Review your configuration and ensure that it is correct and complete, and then create your Cloud Function:

   1. Click Create Function.

      The values in the URI configuration fields are copied to the Options object in the index.js file when you click Create Function. After you click Create Function, to change the configuration, edit the Options object.

   2. Enter a display name and select a region. Names must be unique within a region.

      Note: Don't include an underscore in the name of your Cloud Function.

   3. In the Runtime, build, connections and security settings section, do the following:

      • In the Connections tab, ensure that Allow all traffic is selected.

      • Review the default settings and update them when necessary.

        Note: The default settings for the allocated memory, CPU, and security might not be sufficient. We recommend that the memory be set to at least 2GiB.

      • In the Runtime service account field, select a service account.

   4. Click Apply function.

10. Configure the alerting policy:

    1. Optional: Update the alerting policy name and the duration of failure before notifications are sent.

    2. Add the notification channels.

11. Click Create.

    The Cloud Function that you defined is built and deployed as 2nd gen, and the synthetic monitor is created.

API

The process of creating a broken-link checker checker by using the Cloud Monitoring API is identical to the process of creating any other synthetic monitor. For information about using the Cloud Monitoring API to create a synthetic monitor, see Create a synthetic monitor, and select the Cloud Monitoring tab.

Note: Don't include an underscore in the name of your Cloud Function.

Broken-link checkers use the broken-links-ok template. The configuration for a broken-link checker is specified by the options object of the index.js file.

When the options.screenshot_options structure is defined, the broken-link checker collects screenshots and saves them to a Cloud Storage bucket. If the screenshot_options.storage_location field isn't defined or if the value is an empty string, then Monitoring creates a Cloud Storage bucket and screenshots are saved to that bucket. Monitoring uses the following convention to name the Cloud Storage bucket:

gcm-PROJECT_ID-synthetics-LOCATION

In the previous expression:

• PROJECT_ID: The ID of your Google Cloud project.
• LOCATION: The location of your Cloud Storage bucket.

Terraform

To learn how to apply or remove a Terraform configuration, see Basic Terraform commands. For more information, see the Terraform provider reference documentation.

The process of creating a broken-link checker checker by using Terraform is identical to the process of creating any other synthetic monitor. For information about using the Terraform to create a synthetic monitor, see Create a synthetic monitor, and select the Terraform tab.

Note: Don't include an underscore in the name of your Cloud Function.

Broken-link checkers use the broken-links-ok template. The configuration for a broken-link checker is specified by the options object of the index.js file.

When the options.screenshot_options structure is defined, the broken-link checker collects screenshots and saves them to a Cloud Storage bucket. If the screenshot_options.storage_location field isn't defined or if the value is an empty string, then Monitoring creates a Cloud Storage bucket and screenshots are saved to that bucket. Monitoring uses the following convention to name the Cloud Storage bucket:

gcm-PROJECT_ID-synthetics-LOCATION

In the previous expression:

• PROJECT_ID: The ID of your Google Cloud project.
• LOCATION: The location of your Cloud Storage bucket.

Explore results

For each execution, a broken-link checker does the following:

• Generates a table, where each row provides information about the testing of a specific URI. The summary information includes the target URI, latency, status, and the HTML element identifier. For example, this column lists a when an HTML anchor element is tested. When the row corresponds to the origin URI, the value of HTML element identifier is -.

• Collects metrics, trace data, and log data.

• Collects screenshots, when configured.

For more information about how to explore the collected data, see Explore synthetic monitor results.

Troubleshoot

This section provides information that you can use to help you troubleshoot your broken-link checkers.

Unable to edit the configuration of a broken-link checker

You created a broken-link checker by using the Google Cloud console, and you want to change the HTML elements that are tested, or you want to modify the URI timeout, retries, wait for selector, and per-link options. However, when you edit the broken-link checker, the Google Cloud console doesn't display the configuration fields.

To resolve this failure, do the following:

1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Synthetic monitoring:

   Go to Synthetic monitoring

2. Locate the synthetic monitor that you want to edit, click More options, and then select Edit.
3. Click Edit function.

4. Edit the options object in the index.js file, and then click Apply function.

   For information about the fields and syntax for this object, see broken-links-ok/index.js.

5. Click Save.

Google Cloud console displays that saves of screenshots fail

Note: A synthetic monitor that can't save screenshots continues to test the links contained in a URI.

You created a broken-link checker and configured it to save screenshots. However, the Google Cloud console is displaying one of the following warning messages along with more detailed information:

• InvalidStorageLocation
• StorageValidationError
• BucketCreationError
• ScreenshotFileUploadError

To resolve these failures, try the following:

• If you see the InvalidStorageLocation message, then verify the existence of the Cloud Storage bucket specified in the field named options.screenshot_options.storage_location.

• View the logs related to your Cloud Function. For more information, see Finding logs.

• Verify that the service account being used in the corresponding Cloud Function has an Identity and Access Management role that lets it create, access, and write to Cloud Storage buckets.

What's next

• Samples for synthetic monitors
• Manage synthetic monitors
• Explore synthetic monitor results