Installing the Cloud Logging agent on a single VM

Key Point: Now you can install all of the Google Cloud's operations suite agents across your entire fleet of Compute Engine Linux VMs in a single step. With one command, you can create a policy that governs new and existing VMs, ensuring proper installation and optional auto-upgrade of the agents. For more information, refer to the Managing Agent Policies guide.

The Logging agent streams logs from your VM instances and from selected third-party software packages to Cloud Logging. It is a best practice to run the Logging agent on all your VM instances.

The VM images for Compute Engine and Amazon Elastic Compute Cloud (EC2) don't include the Logging agent, so you must complete these steps to install it on those instances. The agent runs under both Linux and Windows.

If your VMs are running in Google Kubernetes Engine or App Engine, the agent is already included in the VM image, so you can skip this page.

If you are running specialized logging workloads that require higher throughput and/or improved resource-efficiency compared to the standard Cloud Logging agent, consider using the Ops agent.

Before you begin

To install the agent, ensure that you have the following:

• A supported VM instance in a Google Cloud project or Amazon Web Services (AWS) account.

  • When installing the Logging agent, a minimum of 250 MiB memory is required, but 1 GiB is recommended.

  Also ensure your VM is running a supported operating system.

• A Workspace monitoring the AWS account containing the VM instance. For pricing information, go to Pricing for Google Cloud's operations suite.

• Credentials on the VM instance that authorize communication with Cloud Logging or Cloud Monitoring. Compute Engine VM instances generally have the correct credentials by default. If either of the following scenarios applies to you, then you might not have the proper credentials and must complete the Authorizing the agent procedures:

  • Running AWS EC2 VM instances, you must install authorization credentials on your VMs before installing the agent.

  • Running very old Compute Engine instances or Compute Engine instances created without the default credentials.

  To check if you have the proper credentials, run the Verifying Compute Engine credentials procedures.

Google Cloud projects for AWS EC2 VM instances

When the documentation refers to the Google Cloud project associated with your VM instance, for EC2 VM instances, this phrase refers to the AWS connector project linked to your AWS account.

When you connect your AWS account to a Workspace, the AWS connector project is created. To identify the AWS connector project, go to the Workspace's Settings tab. The connector project are listed under the heading AWS Accounts. The Project ID column displays the Google Cloud projects associated with your Workspace.

Refer to the Workspace page for more information about AWS connector projects.

Note: Don't install the agent on an EC2 VM instance until you have connected your AWS account to a Workspace. Until the connection is made, there is no associated Google Cloud project and therefore nowhere to send data.

VMs without remote package access

Installing the Logging agent requires access to remote package repositories, for both the agent package and (on Linux) its dependencies.

If you are using VPC-SC or a private network, the network configuration might also affect your ability to install agent dependencies from upstream repositories. The agent packages themselves are accessible by using Private Google Access.

If your VM host's security policy denies access to remote package repositories, we recommend creating a custom VM image with the agent pre-installed and disabling package management in that image.

Installing the agent on a single VM

To install the agent, use the following instructions.

AMAZON LINUX AMI / CENTOS / RHEL

1. Open a terminal connection to your VM instance using SSH or a similar tool and ensure you have sudo access.

2. Change to a directory you have write access to, for example your home directory.

3. Add the agent's package repository:

   curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
   sudo bash add-logging-agent-repo.sh
   

4. Install the agent:

   1. List the available versions of the agent in order to select which version to install:

      sudo yum list --showduplicates google-fluentd
      

   2. For production environments, you might want to pin to a major version to avoid pulling in major versions that might include backward incompatible changes. To pin to a major version, run:

      sudo yum install -y google-fluentd-major-version.*
      

      For example, to pin to the 1.x.x of the agent, run:

      sudo yum install -y google-fluentd-1.*
      

   3. If you don't want to pin to a major version, you can select one of the other installation options:

      1. To install a specific version of the agent, run:

           sudo yum install -y google-fluentd-version-number
         

      2. To install the latest version of the agent, run:

           sudo yum install -y google-fluentd
         

5. Install the Configuration files.

   1. For unstructured logging, run:

      sudo yum install -y google-fluentd-catch-all-config
      

   2. For structured logging, run:

      sudo yum install -y google-fluentd-catch-all-config-structured
      

      For information about structured logging, refer to the structured logging guide.

6. Restart the agent service

   A restart is required for the configurations that are installed by the catch-all packages above to take effect.

   sudo service google-fluentd restart
   

You can delete the installation script after it runs successfully.

• To verify that the agent is working as expected, run:

  sudo service google-fluentd status
  

  The status of the agent should be OK.

• You can also examine the logs and ensure there are no errors:

  tail /var/log/google-fluentd/google-fluentd.log
  

If you have trouble with the installation, refer to the Troubleshooting page.

DEBIAN / UBUNTU

1. Open a terminal connection to your VM instance using SSH or a similar tool and ensure you have sudo access.

2. Change to a directory you have write access to, for example your home directory.

3. Add the agent's package repository:

   curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh && \
   sudo bash add-logging-agent-repo.sh && \
   sudo apt-get update
   

4. Install the agent:

   1. List the available versions of the agent in order to select which version to install:

      sudo apt-cache madison google-fluentd
      

   2. For production environments, you might want to pin to a major version to avoid pulling in major versions that might include backward incompatible changes. To pin to a major version, run:

      sudo apt-get install -y 'google-fluentd=major-version.*'
      

      For example, to pin to the 1.x.x of the agent, run:

      sudo apt-get install -y 'google-fluentd=1.*'
      

   3. If you don't want to pin to a major version, you can select one of the other installation options:

      1. To install a specific version of the agent, run:

           sudo apt-get install -y google-fluentd=version-number
         

      2. To install the latest version of the agent, run:

           sudo apt-get install -y google-fluentd
         

5. Install the Configuration files.

   1. For unstructured logging, run:

      sudo apt-get install -y google-fluentd-catch-all-config
      

   2. For structured logging, run:

      sudo apt-get install -y google-fluentd-catch-all-config-structured
      

      For information about structured logging, refer to the structured logging guide.

6. Restart the agent service

   A restart is required for the configurations that are installed by the catch-all packages above to take effect.

   sudo service google-fluentd restart
   

You can delete the installation script after it runs successfully.

• To verify that the agent is working as expected, run:

  sudo service google-fluentd status
  

  The status of the agent should be OK.

• You can also examine the logs and ensure there are no errors:

  tail /var/log/google-fluentd/google-fluentd.log
  

If you have trouble with the installation, refer to the Troubleshooting page.

SLES / SUSE

1. Open a terminal connection to your VM instance using SSH or a similar tool and ensure you have sudo access.

2. Change to a directory you have write access to, for example your home directory.

3. Add the agent's package repository:

   curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
   sudo bash add-logging-agent-repo.sh
   

4. Install the agent:

   1. List the available versions of the agent in order to select which version to install:

      sudo zypper search -s google-fluentd
      

   2. For production environments, you might want to pin to a major version to avoid pulling in major versions that might include backward incompatible changes. To pin to a major version, run:

      sudo zypper install -y 'google-fluentd<major-version.0.0'
      

      For example, to pin to the 1.x.x of the agent, run:

      sudo zypper install -y 'google-fluentd<2.0.0'
      

   3. If you don't want to pin to a major version, you can select one of the other installation options:

      1. To install a specific version of the agent, run:

           sudo zypper install -y google-fluentd=version-number
         

      2. To install the latest version of the agent, run:

           sudo zypper install -y google-fluentd
         

5. Install the Configuration files.

   1. For unstructured logging, run:

      sudo zypper install -y google-fluentd-catch-all-config
      

   2. For structured logging, run:

      sudo zypper install -y google-fluentd-catch-all-config-structured
      

      For information about structured logging, refer to the structured logging guide.

6. Restart the agent service

   A restart is required for the configurations that are installed by the catch-all packages above to take effect.

   sudo service google-fluentd restart
   

You can delete the installation script after it runs successfully.

• To verify that the agent is working as expected, run:

  sudo service google-fluentd status
  

  The status of the agent should be OK.

• You can also examine the logs and ensure there are no errors:

  tail /var/log/google-fluentd/google-fluentd.log
  

If you have trouble with the installation, refer to the Troubleshooting page.

WINDOWS

Beta

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

To install the agent on a VM instance running Windows, perform the following steps:

1. Connect to your instance using RDP or a similar tool and login to Windows.

2. Open a PowerShell terminal with administrator privileges by right-clicking the PowerShell icon and selecting Run as Administrator.

3. Run the following PowerShell command:

   (New-Object Net.WebClient).DownloadFile("https://dl.google.com/cloudagents/windows/StackdriverLogging-v1-14.exe", "${env:UserProfile}\StackdriverLogging-v1-14.exe")
   & "${env:UserProfile}\StackdriverLogging-v1-14.exe"
   

   Alternatively, you can browse to the following URL to download and run the agent's installer:

   https://dl.google.com/cloudagents/windows/StackdriverLogging-v1-14.exe

   Caution: You must download the installer to a non-system directory, such as C:\Users\[USERNAME]. For security reasons, the installer doesn't run from system directories, including the directory C:\.
   If you download the installer to a system directory, move it to another directory before running it.
   Note: The agent installer also supports a “silent” option, /S, that can be used in scripting the installation.

   To install the agent silently, append the /S option to the invocation of the installer:

   .\StackdriverLogging-v1-14.exe /S
   

   Caution: Silent installation is an asynchronous operation. When the installation command exits, the agent is not yet installed. You can check for a successful installation by looking for the StackdriverLogging service.
   Note: The installer places the agent in the `C:\Program Files (x86)\Stackdriver\LoggingAgent` directory by default. You can change that directory during installation.

   In “silent” mode use the /D option to specify the installation directory, for example:

   .\StackdriverLogging-v1-14.exe /S /D="C:\Google Cloud's operations suite\Logging\"
   

   You can delete the installer when it completes successfully.

   If you have trouble with the installation, refer to the Troubleshooting page.

Installing the agent on a Compute Engine VM

You can install the Logging agent on a single VM from the pre-configured Monitoring VM Instances dashboard.

To reach this dashboard, do the following:

1. In the Cloud Console, select your Google Cloud project.
   Go to Cloud Console
2. In the navigation pane, select Monitoring.

   If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn't associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.

3. In the Monitoring navigation pane, select Dashboards.
4. In the table of dashboards, locate the VM Instances entry, and then click on the name.

The Inventory tab on the VM Instances dashboard lists all VMs and contains a status column for the agent, as shown in the following screenshot:

If the agent is not detected on a Compute Engine instance, you can click the "Not detected" link to bring up an installation workflow in the dashboard.

Optional tasks

This section describes how to perform common maintenance tasks.

Configuring the agent

The agent comes preconfigured to monitor certain known log locations. On Linux, those locations are described in the package google-fluentd-catch-all-config, which is automatically pulled in by the installation script. On Windows, the agent monitors the Windows Event Log by default.

To adjust the agent configuration, see Configuring the Logging agent.

Configuring an HTTP proxy

If you use an HTTP proxy for proxying requests to the Logging and Monitoring APIs, do the following:

LINUX

1. Edit the following configuration file (should be created if does not exist):

    /etc/default/google-fluentd
   

2. Add the following lines:

    export http_proxy="http://proxy-ip:proxy-port"
    export https_proxy="http://proxy-ip:proxy-port"
    export no_proxy=169.254.169.254  # Skip proxy for the local Metadata Server.
   

3. Restart the Logging agent by running the following command on your VM instance.

    sudo service google-fluentd restart
   

WINDOWS

1. If you use an HTTP proxy, run the following command from an administrator command prompt. This sets the http_proxy and https_proxy environment variables so that the agent can send data to Logging using outbound HTTPS:

   setx http_proxy http://proxy-ip:proxy-port /m
   setx https_proxy http://proxy-ip:proxy-port /m
   setx no_proxy 169.254.169.254 /m
   

Determining the agent version

To determine the version of the Logging agent on your system, run the following commands on your VM instance:

AMAZON LINUX AMI / CENTOS / RHEL

Run the following command on Amazon Linux, Red Hat, or CentOS Linux:

rpm --query --queryformat '%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n' \
     google-fluentd 

     google-fluentd-catch-all-config 

     google-fluentd-catch-all-config-structured

DEBIAN / UBUNTU

Run the following command on Debian or Ubuntu:

dpkg-query --show --showformat \
    '${Package} ${Version} ${Architecture} ${Status}\n' \
     google-fluentd 

     google-fluentd-catch-all-config 

     google-fluentd-catch-all-config-structured

SLES / SUSE

Run the following command on SUSE:

rpm --query --queryformat '%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n' \
     google-fluentd 

     google-fluentd-catch-all-config 

     google-fluentd-catch-all-config-structured

WINDOWS

1. Connect to your instance using RDP or a similar tool and login to Windows.

2. Open a PowerShell terminal with administrator privileges by right-clicking the PowerShell icon and selecting Run as Administrator.

3. Run the following PowerShell command:

    reg query HKLM\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\GoogleStackdriverLoggingAgent\ /v Version
   

Restarting the agent

You must restart the Logging agent to pick up changes in configuration files. To restart the agent, use the following instructions.

LINUX

Run the following command on your instance:

 sudo service google-fluentd restart

After restarting the Logging agent, you might want to send a test message.

WINDOWS

1. Connect to your instance using RDP or a similar tool and login to Windows.

2. Open a PowerShell terminal with administrator privileges by right-clicking the PowerShell icon and selecting Run as Administrator.

3. Run the following PowerShell command:

Restart-Service -Name StackdriverLogging

Upgrading the agent

To upgrade the Logging agent to the latest release, use the following instructions:

Note: If you upgraded your instance's Linux operating system to a new major release, then you should first remove the agent and then re-install it using the procedures on this page, instead of completing these upgrade procedures.

AMAZON LINUX AMI / CENTOS / RHEL

Run the following command on Amazon Linux, Red Hat, or CentOS Linux:

sudo yum upgrade google-fluentd

The previous command doesn't change the agent's configuration file. To get the latest default configuration and catch-all configuration files, run the following commands on your instance:

sudo yum upgrade google-fluentd google-fluentd-catch-all-config

DEBIAN / UBUNTU

Run the following commands on Debian or Ubuntu:

sudo apt-get update
sudo apt-get install -y --only-upgrade google-fluentd

The previous commands don't change the agent's configuration file. To get the latest default configuration and catch-all configuration files, run the following commands on your instance:

sudo apt-get install -y --only-upgrade -o Dpkg::Options::="--force-confnew" google-fluentd-catch-all-config
sudo apt-get install -y --only-upgrade google-fluentd

SLES / SUSE

Run the following command on SUSE:

sudo zypper update google-fluentd

The previous command doesn't change the agent's configuration file. To get the latest default configuration and catch-all configuration files, run the following commands on your instance:

sudo zypper update google-fluentd google-fluentd-catch-all-config

WINDOWS

To upgrade to the latest agent release, install the newest agent as described in Installing on Windows on this page. The installer prompts you to uninstall the previous version of the agent.

Uninstalling the agent

To remove the Logging agent and its configuration files, use the following instructions.

Note: If you added your own fluentd configuration files, copy them from the agent's configuration directory for safekeeping.

AMAZON LINUX AMI / CENTOS / RHEL

Run the following command on Amazon Linux, Red Hat, or CentOS Linux:

sudo service google-fluentd stop
sudo yum remove google-fluentd google-fluentd-catch-all-config

DEBIAN / UBUNTU

Run the following command on Debian or Ubuntu:

sudo service google-fluentd stop
sudo apt-get remove google-fluentd google-fluentd-catch-all-config

SLES / SUSE

Run the following command on SUSE Linux:

sudo service google-fluentd stop
sudo zypper remove google-fluentd google-fluentd-catch-all-config

WINDOWS

In the Windows Control Panel, choose Uninstall a program. You should see the Logging agent in the list of programs that you can uninstall.