Troubleshooting
See the support page for suggestions about how to ask for more help if the sections below don't cover your question.
How can I trace gRPC issues?
When working with libraries that use gRPC, you need to be aware of which gRPC implementation you're using in order to enable logging.
For libraries released from June 2022 onwards, the default gRPC implementation is Grpc.Net.Client in environments where that works (in particular .NET Core 3.1 and .NET 5 and higher). The older Grpc.Core implementation is used in those .NET Framework environments where Grpc.Net.Client isn't available. See the transports documentation for more information about gRPC implementation selection.
Client-level logging
The client builder used to configure clients has a Logger
property
which logs trace-level messages. This only provides basic information on when API calls start, end,
or are retried, but that can still be useful for diagnostic purposes.
The logger will automatically be configured if you use dependency injection to configure the client. If you wish to log manually, you can create a logger factory and then a logger explicitly:
var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole().SetMinimumLevel(LogLevel.Trace));
var client = new ExampleClientBuilder
{
Logger = loggerFactory.CreateLogger("ExampleClient")
}.Build();
Adding this code into the getting started Translation sample code produces output such as this:
trce: TranslationClient[0]
Starting synchronous API call to TranslateText.
trce: TranslationClient[0]
Call completed to TranslateText.
Transport logging with Grpc.Net.Client
The GrpcChannelOptions
class in Grpc.Net.Client has a LoggerFactory
property that can be set
to log debug and trace messages. If you're configuring clients via dependency injection, then simply calling
AddGrpcNetClientAdapter()
on the service collection will ensure that the client is constructed with the loggers
from the resulting service provider. For example, to make a ExampleClient
available to dependency
injection with logging configured automatically, you might use:
builder.Services
.AddGrpcNetClientAdapter()
.AddExampleClient();
If you're not using dependency injection, it's still reasonably straightforward to create a logger factory and configure a client to use it:
var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole().SetMinimumLevel(LogLevel.Debug));
var grpcAdapter = GrpcNetClientAdapter.Default
.WithAdditionalOptions(options => options.LoggerFactory = loggerFactory);
var client = new ExampleClientBuilder
{
GrpcAdapter = grpcAdapter
}.Build();
Note that however the logger is set up, it needs to be configured with a minimum log level of debug or trace in order to be useful.
Adding this code into the getting started Translation sample code produces output such as this:
dbug: Grpc.Net.Client.Internal.GrpcCall[1]
Starting gRPC call. Method type: 'Unary', URI: 'https://translate.googleapis.com/google.cloud.translation.v3.TranslationService/TranslateText'.
dbug: Grpc.Net.Client.Balancer.Subchannel[1]
Subchannel id '1' created with addresses: translate.googleapis.com:443
... (more output) ...
dbug: Grpc.Net.Client.Internal.GrpcCall[18]
Sending message.
dbug: Grpc.Net.Client.Internal.GrpcCall[13]
Reading message.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
Finished gRPC call.
With a minimum log level of Trace, there's considerably more output.
Transport logging with Grpc.Core
There are two aspects to configuring logging with Grpc.Core.
- Setting environment variables
- Directing logs
The environment variables affecting gRPC are listed in the gRPC
repository.
The important ones for diagnostics are GRPC_TRACE
and
GRPC_VERBOSITY
. For example, you might want to start off with
GRPC_TRACE=all
and GRPC_VERBOSITY=debug
which will dump a lot
of information, then tweak them to reduce this to only useful
data... or start with one kind of tracing (e.g.
GRPC_TRACE=call_error
) and add more as required.
By default, the gRPC logs will not be displayed anywhere. The simplest way of seeing gRPC logs in many cases will be to send them to the console:
using Grpc.Core;
using Grpc.Core.Logging;
...
// Call this before you do any gRPC work
GrpcEnvironment.SetLogger(new ConsoleLogger());
Other ILogger
implementations are available, or you can implement
it yourself to integrate with other systems - see the
Grpc.Core.Logging
namespace for details.
Adding this code into the getting started Translation sample code,
using GRPC_VERBOSITY=info
and GRPC_TRACE=all
produces output starting with lines such as this:
D1128 12:45:03.460271 Grpc.Core.Internal.UnmanagedLibrary Attempting to load native library "C:\Users\[...]\GettingStarted\bin\Debug\net48\grpc_csharp_ext.x64.dll"
D1128 12:45:03.549778 Grpc.Core.Internal.NativeExtension gRPC native library loaded successfully.
I1128 12:45:03.564911 0 ..\..\..\src\core\lib\iomgr\timer_manager.cc:88: Spawn timer thread
I1128 12:45:03.564911 0 ..\..\..\src\core\lib\surface\init.cc:196: grpc_init(void)
How can I trace requests and responses in REST-based APIs?
For libraries that use HTTP1.1 and REST (Google.Cloud.Storage.V1, Google.Cloud.BigQuery.V2, Google.Cloud.Translation.V2), it can be useful to perfom request and response logging. There are two aspects to this:
- Registering a global logger
- Configuring the events to log in a specific service
The underlying service is available via the Service
property in each XyzClient
class. Within
that service, you need to configure the HttpClient
's message handler. As a complete example,
here's a call to the Translation API, listing all the available languages, and logging the request
headers and the response body:
// Required using directives:
// using static Google.Apis.Http.ConfigurableMessageHandler;
// using Google;
// using Google.Apis.Logging;
// using Google.Cloud.Translation.V2;
// Register a verbose console logger
ApplicationContext.RegisterLogger(new ConsoleLogger(LogLevel.All));
// Create a translation client
TranslationClient client = TranslationClient.Create();
// Configure which events the message handler will log.
client.Service.HttpClient.MessageHandler.LogEvents =
LogEventType.RequestHeaders | LogEventType.ResponseBody;
// Make the request
client.ListLanguages();
To log all events from the message handler, you can set the LogEvents
property to
~LogEventType.None
.
The above sample code produces output such as this:
D2022-11-28 12:50:31.615723 Google.Apis.Auth.OAuth2.DefaultCredentialProvider Loading Credential from file [...]
D2022-11-28 12:50:31.991560 Google.Apis.Auth.OAuth2.ServiceCredential Token has expired, trying to get a new one.
D2022-11-28 12:50:32.024800 Google.Apis.Auth.OAuth2.ServiceCredential Request a new access token. Assertion data is: [...]
D2022-11-28 12:50:32.027101 Google.Apis.Http.ConfigurableMessageHandler Request[00000001] (triesRemaining=3) URI: 'https://oauth2.googleapis.com/token'
D2022-11-28 12:50:32.351707 Google.Apis.Http.ConfigurableMessageHandler Response[00000001] Response status: OK 'OK'
I2022-11-28 12:50:32.360147 Google.Apis.Auth.OAuth2.ServiceCredential New access token was received successfully
D2022-11-28 12:50:32.366091 Google.Apis.Http.ConfigurableMessageHandler Request[00000001] Headers:
[User-Agent] 'gcloud-dotnet/3.1.0; google-api-dotnet-client/1.57.0.0; (gzip)'
[x-goog-api-client] 'gl-dotnet/7.0.0 gccl/3.1.0 gax/4.0.0 gdcl/1.57.0.875'
[Authorization] 'Bearer [...]'
D2022-11-28 12:50:32.463410 Google.Apis.Http.ConfigurableMessageHandler Response[00000001] Body: '{. "data": {. "languages": [ { ... }, { ... }, ... ] }'
How can I debug into the libraries?
The NuGet packages for the libraries contain the PDB files
and SourceLink information required to enable a rich debugging experience,
but in many build scenarios the PDB files are not automatically copied.
If you need to debug into the library code itself (whether for the
API-specific library or support libraries such as Google.Apis.Auth,
Google.Protobuf or Google.Api.Gax.Grpc), set the
CopyDebugSymbolFilesFromPackages
build property to true
, for example
by adding the element below to a <PropertyGroup>
element in your project file:
<CopyDebugSymbolFilesFromPackages>true</CopyDebugSymbolFilesFromPackages>
Note that this requires the .NET 7 SDK (even if you have a different target for your application), and you should have the "Enable Just My Code" checkbox in Visual Studio unchecked.
Why aren't the gRPC native libraries being found?
The native libraries that gRPC relies on are present in Grpc.Core, and the NuGet package has targets to copy them to appropriate output directories. However, due to the way NuGet dependencies are generated with .NET Core, you may find that with transitive dependencies, the targets aren't executed.
We've set up our client libraries (e.g. Google.Cloud.Datastore.V1
)
so that if you directly depend on any of them, everything should
work - but if your application only has transitive dependencies, you
could run into errors like this:
Unhandled Exception: System.IO.FileNotFoundException:
Error loading native library. Not found in any of the possible locations: [...]
at Grpc.Core.Internal.UnmanagedLibrary.FirstValidLibraryPath(String[] libraryPathAlternatives)
at Grpc.Core.Internal.UnmanagedLibrary..ctor(String[] libraryPathAlternatives)
at Grpc.Core.Internal.NativeExtension.Load()
at Grpc.Core.Internal.NativeExtension..ctor()
at Grpc.Core.Internal.NativeExtension.Get()
at Grpc.Core.GrpcEnvironment.GrpcNativeInit()
at Grpc.Core.GrpcEnvironment..ctor()
...
In that case, the simplest fix is to add a direct dependency to
Grpc.Core
from your application, which will ensure that the
native libraries are copied appropriately.
How can I modify repeated fields and maps in protobuf messages?
This information is now in the protocol buffers guide; this section has been retained for the sake of existing links.
How can I get ErrorInfo from RpcException?
ErrorInfo
in RpcException
provides a consistent and human readable error details.
It contains three major fields - reason, domain and metadata described as below:
Reason
: the proximate cause of the error. This is a stable unique identifier for this type of error within the domain, and can be used to programmatically determine how to handle the error.Domain
: the logical grouping to which the error belongs. The error domain is typically the registered service name of the tool or product that generates the error. Example: "pubsub.googleapis.com". If the error is generated by some common infrastructure, the error domain will be a globally unique value that identifies the infrastructure. For Google API infrastructure, the error domain is "googleapis.com".Metadata
: additional structured details about the error.
using Google.Api.Gax.Grpc;
try
{
// Code that can throw the RpcException.
}
catch (RpcException ex)
{
if (ex.GetErrorInfo() is ErrorInfo errorInfo)
{
var domain = errorInfo.Domain;
var reason = errorInfo.Reason;
var metadata = errorInfo.Metadata;
// Use the domain, reason and metadata of ErrorInfo, e.g. for logging or taking remedial action.
}
else
{
// Use RpcException's other properties (e.g. Status and Message) to handle the exception appropriately.
}
}
How can I diagnose proxy issues?
Proxies can be configured in multiple places in .NET:
- In code (e.g. with
HttpClientHandler.Proxy
orWebRequest.DefaultWebProxy
) - In environment variables
- In system settings
Additionally, the client libraries make multiple requests:
- Auth can sometimes require requests (e.g. to obtain OAuth access tokens, or
to access the Compute metadata server); these always use
HttpClient
- Individual RPCs may be made with different technologies:
- In gRPC based libraries, three different transports may
be used:
HttpClient
via Grpc.Net.Client,HttpClient
using the REST transport, or the native library used by Grpc.Core - In the handwritten layers over HTTP/1.1 + REST libraries,
HttpClient
is always used
- In gRPC based libraries, three different transports may
be used:
The combination of options here can make proxy issues hard to troubleshoot. The symptoms may include
RPC failures such as DeadlineExceeded
, or Unavailable
with a detail message of
"failed to connect to all addresses".
If you are configuring a system-wide proxy, it can be useful to check that it's working with HttpClient
by just making a simple request and logging the results. For example:
var httpClient = new System.Net.Http.HttpClient();
var text = await httpClient.GetStringAsync("https://www.google.com");
Console.WriteLine(text.Substring(0, 80));
If that shows the start of an HTML page, then that's a good sign of the configuration working.
For gRPC-based libraries (which covers the majority of cases), we recommend creating a console app using the Translation API, as shown in the Getting Started guide. (You'll need to enable the Translation API in order to make any successful calls, but if you get as far as a "permission denied" error, that's usually enough to prove you've successfully made a network connection.) You can adapt that example to match your real environment as closely as possible:
- Use the same target framework
- Use the same credentials, if possible, or at least the same form of credentials
- If you're customizing the client, e.g. by specifying a different transport, then do the same in the console app
Follow the gRPC tracing steps described earlier to obtain more detail on the requests being made.
Proxy settings for Grpc.Core
One common way of configuring the proxy in Grpc.Core is to set one environment variable out of
grpc_proxy
, http_proxy
and https_proxy
. It is important to note that these environment
values should include a URI scheme, e.g. http://myproxyaddress:12345
instead of just myproxyaddress:12345
.