MODEL DDL statements and ML functions in Spanner invoke Vertex AI endpoints and can fail due to various reasons:
| Error Code | Error Message | Possible cause | Possible solution |
|---|---|---|---|
| CANCELLED | Call to Vertex AI endpoint {ENDPOINT} was cancelled due to query cancellation. | Query was cancelled by the client application. | Investigate why your client cancelled the query. |
| DEADLINE_EXCEEDED | Vertex AI endpoint {ENDPOINT} exceeded the call deadline. | Query deadline is too short. | Increase query deadline on the client. |
| - | - | Endpoint was too busy. | See Vertex AI monitoring and deploy more nodes. |
| FAILED_PRECONDITION | Vertex AI endpoint {ENDPOINT} returned failed precondition error. | Endpoint has no models deployed. | Deploy models to the endpoint. | INTERNAL | Unknown error when accessing Vertex AI endpoint {ENDPOINT}. | Unexpected internal error. | Use failover endpoints or open a support ticket. |
| INVALID_ARGUMENT | Invalid request to Vertex AI endpoint {ENDPOINT}. Make sure that Vertex AI endpoint and Spanner model schema match. | Spanner model schema and Vertex AI endpoint schema do not match | Update Spanner model's schema. |
| NOT_FOUND | Vertex AI endpoint {ENDPOINT} not found. | Endpoint was deleted. | Update Spanner model's schema. |
| PERMISSION_DENIED | Access to Vertex AI endpoint {ENDPOINT} was denied. | Spanner service agent does not have permissions to access the endpoint | Grant service agent role permissions |
| - | - | VPC SC error | See Vertex AI error message and follow VPC SC troubleshooting | RESOURCE_EXHAUSTED | Vertex AI endpoint {ENDPOINT} quota has been exceeded. | Too many requests to Vertex AI. | Increase online prediction quota | UNAVAILABLE | Could not create service agent for project {PROJECT}. | Service Usage API issue. | Create service agent manually | - | Vertex AI endpoint {ENDPOINT} is unavailable. | Too many requests to Vertex AI. | Deploy more nodes. | - | - | Vertex AI has a regional issue | Use failover endpoints |