Problem
When using classic templates to create data pipelines in Dataflow, the data pipeline interface does not populate pipeline or template information. The Dataflow job is successful but users cannot modify pipeline attributes.
Analyzing the HAR file, a 400 response is thrown with the error:
Fail to process as Flex Template and Legacy Template. Flex Template Process result:No SDK language information is provided., Legacy Template Process result: Unable to open template file: gs://<location>", "status": "INVALID_ARGUMENT"
Environment
- Dataflow Data Pipeline
- Classic Templates
Solution
- Manually add a metadata file to the template Google Cloud Storage location. The use of metadata with Classic Template resolves the error 400.
Cause
The issue is only related to the Data pipelines interface, since it uses metadata to populate Pipeline and Template information. Using metadata with Classic Template is optional so that jobs run successfully.