DataflowPipelineOptions (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineOptions

• All Superinterfaces:

  ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowWorkerLoggingOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions, StreamingOptions

  All Known Subinterfaces:

  BlockingDataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

  ---

  
  

  public interface DataflowPipelineOptions
  extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions, DataflowWorkerLoggingOptions, DataflowProfilingOptions

  Options that can be used to configure the DataflowPipeline.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Interface and Description
    static class	DataflowPipelineOptions.JobNameFactory Returns a normalized job name constructed from ApplicationNameOptions.getAppName(), the local system user name (if available), and the current time.

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

      DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.PathValidatorFactory, DataflowPipelineDebugOptions.StagerFactory

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

      DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.TeardownPolicy, DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

      GcpOptions.CredentialDirFactory, GcpOptions.DefaultProjectFactory, GcpOptions.GcpUserCredentialsFactory

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

      GoogleApiDebugOptions.GoogleApiTracer

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

      PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

      GcsOptions.ExecutorServiceFactory

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

      DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowProfilingOptions

      DataflowProfilingOptions.DataflowProfilingAgentConfiguration

  • Method Summary

    All Methods Instance Methods Abstract Methods 

    Modifier and Type	Method and Description
    String	getJobName() The Dataflow job name is used as an idempotence key within the Dataflow service.
    Map<String,String>	getLabels() Labels that will be applied to the billing records for this job.
    String	getProject() Project id to use when launching jobs.
    String	getServiceAccount() Run the job as a specific service account, instead of the default GCE robot.
    String	getStagingLocation() GCS path for staging local files, e.g.
    boolean	getUpdate() Whether to update the currently running pipeline with the same name as this one.
    void	setJobName(String value)
    void	setLabels(Map<String,String> labels)
    void	setProject(String value)
    void	setServiceAccount(String value)
    void	setStagingLocation(String value)
    void	setUpdate(boolean value)

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions

      getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getDumpHeapOnOOM, getExperiments, getNumberOfWorkerHarnessThreads, getOverrideWindmillBinary, getPathValidator, getPathValidatorClass, getPubsubRootUrl, getStager, getStagerClass, getTransformNameMapping, getWindmillServiceEndpoint, getWindmillServicePort, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setDumpHeapOnOOM, setExperiments, setNumberOfWorkerHarnessThreads, setOverrideWindmillBinary, setPathValidator, setPathValidatorClass, setPubsubRootUrl, setStager, setStagerClass, setTransformNameMapping, setWindmillServiceEndpoint, setWindmillServicePort

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions

      getAutoscalingAlgorithm, getDiskSizeGb, getFilesToStage, getMaxNumWorkers, getNetwork, getNumWorkers, getSubnetwork, getTeardownPolicy, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, getZone, setAutoscalingAlgorithm, setDiskSizeGb, setFilesToStage, setMaxNumWorkers, setNetwork, setNumWorkers, setSubnetwork, setTeardownPolicy, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType, setZone

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.BigQueryOptions

      getTempDatasetId, setTempDatasetId

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.StreamingOptions

      isStreaming, setStreaming

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.ApplicationNameOptions

      getAppName, setAppName

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions

      getAuthorizationServerEncodedUrl, getCredentialDir, getCredentialFactoryClass, getCredentialId, getGcpCredential, getSecretsFile, getServiceAccountKeyfile, getServiceAccountName, getTokenServerUrl, setAuthorizationServerEncodedUrl, setCredentialDir, setCredentialFactoryClass, setCredentialId, setGcpCredential, setSecretsFile, setServiceAccountKeyfile, setServiceAccountName, setTokenServerUrl

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions

      getGoogleApiTrace, setGoogleApiTrace

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

      as, cloneAs, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, outputRuntimeOptions, setOptionsId, setRunner, setStableUniqueNames, setTempLocation

    • Methods inherited from interface com.google.cloud.dataflow.sdk.transforms.display.HasDisplayData

      populateDisplayData

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions

      getExecutorService, getGcsEndpoint, getGcsUploadBufferSizeBytes, getGcsUtil, setExecutorService, setGcsEndpoint, setGcsUploadBufferSizeBytes, setGcsUtil

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions

      getDebuggee, getEnableCloudDebugger, getMaxConditionCost, setDebuggee, setEnableCloudDebugger, setMaxConditionCost

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions

      getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowProfilingOptions

      getEnableProfilingAgent, getProfilingAgentConfiguration, getSaveProfilesToGcs, setEnableProfilingAgent, setProfilingAgentConfiguration, setSaveProfilesToGcs

• • Method Detail

    • getProject

      @Validation.Required
       @Default.InstanceFactory(value=GcpOptions.DefaultProjectFactory.class)
      String getProject()

      Description copied from interface: GcpOptions
      Project id to use when launching jobs.

      Specified by:

      getProject in interface GcpOptions

    • setProject

      void setProject(String value)

      Specified by:

      setProject in interface GcpOptions

    • getStagingLocation

      String getStagingLocation()

      GCS path for staging local files, e.g. gs://bucket/object

      Must be a valid Cloud Storage URL, beginning with the prefix "gs://"

      At least one of PipelineOptions.getTempLocation() or getStagingLocation() must be set. If PipelineOptions.getTempLocation() is not set, then the Dataflow pipeline defaults to using getStagingLocation().

    • setStagingLocation

      void setStagingLocation(String value)

    • getJobName

      @Default.InstanceFactory(value=DataflowPipelineOptions.JobNameFactory.class)
      String getJobName()

      The Dataflow job name is used as an idempotence key within the Dataflow service. If there is an existing job that is currently active, another active job with the same name will not be able to be created. Defaults to using the ApplicationName-UserName-Date.

    • setJobName

      void setJobName(String value)

    • getUpdate

      boolean getUpdate()

      Whether to update the currently running pipeline with the same name as this one.

      Specified by:

      getUpdate in interface DataflowPipelineDebugOptions

    • setUpdate

      void setUpdate(boolean value)

      Specified by:

      setUpdate in interface DataflowPipelineDebugOptions

    • getServiceAccount

      @Hidden
       @Experimental
      String getServiceAccount()

      Run the job as a specific service account, instead of the default GCE robot.

    • setServiceAccount

      void setServiceAccount(String value)

    • getLabels

      Map<String,String> getLabels()

      Labels that will be applied to the billing records for this job.

    • setLabels

      void setLabels(Map<String,String> labels)