Package cloud.google.com/go/storage/dataflux (v1.45.0)

Version 1.45.0

• 1.51.0 (latest)
• 1.50.0
• 1.49.0
• 1.48.0
• 1.47.0
• 1.46.0
• 1.45.0
• 1.44.0
• 1.43.0
• 1.42.0
• 1.41.0
• 1.40.0
• 1.39.1
• 1.38.0
• 1.37.0
• 1.36.0
• 1.35.1
• 1.34.1
• 1.33.0
• 1.32.0
• 1.31.0
• 1.30.1
• 1.29.0
• 1.28.1
• 1.27.0
• 1.26.0
• 1.25.0
• 1.24.0
• 1.23.0
• 1.22.1
• 1.21.0
• 1.20.0
• 1.19.0
• 1.18.2
• 1.17.0
• 1.16.1
• 1.15.0
• 1.14.0
• 1.13.0
• 1.12.0

Note: To get more information about this package, such as access to older versions, view this package on pkg.go.dev.

Package dataflux provides an easy way to parallelize listing in Google Cloud Storage.

More information about Google Cloud Storage is available at https://cloud.google.com/storage/docs.

See https://pkg.go.dev/cloud.google.com/go for authentication, timeouts, connection pooling and similar aspects of this package.

NOTE: This package is in preview. It is not stable, and is likely to change.

Lister

type Lister struct {
	// contains filtered or unexported fields
}

Lister is used for interacting with Dataflux fast-listing. The caller should initialize it with NewLister() instead of creating it directly.

func NewLister

func NewLister(c *storage.Client, in *ListerInput) *Lister

NewLister creates a new dataflux Lister to list objects in the give bucket.

func (*Lister) Close

func (c *Lister) Close()

Close closes the range channel of the Lister.

func (*Lister) NextBatch

func (c *Lister) NextBatch(ctx context.Context) ([]*storage.ObjectAttrs, error)

NextBatch runs worksteal algorithm and sequential listing in parallel to quickly return a list of objects in the bucket. For smaller dataset, sequential listing is expected to be faster. For larger dataset, worksteal listing is expected to be faster.

ListerInput

type ListerInput struct {
	// BucketName is the name of the bucket to list objects from. Required.
	BucketName string

	// Parallelism is number of parallel workers to use for listing.
	// Default value is 10x number of available CPU. Optional.
	Parallelism int

	// BatchSize is the number of objects to list. Default value returns
	// all objects at once. The number of objects returned will be
	// rounded up to a multiple of gcs page size. Optional.
	BatchSize int

	// Query is the query to filter objects for listing. Default value is nil.
	// Use ProjectionNoACL for faster listing. Including ACLs increases
	// latency while fetching objects. Optional.
	Query storage.Query

	// SkipDirectoryObjects is to indicate whether to list directory objects.
	// Default value is false. Optional.
	SkipDirectoryObjects bool
}

ListerInput contains options for listing objects.