google.appengine.ext.bulkload.bulkloader_config module

Summary

Bulkloader Config Parser and runner.

A library to read bulkloader yaml configs. The code to interface between the bulkloader tool and the various connectors and conversions.

Contents

class google.appengine.ext.bulkload.bulkloader_config.BulkloadStatesource

Bases: object

Encapsulates state which is passed to other methods used in bulk loading.

It is optionally passed to import/export transform functions. It is passed to connector objects.

Properties:

filename: The filename flag passed on the command line. loader_opts: The loader_opts flag passed on the command line. exporter_opts: The exporter_opts flag passed on the command line. current_instance: The current entity or model instance. current_entity: On export, the current entity instance. current_dictionary: The current input or output dictionary.

class google.appengine.ext.bulkload.bulkloader_config.DictConvertor(transformer_spec)source

Bases: object

Convert a dict to an App Engine model instance or entity. And back.

The constructor takes a transformer spec representing a single transformer in a bulkloader.yaml.

The DictConvertor object has two public methods, dict_to_entity and entity_to_dict, which do the conversion between a neutral dictionary (the input/output of a connector) and an entity based on the spec.

Note that the model class may be used instead of an entity during the transform–this adds extra validation, etc, but also has a performance hit.

dict_to_entity(input_dict, bulkload_state)source

Transform the dict to a model or entity instance(s).

Parameters

• input_dict – Neutral input dictionary describing a single input record.

• bulkload_state – bulkload_state object describing the state.

Returns

Entity or model instance, or collection of entity or model instances, to be uploaded.

entity_to_dict(entity, bulkload_state)source

Transform the entity to a dict, possibly via a model.

Parameters

• entity – An entity.

• bulkload_state – bulkload_state object describing the global state.

Returns

A neutral output dictionary describing the record to write to the output. In the future this may return zero or multiple output dictionaries.

class google.appengine.ext.bulkload.bulkloader_config.GenericExporter(export_recorder, entity_to_dict, kind, sort_key_from_entity)source

Bases: object

Implements bulkloader.Exporter interface and delegates.

This will delegate to the passed in entity_to_dict method and the methods on the export_recorder which are in the ConnectorInterface.

finalize()source

Performs finalization actions after the download completes.

initialize(filename, exporter_opts)source

Performs initialization and validation of the output file.

Parameters

• filename – The string given as the –filename flag argument.

• exporter_opts – The string given as the –exporter_opts flag argument.

output_entities(entity_iterator)source

Outputs the downloaded entities.

Parameters

entity_iterator – An iterator that yields the downloaded entities in sorted order.

class google.appengine.ext.bulkload.bulkloader_config.GenericImporter(import_record_iterator, dict_to_entity, name, reserve_keys)source

Bases: object

Generic Bulkloader import class for input->dict->model transformation.

The bulkloader will call generate_records and create_entity, and we’ll delegate those to the passed in methods.

create_entity(values, key_name=None, parent=None)source

Creates entity/entities from input values via the dict_to_entity method.

Parameters

• values – Neutral dict from generate_records.

• key_name – record number from generate_key.

• parent – Always None in this implementation of a Loader.

Returns

Entity or model instance, or collection of entity or model instances, to be uploaded.

finalize()source

Performs finalization actions after the upload completes.

If keys with numeric ids were used on import, this will call AllocateIds to ensure that autogenerated IDs will not raise exceptions on conflict with uploaded entities.

generate_key(line_number, unused_values)source

Bulkloader method to generate keys, mostly unused here.

This is called by the bulkloader just before it calls create_entity. The line_number is returned to be passed to the record dict, but otherwise unused.

Parameters

• line_number – Record number from the bulkloader.

• unused_values – Neutral dict from generate_records; unused.

Returns

line_number for use later on.

generate_records(filename)source

Iterator yielding neutral dictionaries from the connector object.

Parameters

filename – Filename argument passed in on the command line.

Returns

Iterator yielding neutral dictionaries, later passed to create_entity.

get_keys_to_reserve()source

Required as part of the bulkloader Loader interface.

At the moment, this is not actually used by the bulkloader for import; instead we will reserve keys if necessary in finalize.

Returns

List of keys to reserve, currently always [].

initialize(filename, loader_opts)source

Performs initialization. Merely records the values for later use.

Parameters

• filename – The string given as the –filename flag argument.

• loader_opts – The string given as the –loader_opts flag argument.

google.appengine.ext.bulkload.bulkloader_config.create_transformer_classes(transformer_spec, config_globals, reserve_keys)source

Create an importer and exporter class from a transformer spec.

Parameters

• transformer_spec – A bulkloader_parser.TransformerEntry.

• config_globals – Dict to use to reference globals for code in the config.

• reserve_keys – Method ReserveKeys(keys) which will advance the id sequence in the datastore beyond each key.id(). Can be None.

Raises

InvalidConfig – when the config is invalid.

Returns

Tuple, (importer class, exporter class), each which is in turn a wrapper for the GenericImporter/GenericExporter class using a DictConvertor object configured as per the transformer_spec.

google.appengine.ext.bulkload.bulkloader_config.default_export_transform(value)source

A default export transform if nothing else is specified.

We assume most export connectors are string based, so a string cast is used. However, casting None to a string leads to ‘None’, so that’s special cased.

Parameters

value – A value of some type.

Returns

unicode(value), or u’’ if value is None

google.appengine.ext.bulkload.bulkloader_config.load_config(filename, update_path=True, reserve_keys=None)source

Load a configuration file and create importer and exporter classes.

Parameters

• filename – Filename of bulkloader.yaml.

• update_path – Should sys.path be extended to include the path of filename?

• reserve_keys – Method ReserveKeys(keys) which will advance the id sequence in the datastore beyond each key.id(). Can be None.

Returns

Tuple, (importer classes, exporter classes) based on the transformers specified in the file.

google.appengine.ext.bulkload.bulkloader_config.load_config_from_stream(stream, reserve_keys=None)source

Parse a bulkloader.yaml file into bulkloader loader classes.

Parameters

• stream – A stream containing bulkloader.yaml data.

• reserve_keys – Method ReserveKeys(keys) which will advance the id sequence in the datastore beyond each key.id(). Can be None.

Returns

Constructors suitable to pass to the bulkloader.

Return type

importer_classes, exporter_classes