google.appengine.ext.bulkload.transform module

Summary

Bulkloader Transform Helper functions.

A collection of helper functions for bulkloading data, typically referenced from a bulkloader.yaml file.

Contents

google.appengine.ext.bulkload.transform.blob_to_file(filename_hint_propertyname=None, directory_hint='')source

Writes the blob contents to a file and replaces them with the filename.

Parameters

• filename_hint_propertyname – If present, the filename will begin with the contents of this value in the entity being exported.

• directory_hint – If present, the files will be stored in this directory.

Returns

A function that writes the input blob to a file.

google.appengine.ext.bulkload.transform.blobproperty_from_base64(value)source

google.appengine.ext.bulkload.transform.bytestring_from_base64(value)source

google.appengine.ext.bulkload.transform.child_node_from_list(child_node_name)source

Returns a value suitable for generating an XML child node on export.

The return value is a list of tuples that the simplexml connector will use to build a child node.

See also list_from_child_node

Parameters

child_node_name – The name to use for each child node.

Returns

A function that works as described in the args.

google.appengine.ext.bulkload.transform.create_deep_key(*path_info)source

A method that makes multi-level Key objects.

Generates a multi-level key from multiple fields in the input dictionary.

This is typically used for keys for entities that have variable parent keys, e.g. ones with owned relationships. It can used for both __key__ and references.

Use create_foreign_key as a simpler way to create single-level keys.

Parameters

• *path_info – A list of tuples, describing (kind, property, is_id=False).

• kind – The kind name.

• property – The external property in the current import dictionary, or transform.CURRENT_PROPERTY for the value passed to the transform.

• is_id –

  If True, converts value to int and treats it as a numeric ID. If False, the value is a string name. Default is False.

  Example: create_deep_key((‘rootkind’, ‘rootcolumn’),

  (‘childkind’, ‘childcolumn’, True), (‘leafkind’, transform.CURRENT_PROPERTY))

Returns

A transform function that parses the info from the current neutral dictionary into a Key with parents as described by path_info.

google.appengine.ext.bulkload.transform.create_foreign_key(kind, key_is_id=False)source

A method that makes single-level Key objects.

These are typically used in ReferenceProperty in Python, where the reference value is a key with kind (or model) name.

This helper method does not support keys with parents. Use create_deep_key instead to create keys with parents.

Parameters

• kind – The kind name of the reference as a string.

• key_is_id – If True, converts the key into an integer to be used as an ID. If False, leaves the key in the input format (typically a string).

Returns

A single-argument function that parses a value into a Key of kind entity_kind.

google.appengine.ext.bulkload.transform.empty_if_none(fn)source

A wrapper that returns ‘’ if its input is None. Useful on export.

Can be used in config files (e.g. “transform.empty_if_none(unicode)”) or as a decorator.

Parameters

fn – Single-argument transform function.

Returns

The wrapped function.

google.appengine.ext.bulkload.transform.export_date_time(format)source

A wrapper around strftime that returns ‘’ if the input is None.

Parameters

format – A format string for strftime.

Returns

A single-argument function that converts a datetime into a string using format.

google.appengine.ext.bulkload.transform.fix_param_typo(oops, fixed)source

A decorator that corrects a misspelled parameter name.

A parameter in the split_string() and join_list() functions was originally misspelled ‘delimeter’ instead of ‘delimiter’. We couldn’t correct the error by simply renaming it, because that would break any client code that named the parameter when invoking either function:

# This is fine: split strings on semi-colons. split_string(‘;’)

# This would break unless the client code also changed delimeter to delimiter. split_string(delimeter=’;’)

But spelling counts, even in code, so here we are.

Parameters

• oops – The misspelled parameter name.

• fixed – The correctly spelled parameter name, which matches the name in the definition of the decorated function.

Returns

A function that calls the decorated function correctly when it is invoked with a misspelled parameter.

google.appengine.ext.bulkload.transform.import_date_time(format, _strptime=None)source

A wrapper around strptime that returns None if the input is empty.

Parameters

format – A format string for strptime.

Returns

A single-argument function that parses a string into a datetime using format.

google.appengine.ext.bulkload.transform.join_list(*args, **kwargs)source

Calls the decorated function with the correct parameter.

google.appengine.ext.bulkload.transform.key_id_or_name_as_string(key)source

google.appengine.ext.bulkload.transform.key_id_or_name_as_string_n(index)source

Retrieves the nth (0-based) key ID or name from a key that has parents.

If a key is present, returns its ID or name as a string.

Note that this loses the distinction between integer IDs and strings that happen to look like integers. Use key_type to distinguish them.

This is a useful complement to create_deep_key.

Parameters

index – The depth of the ID or name to extract, where 0 is the root key and -1 is the leaf key.

Returns

A function that will extract the name or ID of the key at depth index, as a unicode string. The function returns ‘’ if key is empty (unsaved), otherwise raises IndexError if the key is not as deep as described.

google.appengine.ext.bulkload.transform.key_kind(value)source

google.appengine.ext.bulkload.transform.key_kind_n(index)source

Retrieves the nth (0-based) key kind from a key that has parents.

This is a useful complement to create_deep_key.

Parameters

index – The depth of the ID or name to extract, where 0 is the root key and -1 is the leaf key.

Returns

A function that will return the kind of the key at depth index or raise IndexError if the key is not as deep as described.

google.appengine.ext.bulkload.transform.key_type(key)source

google.appengine.ext.bulkload.transform.key_type_n(index)source

Retrieves the nth (0-based) key type from a key that has parents.

This is most useful when paired with key_id_or_name_as_string_n. This is a useful complement to create_deep_key.

Parameters

index – The depth of the ID or name to extract, where 0 is the root key and -1 is the leaf key.

Returns

A function that will return the type (‘ID’ or ‘name’) of the key at depth index. The function returns ‘’ if key is empty (unsaved), otherwise raises IndexError if the key is not as deep as described.

google.appengine.ext.bulkload.transform.list_from_child_node(xpath, suppress_blank=False)source

Returns a list property from child nodes of the current xml node.

This applies only the simplexml helper, as it assumes __node__, the current ElementTree node corresponding to the import record.

Sample usage for structure:

<Visit>

<VisitActivities>

<Activity>A1</Activity> <Activity>A2</Activity>

</VisitActivities>

</Visit>

property: activities external_name: VisitActivities # Ignored on import, used on export. import_transform: list_from_xml_node(‘VisitActivities/Activity’) export_transform: child_node_from_list(‘Activity’)

Parameters

• xpath – XPath to run on the current node.

• suppress_blank – if True, nodes with no text will be skipped.

Returns

A function that works as described in the args.

google.appengine.ext.bulkload.transform.list_from_multiproperty(*external_names)source

Creates a list from multiple properties.

Parameters

*external_names – A list of properties to use.

Returns

A function that returns a list of the properties in external_names.

google.appengine.ext.bulkload.transform.none_if_empty(fn)source

A wrapper that returns None if its input is empty else fn(x).

Useful on import. Can be used in config files (e.g. “transform.none_if_empty(int)”) or as a decorator.

Parameters

fn – Single-argument transform function.

Returns

The wrapped function.

google.appengine.ext.bulkload.transform.property_from_list(index)source

Returns the item at position ‘index’ from a list.

Parameters

index – The (0-based) item in the list to return.

Returns

A function that returns the specified item from a list, or ‘’ if the list contains too few items.

google.appengine.ext.bulkload.transform.regexp_bool(regexp, flags=0)source

Returns a boolean indicating whether the expression matches with re.match.

Note that re.match anchors at the start but not end of the string.

Parameters

• regexp – String, regular expression.

• flags – Optional flags to pass to re.match.

Returns

A function that returns True if the expression matches.

google.appengine.ext.bulkload.transform.regexp_extract(pattern, method=match, group=1)source

Returns the string that matches the specified group in the regex pattern.

Parameters

• pattern – A regular expression to match on with at least one group.

• method – The method to use for matching; normally re.match (the default) or re.search.

• group – The group to use for extracting a value; the first group by default.

Returns

A single-argument function that returns the string that matches the specified group in the pattern, or None if no match was found or the input was empty.

google.appengine.ext.bulkload.transform.regexp_to_list(pattern)source

Returns a list of objects that match a regex.

Useful on import. Uses the provided regex to split a string value into a list of strings. Wrapped by none_if_input_or_result_empty, so returns None if there are no matches for the regex, or if the input is empty.

Parameters

pattern – A regular expression pattern to match against the input string.

Returns

A function that returns None if the input was None or no matches were found, otherwise a list of strings matching the input expression.

google.appengine.ext.bulkload.transform.split_string(*args, **kwargs)source

Calls the decorated function with the correct parameter.