google.appengine.api.files.records module

Summary

Files API.

Deprecated since version 1.8.1: Use Google Cloud Storage Client library instead.

Lightweight record format.

This format implements log file format from leveldb: http://leveldb.googlecode.com/svn/trunk/doc/log_format.txt

Full specification of format follows in case leveldb decides to change it.

The log file contents are a sequence of 32KB blocks. The only exception is that the tail of the file may contain a partial block.

Each block consists of a sequence of records:

block := record* trailer? record :=

checksum: uint32 // masked crc32c of type and data[] length: uint16 type: uint8 // One of FULL, FIRST, MIDDLE, LAST data: uint8[length]

A record never starts within the last six bytes of a block (since it won’t fit). Any leftover bytes here form the trailer, which must consist entirely of zero bytes and must be skipped by readers.

Aside: if exactly seven bytes are left in the current block, and a new non-zero length record is added, the writer must emit a FIRST record (which contains zero bytes of user data) to fill up the trailing seven bytes of the block and then emit all of the user data in subsequent blocks.

More types may be added in the future. Some Readers may skip record types they do not understand, others may report that some data was skipped.

FULL == 1 FIRST == 2 MIDDLE == 3 LAST == 4

The FULL record contains the contents of an entire user record.

FIRST, MIDDLE, LAST are types used for user records that have been split into multiple fragments (typically because of block boundaries). FIRST is the type of the first fragment of a user record, LAST is the type of the last fragment of a user record, and MID is the type of all interior fragments of a user record.

Example: consider a sequence of user records:

A: length 1000 B: length 97270 C: length 8000

A will be stored as a FULL record in the first block.

B will be split into three fragments: first fragment occupies the rest of the first block, second fragment occupies the entirety of the second block, and the third fragment occupies a prefix of the third block. This will leave six bytes free in the third block, which will be left empty as the trailer.

C will be stored as a FULL record in the fourth block.

Contents

exception google.appengine.api.files.records.Errorsource

Bases: exceptions.Exception

Base class for exceptions in this module.

class google.appengine.api.files.records.FileReadersource

Bases: object

Interface specification for writers to be used with recordrecords module.

FileReader defines a reader with position and efficient seek/position determining. All reads occur at current position.

read(size)source

Read data from file.

Reads data from current position and advances position past the read data block.

Parameters

size – number of bytes to read.

Returns

iterable over bytes. If number of bytes read is less then ‘size’ argument, it is assumed that end of file was reached.

tell()source

Get current file position.

Returns

current position as a byte offset in the file as integer.

class google.appengine.api.files.records.FileWritersource

Bases: object

Interface specification for writers to be used with records module.

write(data)source

Write data to the file.

Parameters

data – byte array, string or iterable over bytes.

exception google.appengine.api.files.records.InvalidRecordErrorsource

Bases: google.appengine.api.files.records.Error

Raised when invalid record encountered.

class google.appengine.api.files.records.RecordsReader(reader)source

Bases: object

A reader for records format.

read()source

Reads record from current position in reader.

seek(*args, **kwargs)source

Set the file’s current position.

Arguments are passed directly to the underlying reader.

tell()source

Return file’s current position.

class google.appengine.api.files.records.RecordsWriter(writer, _pad_last_block=True)source

Bases: object

A writer for records format.

This writer should be used only inside with statement:

with records.RecordsWriter(file) as writer:

writer.write(“record”)

RecordsWriter will pad last block with 0 when exiting with statement scope.

close()source
write(data)source

Write single record.

Parameters

data – record data to write as string, byte array or byte sequence.