apache_beam.io.avroio module

PTransforms for reading from and writing to Avro files.

Provides two read PTransform``s, ``ReadFromAvro and ReadAllFromAvro, that produces a PCollection of records. Each record of this PCollection will contain a single record read from an Avro file. Records that are of simple types will be mapped into corresponding Python types. Records that are of Avro type ‘RECORD’ will be mapped to Python dictionaries that comply with the schema contained in the Avro file that contains those records. In this case, keys of each dictionary will contain the corresponding field names and will be of type string while the values of the dictionary will be of the type defined in the corresponding Avro schema.

For example, if schema of the Avro file is the following. {“namespace”: “example.avro”,”type”: “record”,”name”: “User”,”fields”: [{“name”: “name”, “type”: “string”}, {“name”: “favorite_number”, “type”: [“int”, “null”]}, {“name”: “favorite_color”, “type”: [“string”, “null”]}]}

Then records generated by read transforms will be dictionaries of the following form. {‘name’: ‘Alyssa’, ‘favorite_number’: 256, ‘favorite_color’: None}).

Additionally, this module provides a write PTransform WriteToAvro that can be used to write a given PCollection of Python objects to an Avro file.

class apache_beam.io.avroio.ReadFromAvro(file_pattern=None, min_bundle_size=0, validate=True, use_fastavro=True)[source]

Bases: apache_beam.transforms.ptransform.PTransform

A PTransform for reading avro files.

Initializes ReadFromAvro.

Uses source _AvroSource to read a set of Avro files defined by a given file pattern.

If /mypath/myavrofiles* is a file-pattern that points to a set of Avro files, a PCollection for the records in these Avro files can be created in the following manner.

with beam.Pipeline() as p:
  records = p | 'Read' >> beam.io.ReadFromAvro('/mypath/myavrofiles*')

Each record of this PCollection will contain a single record read from a source. Records that are of simple types will be mapped into corresponding Python types. Records that are of Avro type RECORD will be mapped to Python dictionaries that comply with the schema contained in the Avro file that contains those records. In this case, keys of each dictionary will contain the corresponding field names and will be of type str while the values of the dictionary will be of the type defined in the corresponding Avro schema.

For example, if schema of the Avro file is the following.

{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [

    {"name": "name",
     "type": "string"},

    {"name": "favorite_number",
     "type": ["int", "null"]},

    {"name": "favorite_color",
     "type": ["string", "null"]}

  ]
}

Then records generated by _AvroSource will be dictionaries of the following form.

{'name': 'Alyssa', 'favorite_number': 256, 'favorite_color': None}).
Parameters:
  • file_pattern (str) – the file glob to read
  • min_bundle_size (int) – the minimum size in bytes, to be considered when splitting the input into bundles.
  • validate (bool) – flag to verify that the files exist during the pipeline creation time.
  • use_fastavro (bool) – is significantly faster, and is now the default.
expand(pvalue)[source]
display_data()[source]
annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]
default_label()
default_type_hints()
classmethod from_runner_api(proto, context)
get_type_hints()

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs)

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)
label
pipeline = None
classmethod register_urn(urn, parameter_type, constructor=None)
runner_api_requires_keyed_input()
side_inputs = ()
to_runner_api(context, has_parts=False, **extra_kwargs)
to_runner_api_parameter(unused_context)
to_runner_api_pickled(unused_context)
type_check_inputs(pvalueish)
type_check_inputs_or_outputs(pvalueish, input_or_output)
type_check_outputs(pvalueish)
with_input_types(input_type_hint)

Annotates the input type of a PTransform with a type-hint.

Parameters:input_type_hint (type) – An instance of an allowed built-in type, a custom class, or an instance of a TypeConstraint.
Raises:TypeError – If input_type_hint is not a valid type-hint. See apache_beam.typehints.typehints.validate_composite_type_param() for further details.
Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.
Return type:PTransform
with_output_types(type_hint)

Annotates the output type of a PTransform with a type-hint.

Parameters:type_hint (type) – An instance of an allowed built-in type, a custom class, or a TypeConstraint.
Raises:TypeError – If type_hint is not a valid type-hint. See validate_composite_type_param() for further details.
Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.
Return type:PTransform
class apache_beam.io.avroio.ReadAllFromAvro(min_bundle_size=0, desired_bundle_size=67108864, use_fastavro=True, label='ReadAllFiles')[source]

Bases: apache_beam.transforms.ptransform.PTransform

A PTransform for reading PCollection of Avro files.

Uses source ‘_AvroSource’ to read a PCollection of Avro files or file patterns and produce a PCollection of Avro records.

Initializes ReadAllFromAvro.

Parameters:
  • min_bundle_size – the minimum size in bytes, to be considered when splitting the input into bundles.
  • desired_bundle_size – the desired size in bytes, to be considered when splitting the input into bundles.
  • use_fastavro (bool) – is significantly faster, and is now the default.
DEFAULT_DESIRED_BUNDLE_SIZE = 67108864
label
expand(pvalue)[source]
annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]
default_label()
default_type_hints()
display_data()

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:
{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}
Return type:Dict[str, Any]
classmethod from_runner_api(proto, context)
get_type_hints()

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs)

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)
pipeline = None
classmethod register_urn(urn, parameter_type, constructor=None)
runner_api_requires_keyed_input()
side_inputs = ()
to_runner_api(context, has_parts=False, **extra_kwargs)
to_runner_api_parameter(unused_context)
to_runner_api_pickled(unused_context)
type_check_inputs(pvalueish)
type_check_inputs_or_outputs(pvalueish, input_or_output)
type_check_outputs(pvalueish)
with_input_types(input_type_hint)

Annotates the input type of a PTransform with a type-hint.

Parameters:input_type_hint (type) – An instance of an allowed built-in type, a custom class, or an instance of a TypeConstraint.
Raises:TypeError – If input_type_hint is not a valid type-hint. See apache_beam.typehints.typehints.validate_composite_type_param() for further details.
Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.
Return type:PTransform
with_output_types(type_hint)

Annotates the output type of a PTransform with a type-hint.

Parameters:type_hint (type) – An instance of an allowed built-in type, a custom class, or a TypeConstraint.
Raises:TypeError – If type_hint is not a valid type-hint. See validate_composite_type_param() for further details.
Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.
Return type:PTransform
class apache_beam.io.avroio.WriteToAvro(file_path_prefix, schema, codec='deflate', file_name_suffix='', num_shards=0, shard_name_template=None, mime_type='application/x-avro', use_fastavro=True)[source]

Bases: apache_beam.transforms.ptransform.PTransform

A PTransform for writing avro files.

Initialize a WriteToAvro transform.

Parameters:
  • file_path_prefix – The file path to write to. The files written will begin with this prefix, followed by a shard identifier (see num_shards), and end in a common extension, if given by file_name_suffix. In most cases, only this argument is specified and num_shards, shard_name_template, and file_name_suffix use default values.
  • schema – The schema to use (dict). If using with avro-python3 via use_fastavro=False, provide parsed schema as returned by avro.schema.Parse().
  • codec – The codec to use for block-level compression. Any string supported by the Avro specification is accepted (for example ‘null’).
  • file_name_suffix – Suffix for the files written.
  • num_shards – The number of files (shards) used for output. If not set, the service will decide on the optimal number of shards. Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.
  • shard_name_template – A template string containing placeholders for the shard number and shard count. When constructing a filename for a particular shard number, the upper-case letters ‘S’ and ‘N’ are replaced with the 0-padded shard number and shard count respectively. This argument can be ‘’ in which case it behaves as if num_shards was set to 1 and only one file will be generated. The default pattern used is ‘-SSSSS-of-NNNNN’ if None is passed as the shard_name_template.
  • mime_type – The MIME type to use for the produced files, if the filesystem supports specifying MIME types.
  • use_fastavro (bool) – is significantly faster, and is now the default.
Returns:

A WriteToAvro transform usable for writing.
expand(pcoll)[source]
display_data()[source]
annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]
default_label()
default_type_hints()
classmethod from_runner_api(proto, context)
get_type_hints()

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs)

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)
label
pipeline = None
classmethod register_urn(urn, parameter_type, constructor=None)
runner_api_requires_keyed_input()
side_inputs = ()
to_runner_api(context, has_parts=False, **extra_kwargs)
to_runner_api_parameter(unused_context)
to_runner_api_pickled(unused_context)
type_check_inputs(pvalueish)
type_check_inputs_or_outputs(pvalueish, input_or_output)
type_check_outputs(pvalueish)
with_input_types(input_type_hint)

Annotates the input type of a PTransform with a type-hint.

Parameters:input_type_hint (type) – An instance of an allowed built-in type, a custom class, or an instance of a TypeConstraint.
Raises:TypeError – If input_type_hint is not a valid type-hint. See apache_beam.typehints.typehints.validate_composite_type_param() for further details.
Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.
Return type:PTransform
with_output_types(type_hint)

Annotates the output type of a PTransform with a type-hint.

Parameters:type_hint (type) – An instance of an allowed built-in type, a custom class, or a TypeConstraint.
Raises:TypeError – If type_hint is not a valid type-hint. See validate_composite_type_param() for further details.
Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.
Return type:PTransform