apache_beam.transforms.periodicsequence module¶

class apache_beam.transforms.periodicsequence.ImpulseSeqGenRestrictionProvider[source]¶

Bases: apache_beam.transforms.core.RestrictionProvider

initial_restriction(element)[source]¶

create_tracker(restriction)[source]¶

restriction_size(unused_element, restriction)[source]¶

restriction_coder()¶

Returns a Coder for restrictions.

Returned``Coder`` will be used for the restrictions produced by the current RestrictionProvider.

Returns:	an object of type `Coder`.

split(element, restriction)¶

Splits the given element and restriction initially.

This method enables runners to perform bulk splitting initially allowing for a rapid increase in parallelism. Note that initial split is a different concept from the split during element processing time. Please refer to iobase.RestrictionTracker.try_split for details about splitting when the current element and restriction are actively being processed.

Returns an iterator of restrictions. The total set of elements produced by reading input element for each of the returned restrictions should be the same as the total set of elements produced by reading the input element for the input restriction.

This API is optional if split_and_size has been implemented.

If this method is not override, there is no initial splitting happening on each restriction.

split_and_size(element, restriction)¶

Like split, but also does sizing, returning (restriction, size) pairs.

This API is optional if split and restriction_size have been implemented.

truncate(element, restriction)¶

Truncates the provided restriction into a restriction representing a finite amount of work when the pipeline is draining <https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8/edit#> for additional details about drain.>_. # pylint: disable=line-too-long By default, if the restriction is bounded then the restriction will be returned otherwise None will be returned.

This API is optional and should only be implemented if more granularity is required.

Return a truncated finite restriction if further processing is required otherwise return None to represent that no further processing of this restriction is required.

The default behavior when a pipeline is being drained is that bounded restrictions process entirely while unbounded restrictions process till a checkpoint is possible.

class apache_beam.transforms.periodicsequence.ImpulseSeqGenDoFn(*unused_args, **unused_kwargs)[source]¶

Bases: apache_beam.transforms.core.DoFn

ImpulseSeqGenDoFn fn receives tuple elements with three parts:

first_timestamp = first timestamp to output element for.
last_timestamp = last timestamp/time to output element for.
fire_interval = how often to fire an element.

For each input element received, ImpulseSeqGenDoFn fn will start generating output elements in following pattern:

if element timestamp is less than current runtime then output element.
if element timestamp is greater than current runtime, wait until next element timestamp.

ImpulseSeqGenDoFn can’t guarantee that each element is output at exact time. ImpulseSeqGenDoFn guarantees that elements would not be output prior to given runtime timestamp.

process(element, restriction_tracker=RestrictionParam(ImpulseSeqGenRestrictionProvider))[source]¶

Parameters:	element – (start_timestamp, end_timestamp, interval) restriction_tracker –
Returns:	yields elements at processing real-time intervals with value of target output timestamp for the element.

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

finish_bundle()¶: Called after a bundle of elements is processed on a worker.

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

start_bundle()¶

Called before a bundle of elements is processed on a worker.

Elements to be processed are split into bundles and distributed to workers. Before a worker calls process() on the first element of its bundle, it calls this method.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.transforms.periodicsequence.PeriodicSequence(label=None)[source]¶

Bases: apache_beam.transforms.ptransform.PTransform

PeriodicSequence transform receives tuple elements with three parts:

first_timestamp = first timestamp to output element for.
last_timestamp = last timestamp/time to output element for.
fire_interval = how often to fire an element.

For each input element received, PeriodicSequence transform will start generating output elements in following pattern:

if element timestamp is less than current runtime then output element.
if element timestamp is greater than current runtime, wait until next element timestamp.

PeriodicSequence can’t guarantee that each element is output at exact time. PeriodicSequence guarantees that elements would not be output prior to given runtime timestamp.

expand(pcoll)[source]¶

annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

classmethod from_runner_api(proto, context)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs)¶

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)¶

label¶

pipeline = None¶

classmethod register_urn(urn, parameter_type, constructor=None)¶

runner_api_requires_keyed_input()¶

side_inputs = ()¶

to_runner_api(context, has_parts=False, **extra_kwargs)¶

to_runner_api_parameter(unused_context)¶

to_runner_api_pickled(unused_context)¶

type_check_inputs(pvalueish)¶

type_check_inputs_or_outputs(pvalueish, input_or_output)¶

type_check_outputs(pvalueish)¶

with_input_types(input_type_hint)¶

Annotates the input type of a PTransform with a type-hint.

Parameters:	input_type_hint (type) – An instance of an allowed built-in type, a custom class, or an instance of a `TypeConstraint`.
Raises:	`TypeError` – If input_type_hint is not a valid type-hint. See `apache_beam.typehints.typehints.validate_composite_type_param()` for further details.
Returns:	A reference to the instance of this particular `PTransform` object. This allows chaining type-hinting related methods.
Return type:	PTransform

with_output_types(type_hint)¶

Annotates the output type of a PTransform with a type-hint.

Parameters:	type_hint (type) – An instance of an allowed built-in type, a custom class, or a `TypeConstraint`.
Raises:	`TypeError` – If type_hint is not a valid type-hint. See `validate_composite_type_param()` for further details.
Returns:	A reference to the instance of this particular `PTransform` object. This allows chaining type-hinting related methods.
Return type:	PTransform

class apache_beam.transforms.periodicsequence.PeriodicImpulse(start_timestamp=Timestamp(1617231560.015922), stop_timestamp=Timestamp(9223372036854.775000), fire_interval=360.0, apply_windowing=False)[source]¶

Bases: apache_beam.transforms.ptransform.PTransform

PeriodicImpulse transform generates an infinite sequence of elements with given runtime interval.

PeriodicImpulse transform behaves same as {@link PeriodicSequence} transform, but can be used as first transform in pipeline.

Parameters:	start_timestamp – Timestamp for first element. stop_timestamp – Timestamp after which no elements will be output. fire_interval – Interval at which to output elements. apply_windowing – Whether each element should be assigned to individual window. If false, all elements will reside in global window.

expand(pbegin)[source]¶

annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

classmethod from_runner_api(proto, context)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs)¶

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)¶

label¶

pipeline = None¶

classmethod register_urn(urn, parameter_type, constructor=None)¶

runner_api_requires_keyed_input()¶

side_inputs = ()¶

to_runner_api(context, has_parts=False, **extra_kwargs)¶

to_runner_api_parameter(unused_context)¶

to_runner_api_pickled(unused_context)¶

type_check_inputs(pvalueish)¶

type_check_inputs_or_outputs(pvalueish, input_or_output)¶

type_check_outputs(pvalueish)¶

with_input_types(input_type_hint)¶

Annotates the input type of a PTransform with a type-hint.

Parameters:	input_type_hint (type) – An instance of an allowed built-in type, a custom class, or an instance of a `TypeConstraint`.
Raises:	`TypeError` – If input_type_hint is not a valid type-hint. See `apache_beam.typehints.typehints.validate_composite_type_param()` for further details.
Returns:	A reference to the instance of this particular `PTransform` object. This allows chaining type-hinting related methods.
Return type:	PTransform

with_output_types(type_hint)¶

Annotates the output type of a PTransform with a type-hint.

Parameters:	type_hint (type) – An instance of an allowed built-in type, a custom class, or a `TypeConstraint`.
Raises:	`TypeError` – If type_hint is not a valid type-hint. See `validate_composite_type_param()` for further details.
Returns:	A reference to the instance of this particular `PTransform` object. This allows chaining type-hinting related methods.
Return type:	PTransform