apache_beam.runners.interactive.interactive_runner module

A runner that allows running of Beam pipelines interactively.

This module is experimental. No backwards-compatibility guarantees.

class apache_beam.runners.interactive.interactive_runner.InteractiveRunner(underlying_runner=None, render_option=None, skip_display=True, force_compute=True, blocking=True)

Bases: apache_beam.runners.runner.PipelineRunner

An interactive runner for Beam Python pipelines.

Allows interactively building and running Beam Python pipelines.

Constructor of InteractiveRunner.

Parameters:

• underlying_runner – (runner.PipelineRunner)
• render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.
• skip_display – (bool) whether to skip display operations when running the pipeline. Useful if running large pipelines when display is not needed.
• force_compute – (bool) whether sequential pipeline runs can use cached data of PCollections computed from the previous runs including show API invocation from interactive_beam module. If True, always run the whole pipeline and compute data for PCollections forcefully. If False, use available data and run minimum pipeline fragment to only compute data not available.
• blocking – (bool) whether the pipeline run should be blocking or not.

is_fnapi_compatible()

set_render_option(render_option)

Sets the rendering option.

Parameters:render_option – (str) this parameter decides how the pipeline graph is rendered. See display.pipeline_graph_renderer for available options.

start_session()

Start the session that keeps back-end managers and workers alive.

end_session()

End the session that keeps backend managers and workers alive.

apply(transform, pvalueish, options)

run_pipeline(pipeline, options)

apply_PTransform(transform, input, options)

run(transform, options=None)

Run the given transform or callable with this runner.

Blocks until the pipeline is complete. See also PipelineRunner.run_async.

run_async(transform, options=None)

Run the given transform or callable with this runner.

May return immediately, executing the pipeline in the background. The returned result object can be queried for progress, and wait_until_finish may be called to block until completion.

run_transform(transform_node, options)

Runner callback for a pipeline.run call.

Parameters:transform_node – transform node for the transform to run.

A concrete implementation of the Runner class must implement run_Abc for some class Abc in the method resolution order for every non-composite transform Xyz in the pipeline.

visit_transforms(pipeline, options)

class apache_beam.runners.interactive.interactive_runner.PipelineResult(underlying_result, pipeline_instrument)

Bases: apache_beam.runners.runner.PipelineResult

Provides access to information about a pipeline.

Constructor of PipelineResult.

Parameters:

• underlying_result – (PipelineResult) the result returned by the underlying runner running the pipeline.
• pipeline_instrument – (PipelineInstrument) pipeline instrument describing the pipeline being executed with interactivity applied and related metadata including where the interactivity-backing cache lies.

state

wait_until_finish()

get(pcoll, include_window_info=False)

Materializes the PCollection into a list.

If include_window_info is True, then returns the elements as WindowedValues. Otherwise, return the element as itself.

read(pcoll, include_window_info=False)

Reads the PCollection one element at a time from cache.

If include_window_info is True, then returns the elements as WindowedValues. Otherwise, return the element as itself.

cancel()

aggregated_values(aggregator_or_name)

Return a dict of step names to values of the Aggregator.

metrics()

Returns MetricResults object to query metrics from the runner.

Raises:NotImplementedError – If the runner does not support this operation.