apache_beam.runners.direct.direct_runner module

DirectRunner, executing on the local machine.

The DirectRunner is a runner implementation that executes the entire graph of transformations belonging to a pipeline on the local machine.

class apache_beam.runners.direct.direct_runner.SwitchingDirectRunner

Bases: apache_beam.runners.runner.PipelineRunner

Executes a single pipeline on the local machine.

This implementation switches between using the FnApiRunner (which has high throughput for batch jobs) and using the BundleBasedDirectRunner, which supports streaming execution and certain primitives not yet implemented in the FnApiRunner.

is_fnapi_compatible()

run_pipeline(pipeline, options)

apply(transform, input, options)

Runner callback for a pipeline.apply call.

Parameters:

• transform – the transform to apply.
• input – transform’s input (typically a PCollection).

A concrete implementation of the Runner class may want to do custom pipeline construction for a given transform. To override the behavior for a transform class Xyz, implement an apply_Xyz method with this same signature.

apply_PTransform(transform, input, options)

run(transform, options=None)

Run the given transform or callable with this runner.

Blocks until the pipeline is complete. See also PipelineRunner.run_async.

run_async(transform, options=None)

Run the given transform or callable with this runner.

May return immediately, executing the pipeline in the background. The returned result object can be queried for progress, and wait_until_finish may be called to block until completion.

run_transform(transform_node, options)

Runner callback for a pipeline.run call.

Parameters:transform_node – transform node for the transform to run.

A concrete implementation of the Runner class must implement run_Abc for some class Abc in the method resolution order for every non-composite transform Xyz in the pipeline.

visit_transforms(pipeline, options)

class apache_beam.runners.direct.direct_runner.BundleBasedDirectRunner

Bases: apache_beam.runners.runner.PipelineRunner

Executes a single pipeline on the local machine.

static is_fnapi_compatible()

run_pipeline(pipeline, options)

Execute the entire pipeline and returns an DirectPipelineResult.

apply(transform, input, options)

Runner callback for a pipeline.apply call.

Parameters:

• transform – the transform to apply.
• input – transform’s input (typically a PCollection).

A concrete implementation of the Runner class may want to do custom pipeline construction for a given transform. To override the behavior for a transform class Xyz, implement an apply_Xyz method with this same signature.

apply_PTransform(transform, input, options)

run(transform, options=None)

Run the given transform or callable with this runner.

Blocks until the pipeline is complete. See also PipelineRunner.run_async.

run_async(transform, options=None)

Run the given transform or callable with this runner.

May return immediately, executing the pipeline in the background. The returned result object can be queried for progress, and wait_until_finish may be called to block until completion.

run_transform(transform_node, options)

Runner callback for a pipeline.run call.

Parameters:transform_node – transform node for the transform to run.

A concrete implementation of the Runner class must implement run_Abc for some class Abc in the method resolution order for every non-composite transform Xyz in the pipeline.

visit_transforms(pipeline, options)

apache_beam.runners.direct.direct_runner.DirectRunner

alias of apache_beam.runners.direct.direct_runner.SwitchingDirectRunner