apache_beam.runners.dataflow.ptransform_overrides module¶

Ptransform overrides for DataflowRunner.

class apache_beam.runners.dataflow.ptransform_overrides.CreatePTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Create in streaming mode.

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(applied_ptransform)[source]¶

get_replacement_inputs(applied_ptransform)¶

Provides inputs that will be passed to the replacement PTransform.

Parameters:	applied_ptransform – Original AppliedPTransform containing the PTransform to be replaced.
Returns:	An iterable of PValues that will be passed to the expand() method of the replacement PTransform.

get_replacement_transform(ptransform)¶

Provides a runner specific override for a given PTransform.

Parameters:	ptransform – PTransform to be replaced.
Returns:	A PTransform that will be the replacement for the PTransform given as an argument.

class apache_beam.runners.dataflow.ptransform_overrides.ReadPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Read(BoundedSource)

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(applied_ptransform)[source]¶

get_replacement_inputs(applied_ptransform)¶

Provides inputs that will be passed to the replacement PTransform.

Parameters:	applied_ptransform – Original AppliedPTransform containing the PTransform to be replaced.
Returns:	An iterable of PValues that will be passed to the expand() method of the replacement PTransform.

get_replacement_transform(ptransform)¶

Provides a runner specific override for a given PTransform.

Parameters:	ptransform – PTransform to be replaced.
Returns:	A PTransform that will be the replacement for the PTransform given as an argument.

class apache_beam.runners.dataflow.ptransform_overrides.JrhReadPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Read(BoundedSource)

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(applied_ptransform)[source]¶

get_replacement_inputs(applied_ptransform)¶

Provides inputs that will be passed to the replacement PTransform.

Parameters:	applied_ptransform – Original AppliedPTransform containing the PTransform to be replaced.
Returns:	An iterable of PValues that will be passed to the expand() method of the replacement PTransform.

get_replacement_transform(ptransform)¶

Provides a runner specific override for a given PTransform.

Parameters:	ptransform – PTransform to be replaced.
Returns:	A PTransform that will be the replacement for the PTransform given as an argument.

class apache_beam.runners.dataflow.ptransform_overrides.CombineValuesPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for CombineValues.

The DataflowRunner expects that the CombineValues PTransform acts as a primitive. So this override replaces the CombineValues with a primitive.

matches(applied_ptransform)[source]¶

get_replacement_transform(ptransform)[source]¶

get_replacement_inputs(applied_ptransform)¶

Provides inputs that will be passed to the replacement PTransform.

Parameters:	applied_ptransform – Original AppliedPTransform containing the PTransform to be replaced.
Returns:	An iterable of PValues that will be passed to the expand() method of the replacement PTransform.

get_replacement_transform_for_applied_ptransform(applied_ptransform)¶

Provides a runner specific override for a given AppliedPTransform.

Parameters:	applied_ptransform – AppliedPTransform containing the PTransform to be replaced.
Returns:	A PTransform that will be the replacement for the PTransform inside the AppliedPTransform given as an argument.

class apache_beam.runners.dataflow.ptransform_overrides.NativeReadPTransformOverride[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for Read using native sources.

The DataflowRunner expects that the Read PTransform using native sources act as a primitive. So this override replaces the Read with a primitive.

matches(applied_ptransform)[source]¶

get_replacement_transform(ptransform)[source]¶

get_replacement_inputs(applied_ptransform)¶

Provides inputs that will be passed to the replacement PTransform.

Parameters:	applied_ptransform – Original AppliedPTransform containing the PTransform to be replaced.
Returns:	An iterable of PValues that will be passed to the expand() method of the replacement PTransform.

get_replacement_transform_for_applied_ptransform(applied_ptransform)¶

Provides a runner specific override for a given AppliedPTransform.

Parameters:	applied_ptransform – AppliedPTransform containing the PTransform to be replaced.
Returns:	A PTransform that will be the replacement for the PTransform inside the AppliedPTransform given as an argument.

class apache_beam.runners.dataflow.ptransform_overrides.WriteToBigQueryPTransformOverride(pipeline, options)[source]¶

Bases: apache_beam.pipeline.PTransformOverride

matches(applied_ptransform)[source]¶

get_replacement_transform(ptransform)[source]¶

get_replacement_inputs(applied_ptransform)¶

Provides inputs that will be passed to the replacement PTransform.

Parameters:	applied_ptransform – Original AppliedPTransform containing the PTransform to be replaced.
Returns:	An iterable of PValues that will be passed to the expand() method of the replacement PTransform.

get_replacement_transform_for_applied_ptransform(applied_ptransform)¶

Provides a runner specific override for a given AppliedPTransform.

Parameters:	applied_ptransform – AppliedPTransform containing the PTransform to be replaced.
Returns:	A PTransform that will be the replacement for the PTransform inside the AppliedPTransform given as an argument.

class apache_beam.runners.dataflow.ptransform_overrides.GroupIntoBatchesWithShardedKeyPTransformOverride(dataflow_runner, options)[source]¶

Bases: apache_beam.pipeline.PTransformOverride

A PTransformOverride for GroupIntoBatches.WithShardedKey.

This override simply returns the original transform but additionally records the output PCollection in order to append required step properties during graph translation.

matches(applied_ptransform)[source]¶

get_replacement_transform_for_applied_ptransform(ptransform)[source]¶

get_replacement_inputs(applied_ptransform)¶

Provides inputs that will be passed to the replacement PTransform.

Parameters:	applied_ptransform – Original AppliedPTransform containing the PTransform to be replaced.
Returns:	An iterable of PValues that will be passed to the expand() method of the replacement PTransform.

get_replacement_transform(ptransform)¶

Provides a runner specific override for a given PTransform.

Parameters:	ptransform – PTransform to be replaced.
Returns:	A PTransform that will be the replacement for the PTransform given as an argument.