apache_beam.io.gcp.bigquery_file_loads module¶

Functionality to perform file loads into BigQuery for Batch and Streaming pipelines.

This source is able to work around BigQuery load quotas and limitations. When destinations are dynamic, or when data for a single job is too large, the data will be split into multiple jobs.

NOTHING IN THIS FILE HAS BACKWARDS COMPATIBILITY GUARANTEES.

apache_beam.io.gcp.bigquery_file_loads.file_prefix_generator(with_validation=True, pipeline_gcs_location=None, temp_location=None)[source]¶

class apache_beam.io.gcp.bigquery_file_loads.WriteRecordsToFile(schema, max_files_per_bundle=20, max_file_size=4398046511104, file_format=None)[source]¶

Bases: apache_beam.transforms.core.DoFn

Write input records to files before triggering a load job.

This transform keeps up to max_files_per_bundle files open to write to. It receives (destination, record) tuples, and it writes the records to different files for each destination.

If there are more than max_files_per_bundle destinations that we need to write to, then those records are grouped by their destination, and later written to files by WriteGroupedRecordsToFile.

It outputs two PCollections.

Initialize a WriteRecordsToFile.

Parameters:	max_files_per_bundle (int) – The maximum number of files that can be kept open during execution of this step in a worker. This is to avoid over- whelming the worker memory. max_file_size (int) – The maximum size in bytes for a file to be used in an export job.

UNWRITTEN_RECORD_TAG = 'UnwrittenRecords'¶

WRITTEN_FILE_TAG = 'WrittenFiles'¶

display_data()[source]¶

start_bundle()[source]¶

process(element, file_prefix, *schema_side_inputs)[source]¶

Take a tuple with (destination, row) and write to file or spill out.

Destination may be a TableReference or a string, and row is a Python dictionary for a row to be inserted to BigQuery.

finish_bundle()[source]¶

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.io.gcp.bigquery_file_loads.WriteGroupedRecordsToFile(schema, max_file_size=4398046511104, file_format=None)[source]¶

Bases: apache_beam.transforms.core.DoFn

Receives collection of dest-iterable(records), writes it to files.

This is different from WriteRecordsToFile because it receives records grouped by destination. This means that it’s not necessary to keep multiple file descriptors open, because we know for sure when records for a single destination have been written out.

Experimental; no backwards compatibility guarantees.

process(element, file_prefix, *schema_side_inputs)[source]¶

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

finish_bundle()¶: Called after a bundle of elements is processed on a worker.

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

start_bundle()¶

Called before a bundle of elements is processed on a worker.

Elements to be processed are split into bundles and distributed to workers. Before a worker calls process() on the first element of its bundle, it calls this method.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.io.gcp.bigquery_file_loads.TriggerCopyJobs(create_disposition=None, write_disposition=None, test_client=None, step_name=None)[source]¶

Bases: apache_beam.transforms.core.DoFn

Launches jobs to copy from temporary tables into the main target table.

When a job needs to write to multiple destination tables, or when a single destination table needs to have multiple load jobs to write to it, files are loaded into temporary tables, and those tables are later copied to the destination tables.

This transform emits (destination, job_reference) pairs.

TODO(BEAM-7822): In file loads method of writing to BigQuery,: copying from temp_tables to destination_table is not atomic. See: https://issues.apache.org/jira/browse/BEAM-7822

display_data()[source]¶

start_bundle()[source]¶

process(element, job_name_prefix=None)[source]¶

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

finish_bundle()¶: Called after a bundle of elements is processed on a worker.

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.io.gcp.bigquery_file_loads.TriggerLoadJobs(schema=None, create_disposition=None, write_disposition=None, test_client=None, temporary_tables=False, additional_bq_parameters=None, source_format=None, step_name=None)[source]¶

Bases: apache_beam.transforms.core.DoFn

Triggers the import jobs to BQ.

Experimental; no backwards compatibility guarantees.

TEMP_TABLES = 'TemporaryTables'¶

display_data()[source]¶

start_bundle()[source]¶

process(element, load_job_name_prefix, *schema_side_inputs)[source]¶

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

finish_bundle()¶: Called after a bundle of elements is processed on a worker.

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.io.gcp.bigquery_file_loads.PartitionFiles(max_partition_size, max_files_per_partition)[source]¶

Bases: apache_beam.transforms.core.DoFn

MULTIPLE_PARTITIONS_TAG = 'MULTIPLE_PARTITIONS'¶

SINGLE_PARTITION_TAG = 'SINGLE_PARTITION'¶

class Partition(max_size, max_files, files=None, size=0)[source]¶

Bases: object

can_accept(file_size, no_of_files=1)[source]¶

add(file_path, file_size)[source]¶

process(element)[source]¶

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

finish_bundle()¶: Called after a bundle of elements is processed on a worker.

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

start_bundle()¶

Called before a bundle of elements is processed on a worker.

Elements to be processed are split into bundles and distributed to workers. Before a worker calls process() on the first element of its bundle, it calls this method.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.io.gcp.bigquery_file_loads.WaitForBQJobs(test_client=None)[source]¶

Bases: apache_beam.transforms.core.DoFn

Takes in a series of BQ job names as side input, and waits for all of them.

If any job fails, it will fail. If all jobs succeed, it will succeed.

Experimental; no backwards compatibility guarantees.

start_bundle()[source]¶

process(element, dest_ids_list)[source]¶

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

finish_bundle()¶: Called after a bundle of elements is processed on a worker.

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.io.gcp.bigquery_file_loads.DeleteTablesFn(test_client=None)[source]¶

Bases: apache_beam.transforms.core.DoFn

start_bundle()[source]¶

process(table_reference)[source]¶

BundleFinalizerParam¶: alias of apache_beam.transforms.core._BundleFinalizerParam

DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam, WindowParam, <class 'apache_beam.transforms.core._WatermarkEstimatorParam'>, PaneInfoParam, <class 'apache_beam.transforms.core._BundleFinalizerParam'>, KeyParam, <class 'apache_beam.transforms.core._StateDoFnParam'>, <class 'apache_beam.transforms.core._TimerDoFnParam'>]¶

DynamicTimerTagParam = DynamicTimerTagParam¶

ElementParam = ElementParam¶

KeyParam = KeyParam¶

PaneInfoParam = PaneInfoParam¶

RestrictionParam¶: alias of apache_beam.transforms.core._RestrictionDoFnParam

SideInputParam = SideInputParam¶

StateParam¶: alias of apache_beam.transforms.core._StateDoFnParam

TimerParam¶: alias of apache_beam.transforms.core._TimerDoFnParam

TimestampParam = TimestampParam¶

WatermarkEstimatorParam¶: alias of apache_beam.transforms.core._WatermarkEstimatorParam

WindowParam = WindowParam¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

finish_bundle()¶: Called after a bundle of elements is processed on a worker.

static from_callable(fn)¶

classmethod from_runner_api(fn_proto, context)¶

Converts from an FunctionSpec to a Fn object.

Prefer registering a urn with its parameter type and constructor.

get_function_arguments(func)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

infer_output_type(input_type)¶

classmethod register_pickle_urn(pickle_urn)¶: Registers and implements the given urn via pickling.

classmethod register_urn(urn, parameter_type, fn=None)¶

Registers a urn with a constructor.

For example, if ‘beam:fn:foo’ had parameter type FooPayload, one could write RunnerApiFn.register_urn(‘bean:fn:foo’, FooPayload, foo_from_proto) where foo_from_proto took as arguments a FooPayload and a PipelineContext. This function can also be used as a decorator rather than passing the callable in as the final parameter.

A corresponding to_runner_api_parameter method would be expected that returns the tuple (‘beam:fn:foo’, FooPayload)

setup()¶

Called to prepare an instance for processing bundles of elements.

This is a good place to initialize transient in-memory resources, such as network connections. The resources can then be disposed in DoFn.teardown.

teardown()¶

Called to use to clean up this instance before it is discarded.

A runner will do its best to call this method on any given instance to prevent leaks of transient resources, however, there may be situations where this is impossible (e.g. process crash, hardware failure, etc.) or unnecessary (e.g. the pipeline is shutting down and the process is about to be killed anyway, so all transient resources will be released automatically by the OS). In these cases, the call may not happen. It will also not be retried, because in such situations the DoFn instance no longer exists, so there’s no instance to retry it on.

Thus, all work that depends on input elements, and all externally important side effects, must be performed in DoFn.process or DoFn.finish_bundle.

to_runner_api(context)¶

Returns an FunctionSpec encoding this Fn.

Prefer overriding self.to_runner_api_parameter.

to_runner_api_parameter(context)¶

static unbounded_per_element()¶: A decorator on process fn specifying that the fn performs an unbounded amount of work per input element.

with_input_types(*arg_hints, **kwarg_hints)¶

with_output_types(*arg_hints, **kwarg_hints)¶

class apache_beam.io.gcp.bigquery_file_loads.BigQueryBatchFileLoads(destination, schema=None, custom_gcs_temp_location=None, create_disposition=None, write_disposition=None, triggering_frequency=None, with_auto_sharding=False, temp_file_format=None, max_file_size=None, max_files_per_bundle=None, max_partition_size=None, max_files_per_partition=None, additional_bq_parameters=None, table_side_inputs=None, schema_side_inputs=None, test_client=None, validate=True, is_streaming_pipeline=False)[source]¶

Bases: apache_beam.transforms.ptransform.PTransform

Takes in a set of elements, and inserts them to BigQuery via batch loads.

DESTINATION_JOBID_PAIRS = 'destination_load_jobid_pairs'¶

DESTINATION_FILE_PAIRS = 'destination_file_pairs'¶

DESTINATION_COPY_JOBID_PAIRS = 'destination_copy_jobid_pairs'¶

COUNT = 0¶

verify()[source]¶

annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]¶

default_label()¶

default_type_hints()¶

display_data()¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type: Dict[str, Any]

classmethod from_runner_api(proto, context)¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs)¶

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)¶

label¶

pipeline = None¶

classmethod register_urn(urn, parameter_type, constructor=None)¶

runner_api_requires_keyed_input()¶

side_inputs = ()¶

to_runner_api(context, has_parts=False, **extra_kwargs)¶

to_runner_api_parameter(unused_context)¶

to_runner_api_pickled(unused_context)¶

type_check_inputs(pvalueish)¶

type_check_inputs_or_outputs(pvalueish, input_or_output)¶

type_check_outputs(pvalueish)¶

with_input_types(input_type_hint)¶

Annotates the input type of a PTransform with a type-hint.

Parameters:	input_type_hint (type) – An instance of an allowed built-in type, a custom class, or an instance of a `TypeConstraint`.
Raises:	`TypeError` – If input_type_hint is not a valid type-hint. See `apache_beam.typehints.typehints.validate_composite_type_param()` for further details.
Returns:	A reference to the instance of this particular `PTransform` object. This allows chaining type-hinting related methods.
Return type:	PTransform

with_output_types(type_hint)¶

Annotates the output type of a PTransform with a type-hint.

Parameters:	type_hint (type) – An instance of an allowed built-in type, a custom class, or a `TypeConstraint`.
Raises:	`TypeError` – If type_hint is not a valid type-hint. See `validate_composite_type_param()` for further details.
Returns:	A reference to the instance of this particular `PTransform` object. This allows chaining type-hinting related methods.
Return type:	PTransform

expand(pcoll)[source]¶