apache_beam.transforms.sql module

Package for SqlTransform and related classes.

class apache_beam.transforms.sql.SqlTransform(query, dialect=None, expansion_service=None)

Bases: apache_beam.transforms.external.ExternalTransform

A transform that can translate a SQL query into PTransforms.

Input PCollections must have a schema. Currently, there are two ways to define a schema for a PCollection:

1. Register a typing.NamedTuple type to use RowCoder, and specify it as the output type. For example:

   Purchase = typing.NamedTuple('Purchase',
                                [('item_name', unicode), ('price', float)])
   coders.registry.register_coder(Purchase, coders.RowCoder)
   with Pipeline() as p:
     purchases = (p | beam.io...
                    | beam.Map(..).with_output_types(Purchase))
   

2. Produce beam.Row instances. Note this option will fail if Beam is unable to infer data types for any of the fields. For example:

   with Pipeline() as p:
     purchases = (p | beam.io...
                    | beam.Map(lambda x: beam.Row(item_name=unicode(..),
                                                  price=float(..))))
   

Similarly, the output of SqlTransform is a PCollection with a schema. The columns produced by the query can be accessed as attributes. For example:

purchases | SqlTransform("""
              SELECT item_name, COUNT(*) AS `count`
              FROM PCOLLECTION GROUP BY item_name""")
          | beam.Map(lambda row: "We've sold %d %ss!" % (row.count,
                                                         row.item_name))

Additional examples can be found in apache_beam.examples.wordcount_xlang_sql, apache_beam.examples.sql_taxi, and apache_beam.transforms.sql_test.

For more details about Beam SQL in general see the Java transform, and the documentation.

Creates a SqlTransform which will be expanded to Java’s SqlTransform. (See class docs). :param query: The SQL query. :param dialect: (optional) The dialect, e.g. use ‘zetasql’ for ZetaSQL. :param expansion_service: (optional) The URL of the expansion service to use

URN = 'beam:external:java:sql:v1'

annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]

default_label()

default_type_hints()

display_data()

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

Return type:Dict[str, Any]

expand(pvalueish)

classmethod from_runner_api(proto, context)

classmethod get_local_namespace()

get_type_hints()

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs)

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)

label

classmethod outer_namespace(namespace)

pipeline = None

classmethod register_urn(urn, parameter_type, constructor=None)

runner_api_requires_keyed_input()

side_inputs = ()

to_runner_api(context, has_parts=False, **extra_kwargs)

to_runner_api_parameter(unused_context)

to_runner_api_pickled(unused_context)

to_runner_api_transform(context, full_label)

type_check_inputs(pvalueish)

type_check_inputs_or_outputs(pvalueish, input_or_output)

type_check_outputs(pvalueish)

with_input_types(input_type_hint)

Annotates the input type of a PTransform with a type-hint.

Parameters:input_type_hint (type) – An instance of an allowed built-in type, a custom class, or an instance of a TypeConstraint. Raises:TypeError – If input_type_hint is not a valid type-hint. See apache_beam.typehints.typehints.validate_composite_type_param() for further details. Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods. Return type:PTransform

with_output_types(type_hint)

Annotates the output type of a PTransform with a type-hint.

Parameters:type_hint (type) – An instance of an allowed built-in type, a custom class, or a TypeConstraint. Raises:TypeError – If type_hint is not a valid type-hint. See validate_composite_type_param() for further details. Returns:A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods. Return type:PTransform