Spark.Core.Internal.OpStructures

A description of the operations that can be performed on nodes and columns.

Documentation

data TransformInvariant

The invariant respected by a transform.

Depending on the value of the invariant, different optimizations may be available.

Constructors

Opaque

This operator has no special property. It may depend on the partitioning layout, the number of partitions, the order of elements in the partitions, etc. This sort of operator is unwelcome in Krapsh...

PartitioningInvariant

This operator respects the canonical partition order, but may not have the same number of elements. For example, this could be a flatMap on an RDD (filter, etc.). This operator can be used locally with the signature a -> [a]

DirectPartitioningInvariant

The strongest invariant. It respects the canonical partition order and it outputs the same number of elements. This is typically a map. This operator can be used locally with the signature a -> a

data Locality

The dynamic value of locality. There is still a tag on it, but it can be easily dropped.

Constructors

Local

The data associated to this node is local. It can be materialized and accessed by the user.

Distributed

The data associated to this node is distributed or not accessible locally. It cannot be accessed by the user.

Instances

Eq Locality

Methods

(==) :: Locality -> Locality -> Bool

(/=) :: Locality -> Locality -> Bool

Show Locality

Methods

showsPrec :: Int -> Locality -> ShowS

show :: Locality -> String

showList :: [Locality] -> ShowS

PHYSICAL OPERATORS ***********

data StandardOperator

An operator defined by default in the release of Krapsh. All other physical operators can be converted to a standard operators.

Constructors

StandardOperator

Fields

• soName :: !Text
• soOutputType :: !DataType
• soExtra :: !Value

Instances

Eq StandardOperator

Methods

(==) :: StandardOperator -> StandardOperator -> Bool

(/=) :: StandardOperator -> StandardOperator -> Bool

Show StandardOperator

Methods

showsPrec :: Int -> StandardOperator -> ShowS

show :: StandardOperator -> String

showList :: [StandardOperator] -> ShowS

data ScalaStaticFunctionApplication

A scala method of a singleton object.

Constructors

ScalaStaticFunctionApplication

Fields

• sfaObjectName :: !Text
• sfaMethodName :: !Text

data ColOp

The different kinds of column operations. These operations describe the physical operations on columns as supported by Spark SQL. They can operate on column -> column, column -> row, row->row. Of course, not all operators are valid for each configuration.

Constructors

ColExtraction !FieldPath

A projection onto a single column An extraction is always direct.

ColFunction !Text !(Vector ColOp)

A function of other columns. In this case, the other columns may matter TODO(kps) add if this function is partition invariant. It should be the case most of the time.

ColLit !DataType !Value

A constant defined for each element. The type should be the same as for the column A literal is always direct

ColStruct !(Vector TransformField)

A structure.

Instances

Eq ColOp

Methods

(==) :: ColOp -> ColOp -> Bool

(/=) :: ColOp -> ColOp -> Bool

Show ColOp

Methods

showsPrec :: Int -> ColOp -> ShowS

show :: ColOp -> String

showList :: [ColOp] -> ShowS

data TransformField

A field in a structure.

Constructors

TransformField

Fields

• tfName :: !FieldName
• tfValue :: !ColOp

Instances

Eq TransformField

Methods

(==) :: TransformField -> TransformField -> Bool

(/=) :: TransformField -> TransformField -> Bool

Show TransformField

Methods

showsPrec :: Int -> TransformField -> ShowS

show :: TransformField -> String

showList :: [TransformField] -> ShowS

data StructuredTransform

The content of a structured transform.

Constructors

InnerOp !ColOp

InnerStruct !(Vector TransformField)

Instances

Eq StructuredTransform

Methods

(==) :: StructuredTransform -> StructuredTransform -> Bool

(/=) :: StructuredTransform -> StructuredTransform -> Bool

Show StructuredTransform

Methods

showsPrec :: Int -> StructuredTransform -> ShowS

show :: StructuredTransform -> String

showList :: [StructuredTransform] -> ShowS

DATASET OPERATORS ************

data DatasetTransformDesc

Constructors

DSScalaStaticFunction !ScalaStaticFunctionApplication

DSStructuredTransform !ColOp

DSOperator !StandardOperator

OBSERVABLE OPERATORS *******

AGGREGATION OPERATORS *****

data UniversalAggregatorOp

Constructors

UniversalAggregatorOp

Fields

• uaoMergeType :: !DataType
• uaoInitialOuter :: !StandardOperator
• uaoMergeBuffer :: !StandardOperator

Instances

Eq UniversalAggregatorOp

Methods

(==) :: UniversalAggregatorOp -> UniversalAggregatorOp -> Bool

(/=) :: UniversalAggregatorOp -> UniversalAggregatorOp -> Bool

Show UniversalAggregatorOp

Methods

showsPrec :: Int -> UniversalAggregatorOp -> ShowS

show :: UniversalAggregatorOp -> String

showList :: [UniversalAggregatorOp] -> ShowS

data NodeOp

Constructors

NodeLocalOp StandardOperator

An operation between local nodes: [Observable] -> Observable

NodeLocalLit !DataType !Value

An observable literal

NodeOpaqueAggregator StandardOperator

Some aggregator that does not respect any particular invariant.

NodeUniversalAggregator UniversalAggregatorOp

A universal aggregator.

NodeStructuredTransform !ColOp

A structured transform, performed either on a local node or a distributed node.

NodeDistributedLit !DataType !(Vector Value)

A distributed dataset (with no partition information)

NodeDistributedOp StandardOperator

An opaque distributed operator.

Instances

Eq NodeOp

Methods

(==) :: NodeOp -> NodeOp -> Bool

(/=) :: NodeOp -> NodeOp -> Bool

Show NodeOp

Methods

showsPrec :: Int -> NodeOp -> ShowS

show :: NodeOp -> String

showList :: [NodeOp] -> ShowS

makeOperator :: Text -> SQLType a -> StandardOperator

Makes a standard operator with no extra value