Safe Haskell | None |
---|---|
Language | Haskell2010 |
Spark.Core.Internal.TypesStructures
Description
The structures of data types in Krapsh.
For a detailed description of the supported types, see http:/spark.apache.orgdocslatestsql-programming-guide.html#data-types
At a high-level, Spark DataFrames and Datasets are equivalent to lists of objects whose type can be mapped to the same StructType: Dataset a ~ ArrayType StructType (...) Columns of a dataset are equivalent to lists of object whose type can be mapped to the same DataType (either Strict or Nullable) Local data (or "blobs") are single elements whose type can be mapped to a DataType (either strict or nullable)
- data StrictDataType
- = IntType
- | StringType
- | Struct !StructType
- | ArrayType {
- elementType :: !DataType
- data DataType
- data StructField = StructField {}
- data StructType = StructType {
- structFields :: !(Vector StructField)
- data Nullable
- data NullableDataType = NullableDataType !StrictDataType
- data SQLType a = SQLType {}
- _fieldToJson :: StructField -> (Text, Value)
Documentation
data StrictDataType Source #
The data types that are guaranteed to not be null: evaluating them will return a value.
Constructors
IntType | |
StringType | |
Struct !StructType | |
ArrayType | |
Fields
|
All the data types supported by the Spark engine. The data types can either be nullable (they may contain null values) or strict (all the values are present). There are a couple of differences with the algebraic data types in Haskell: Maybe (Maybe a) ~ Maybe a which implies that arbitrary nesting of values will be flattened to a top-level Nullable Similarly, [[]] ~ []
Constructors
StrictType !StrictDataType | |
NullableType !StrictDataType |
data StructType Source #
The main structure of a dataframe or a dataset
Constructors
StructType | |
Fields
|
Instances
Represents the choice between a strict and a nullable field
data NullableDataType Source #
Encodes the type of all the nullable data types
Constructors
NullableDataType !StrictDataType |
Instances
A tagged datatype that encodes the sql types This is the main type information that should be used by users.
_fieldToJson :: StructField -> (Text, Value) Source #