Spark.Core.Context

This module defines session objects that act as entry points to spark.

There are two ways to interact with Spark: using an explicit state object, or using the default state object (interactive session).

While the interactive session is the most convenient, it should not be used for more than quick experimentations. Any complex code should use the SparkSession and SparkState objects.

Documentation

data SparkSessionConf

The configuration of a remote spark session in krapsh.

Constructors

SparkSessionConf

Fields

• confEndPoint :: !Text

  The URL of the end point.

• confPort :: !Int

  The port used to configure the end point.

• confPollingIntervalMillis :: !Int

  (internal) the polling interval

• confRequestedSessionName :: !Text

  (optional) the requested name of the session. This name must obey a number of rules: - it must consist in alphanumerical and -,_: [a-zA-Z0-9-_] - if it already exists on the server, it will be reconnected to

  The default value is "" (a new random context name will be chosen).

Instances

Show SparkSessionConf

Methods

showsPrec :: Int -> SparkSessionConf -> ShowS

show :: SparkSessionConf -> String

showList :: [SparkSessionConf] -> ShowS

data SparkSession

A session in Spark. Encapsualates all the state needed to communicate with Spark and to perfor some simple optimizations on the code.

Instances

Show SparkSession

Methods

showsPrec :: Int -> SparkSession -> ShowS

show :: SparkSession -> String

showList :: [SparkSession] -> ShowS

type SparkState a = LoggingT (StateT SparkSession IO) a

Represents the state of a session and accounts for the communication with the server.

data SparkInteractiveException

The exception thrown when a request cannot be completed in an interactive session.

Instances

Show SparkInteractiveException

Methods

showsPrec :: Int -> SparkInteractiveException -> ShowS

show :: SparkInteractiveException -> String

showList :: [SparkInteractiveException] -> ShowS

Exception SparkInteractiveException

Methods

toException :: SparkInteractiveException -> SomeException

fromException :: SomeException -> Maybe SparkInteractiveException

displayException :: SparkInteractiveException -> String

class FromSQL a

Instances

FromSQL Int

Methods

_cellToValue :: Cell -> TryS Int

FromSQL Cell

Methods

_cellToValue :: Cell -> TryS Cell

FromSQL a => FromSQL [a]

Methods

_cellToValue :: Cell -> TryS [a]

FromSQL a => FromSQL (Maybe a)

Methods

_cellToValue :: Cell -> TryS (Maybe a)

defaultConf :: SparkSessionConf

The default configuration if the krapsh server is being run locally.

executeCommand1 :: forall a. (FromSQL a, HasCallStack) => LocalData a -> SparkState (Try a)

Executes a command: - performs the transforms and the optimizations in the pure state - sends the computation to the backend - waits for the terminal nodes to reach a final state - commits the final results to the state

If any failure is detected that is internal to Krapsh, it returns an error. If the error comes from an underlying library (http stack, programming failure), an exception may be thrown instead.

createSparkSessionDef :: HasCallStack => SparkSessionConf -> IO ()

Creates a spark session that will be used as the default session.

If a session already exists, an exception will be thrown.

closeSparkSessionDef :: HasCallStack => IO ()

Closes the default session. The default session is empty after this call completes.

NOTE: This does not currently clear up the resources! It is a stub implementation used in testing.

exec1Def :: (FromSQL a, HasCallStack) => LocalData a -> IO a

Executes a command using the default spark session.

This is the most unsafe way of running a command: it executes a command using the default spark session, and throws an exception if any error happens.