Spark.Core.Internal.ContextInternal

Documentation

type FinalResult = Either NodeComputationFailure NodeComputationSuccess

prepareExecution1 :: LocalData a -> SparkStatePure (Try Computation)

buildComputationGraph :: ComputeNode loc a -> Try ComputeGraph

Builds the computation graph by expanding a single node until a transitive closure is reached.

It performs the naming, node deduplication and cycle detection.

TODO(kps) use the caching information to have a correct fringe

performGraphTransforms :: ComputeGraph -> Try ComputeGraph

Performs all the operations that are done on the compute graph:

• fullfilling autocache requests
• checking the cache/uncache pairs
• deconstructions of the unions (in the future)

This could all be done on the server side at this point.

getTargetNodes :: HasCallStack => Computation -> [UntypedLocalData]

storeResults :: Computation -> [(LocalData Cell, FinalResult)] -> SparkStatePure (Try Cell)