Streamly.Internal.Data.Array

Setup

>>> :m
>>> :set -XFlexibleContexts
>>> :set -XMagicHash
>>> import Data.Function ((&))
>>> import Data.Functor.Identity (Identity(..))
>>> import System.IO.Unsafe (unsafePerformIO)

>>> import Streamly.Data.Array (Array)
>>> import Streamly.Data.Stream (Stream)

>>> import qualified Streamly.Data.Array as Array
>>> import qualified Streamly.Data.Fold as Fold
>>> import qualified Streamly.Data.Stream as Stream

For APIs that have not been released yet.

>>> import qualified Streamly.Internal.Data.Array as Array

Design Notes

To summarize:

• Arrays are finite and fixed in size
• provide O(1) access to elements
• store only data and not functions
• provide efficient IO interfacing

Foldable instance is not provided because the implementation would be much less efficient compared to folding via streams. Semigroup and Monoid instances should be used with care; concatenating arrays using binary operations can be highly inefficient. Instead, use toArray to concatenate N arrays at once.

Each array is one pointer visible to the GC. Too many small arrays (e.g. single byte) are only as good as holding those elements in a Haskell list. However, small arrays can be compacted into large ones to reduce the overhead. To hold 32GB memory in 32k sized buffers we need 1 million arrays if we use one array for each chunk. This is still significant to add pressure to GC.

The Array Type

Type

We can use an Unbox constraint in the Array type and the constraint can be automatically provided to a function that pattern matches on the Array type. However, it has huge performance cost, so we do not use it. Investigate a GHC improvement possiblity.

data Array a

Constructors

Array

Fields

• arrContents :: !MutByteArray
• arrStart :: !Int
• arrEnd :: !Int

Instances

a ~ Char => IsString (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

fromString :: String -> Array a

Unbox a => Monoid (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

mempty :: Array a

mappend :: Array a -> Array a -> Array a

mconcat :: [Array a] -> Array a

Unbox a => Semigroup (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

(<>) :: Array a -> Array a -> Array a

sconcat :: NonEmpty (Array a) -> Array a

stimes :: Integral b => b -> Array a -> Array a

Unbox a => IsList (Array a)

Defined in Streamly.Internal.Data.Array.Type

Associated Types

type Item (Array a)

Methods

fromList :: [Item (Array a)] -> Array a

fromListN :: Int -> [Item (Array a)] -> Array a

toList :: Array a -> [Item (Array a)]

(Unbox a, Read a, Show a) => Read (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

readsPrec :: Int -> ReadS (Array a)

readList :: ReadS [Array a]

readPrec :: ReadPrec (Array a)

readListPrec :: ReadPrec [Array a]

(Show a, Unbox a) => Show (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

showsPrec :: Int -> Array a -> ShowS

show :: Array a -> String

showList :: [Array a] -> ShowS

(Unbox a, Eq a) => Eq (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

(==) :: Array a -> Array a -> Bool

(/=) :: Array a -> Array a -> Bool

(Unbox a, Ord a) => Ord (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

compare :: Array a -> Array a -> Ordering

(<) :: Array a -> Array a -> Bool

(<=) :: Array a -> Array a -> Bool

(>) :: Array a -> Array a -> Bool

(>=) :: Array a -> Array a -> Bool

max :: Array a -> Array a -> Array a

min :: Array a -> Array a -> Array a

Serialize (Array a)

Defined in Streamly.Internal.Data.Serialize.Type

Methods

addSizeTo :: Int -> Array a -> Int

deserializeAt :: Int -> MutByteArray -> Int -> IO (Int, Array a)

serializeAt :: Int -> MutByteArray -> Array a -> IO Int

type Item (Array a)

Defined in Streamly.Internal.Data.Array.Type

type Item (Array a) = a

asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b

Use an Array a as Ptr a.

See asPtrUnsafe in the Mutable array module for more details.

Unsafe

Pre-release

nil :: Array a

Freezing and Thawing

unsafeFreeze :: MutArray a -> Array a

Makes an immutable array using the underlying memory of the mutable array.

Please make sure that there are no other references to the mutable array lying around, so that it is never used after freezing it using unsafeFreeze. If the underlying array is mutated, the immutable promise is lost.

Pre-release

unsafeFreezeWithShrink :: Unbox a => MutArray a -> Array a

Similar to unsafeFreeze but uses rightSize on the mutable array first.

unsafeThaw :: Array a -> MutArray a

Makes a mutable array using the underlying memory of the immutable array.

Please make sure that there are no other references to the immutable array lying around, so that it is never used after thawing it using unsafeThaw. If the resulting array is mutated, any references to the older immutable array are mutated as well.

Pre-release

Pinning and Unpinning

pin :: Array a -> IO (Array a)

Return a copy of the Array in pinned memory if unpinned, else return the original array.

unpin :: Array a -> IO (Array a)

Return a copy of the Array in unpinned memory if pinned, else return the original array.

isPinned :: Array a -> Bool

Return True if the array is allocated in pinned memory.

Construction

splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)

fromList :: Unbox a => [a] -> Array a

Create an Array from a list. The list must be of finite size.

pinnedFromList :: Unbox a => [a] -> Array a

Like fromList but creates a pinned array.

fromListN :: Unbox a => Int -> [a] -> Array a

Create an Array from the first N elements of a list. The array is allocated to size N, if the list terminates before N elements then the array may hold less than N elements.

pinnedFromListN :: Unbox a => Int -> [a] -> Array a

Like fromListN but creates a pinned array.

fromListRev :: Unbox a => [a] -> Array a

Create an Array from a list in reverse order. The list must be of finite size.

Pre-release

fromListRevN :: Unbox a => Int -> [a] -> Array a

Create an Array from the first N elements of a list in reverse order. The array is allocated to size N, if the list terminates before N elements then the array may hold less than N elements.

Pre-release

fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)

fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)

fromPureStream :: Unbox a => Stream Identity a -> Array a

Convert a pure stream in Identity monad to an immutable array.

Same as the following but with better performance:

>>> fromPureStream = Array.fromList . runIdentity . Stream.toList

fromByteStr# :: Addr# -> Array Word8

Copy a null terminated immutable Addr# Word8 sequence into an array.

Unsafe: The caller is responsible for safe addressing.

Note that this is completely safe when reading from Haskell string literals because they are guaranteed to be NULL terminated:

>>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#
[1,2,3]

Split

breakOn :: MonadIO m => Word8 -> Array Word8 -> m (Array Word8, Maybe (Array Word8))

Cloning arrays

clone :: MonadIO m => Array a -> m (Array a)

pinnedClone :: MonadIO m => Array a -> m (Array a)

Elimination

unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a

Return element at the specified index without checking the bounds.

Unsafe because it does not check the bounds of the array.

getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a

Return element at the specified index without checking the bounds.

byteLength :: Array a -> Int

O(1) Get the byte length of the array.

length :: Unbox a => Array a -> Int

O(1) Get the length of the array i.e. the number of elements in the array.

foldl' :: forall a b. Unbox a => (b -> a -> b) -> b -> Array a -> b

foldr :: Unbox a => (a -> b -> b) -> b -> Array a -> b

splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)

Create two slices of an array without copying the original array. The specified index i is the first index of the second slice.

toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a

toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a

toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a

toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a

toStream :: (Monad m, Unbox a) => Array a -> Stream m a

Deprecated: Please use read instead.

Same as read

toStreamRev :: (Monad m, Unbox a) => Array a -> Stream m a

Deprecated: Please use readRev instead.

Same as readRev

read :: (Monad m, Unbox a) => Array a -> Stream m a

Convert an Array into a stream.

Pre-release

readRev :: (Monad m, Unbox a) => Array a -> Stream m a

Convert an Array into a stream in reverse order.

Pre-release

readerRev :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a

Unfold an array into a stream in reverse order.

toList :: Unbox a => Array a -> [a]

Convert an Array into a list.

Folds

writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

writeN n folds a maximum of n elements from the input stream to an Array.

pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

Like fromListN but creates a pinned array.

writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

Like writeN but does not check the array bounds when writing. The fold driver must not call the step function more than n times otherwise it will corrupt the memory and crash. This function exists mainly because any conditional in the step function blocks fusion causing 10x performance slowdown.

pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

data ArrayUnsafe a

Constructors

ArrayUnsafe !MutByteArray !Int !Int

pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)

pinnedWriteNAligned alignment n folds a maximum of n elements from the input stream to an Array aligned to the given size.

Pre-release

write :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)

Fold the whole input to a single array.

Caution! Do not use this on infinite streams.

pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)

Like write but creates a pinned array.

unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b

Fold "step" has a dependency on "initial", and each step is dependent on the previous invocation of step due to state passing, finally extract depends on the result of step, therefore, as long as the fold is driven in the correct order the operations would be correctly ordered. We need to ensure that we strictly evaluate the previous step completely before the next step.

To not share the same array we need to make sure that the result of "initial" is not shared. Existential type ensures that it does not get shared across different folds. However, if we invoke "initial" multiple times for the same fold, there is a possiblity of sharing the two because the compiler would consider it as a pure value. One such example is the chunksOf combinator, or using an array creation fold with foldMany combinator. Is there a proper way in GHC to tell it to not share a pure expression in a particular case?

For this reason array creation folds have a MonadIO constraint. Pure folds could be unsafe and dangerous. This is dangerous especially when used with foldMany like operations.

>>> unsafePureWrite = Array.unsafeMakePure Array.write

Streams of arrays

chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)

chunksOf n stream groups the elements in the input stream into arrays of n elements each.

Same as the following but may be more efficient:

>>> chunksOf n = Stream.foldMany (Array.writeN n)

Pre-release

pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)

Like chunksOf but creates pinned arrays.

bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))

flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a

Use the "read" unfold instead.

flattenArrays = unfoldMany read

We can try this if there are any fusion issues in the unfold.

flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a

Use the "readRev" unfold instead.

flattenArrays = unfoldMany readRev

We can try this if there are any fusion issues in the unfold.

Deprecated

unsafeIndex :: forall a. Unbox a => Int -> Array a -> a

Deprecated: Please use getIndexUnsafe instead

Type

We can use an Unbox constraint in the Array type and the constraint can be automatically provided to a function that pattern matches on the Array type. However, it has huge performance cost, so we do not use it. Investigate a GHC improvement possiblity.

data Array a

Constructors

Array

Fields

• arrContents :: !MutByteArray
• arrStart :: !Int
• arrEnd :: !Int

Instances

a ~ Char => IsString (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

fromString :: String -> Array a

Unbox a => Monoid (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

mempty :: Array a

mappend :: Array a -> Array a -> Array a

mconcat :: [Array a] -> Array a

Unbox a => Semigroup (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

(<>) :: Array a -> Array a -> Array a

sconcat :: NonEmpty (Array a) -> Array a

stimes :: Integral b => b -> Array a -> Array a

Unbox a => IsList (Array a)

Defined in Streamly.Internal.Data.Array.Type

Associated Types

type Item (Array a)

Methods

fromList :: [Item (Array a)] -> Array a

fromListN :: Int -> [Item (Array a)] -> Array a

toList :: Array a -> [Item (Array a)]

(Unbox a, Read a, Show a) => Read (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

readsPrec :: Int -> ReadS (Array a)

readList :: ReadS [Array a]

readPrec :: ReadPrec (Array a)

readListPrec :: ReadPrec [Array a]

(Show a, Unbox a) => Show (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

showsPrec :: Int -> Array a -> ShowS

show :: Array a -> String

showList :: [Array a] -> ShowS

(Unbox a, Eq a) => Eq (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

(==) :: Array a -> Array a -> Bool

(/=) :: Array a -> Array a -> Bool

(Unbox a, Ord a) => Ord (Array a)

Defined in Streamly.Internal.Data.Array.Type

Methods

compare :: Array a -> Array a -> Ordering

(<) :: Array a -> Array a -> Bool

(<=) :: Array a -> Array a -> Bool

(>) :: Array a -> Array a -> Bool

(>=) :: Array a -> Array a -> Bool

max :: Array a -> Array a -> Array a

min :: Array a -> Array a -> Array a

Serialize (Array a)

Defined in Streamly.Internal.Data.Serialize.Type

Methods

addSizeTo :: Int -> Array a -> Int

deserializeAt :: Int -> MutByteArray -> Int -> IO (Int, Array a)

serializeAt :: Int -> MutByteArray -> Array a -> IO Int

type Item (Array a)

Defined in Streamly.Internal.Data.Array.Type

type Item (Array a) = a

asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b

Use an Array a as Ptr a.

See asPtrUnsafe in the Mutable array module for more details.

Unsafe

Pre-release

nil :: Array a

Freezing and Thawing

unsafeFreeze :: MutArray a -> Array a

Makes an immutable array using the underlying memory of the mutable array.

Please make sure that there are no other references to the mutable array lying around, so that it is never used after freezing it using unsafeFreeze. If the underlying array is mutated, the immutable promise is lost.

Pre-release

unsafeFreezeWithShrink :: Unbox a => MutArray a -> Array a

Similar to unsafeFreeze but uses rightSize on the mutable array first.

unsafeThaw :: Array a -> MutArray a

Makes a mutable array using the underlying memory of the immutable array.

Please make sure that there are no other references to the immutable array lying around, so that it is never used after thawing it using unsafeThaw. If the resulting array is mutated, any references to the older immutable array are mutated as well.

Pre-release

Pinning and Unpinning

pin :: Array a -> IO (Array a)

Return a copy of the Array in pinned memory if unpinned, else return the original array.

unpin :: Array a -> IO (Array a)

Return a copy of the Array in unpinned memory if pinned, else return the original array.

isPinned :: Array a -> Bool

Return True if the array is allocated in pinned memory.

Construction

splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)

fromList :: Unbox a => [a] -> Array a

Create an Array from a list. The list must be of finite size.

pinnedFromList :: Unbox a => [a] -> Array a

Like fromList but creates a pinned array.

fromListN :: Unbox a => Int -> [a] -> Array a

Create an Array from the first N elements of a list. The array is allocated to size N, if the list terminates before N elements then the array may hold less than N elements.

pinnedFromListN :: Unbox a => Int -> [a] -> Array a

Like fromListN but creates a pinned array.

fromListRev :: Unbox a => [a] -> Array a

Create an Array from a list in reverse order. The list must be of finite size.

Pre-release

fromListRevN :: Unbox a => Int -> [a] -> Array a

Create an Array from the first N elements of a list in reverse order. The array is allocated to size N, if the list terminates before N elements then the array may hold less than N elements.

Pre-release

fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)

fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)

fromPureStream :: Unbox a => Stream Identity a -> Array a

Convert a pure stream in Identity monad to an immutable array.

Same as the following but with better performance:

>>> fromPureStream = Array.fromList . runIdentity . Stream.toList

fromByteStr# :: Addr# -> Array Word8

Copy a null terminated immutable Addr# Word8 sequence into an array.

Unsafe: The caller is responsible for safe addressing.

Note that this is completely safe when reading from Haskell string literals because they are guaranteed to be NULL terminated:

>>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#
[1,2,3]

Split

breakOn :: MonadIO m => Word8 -> Array Word8 -> m (Array Word8, Maybe (Array Word8))

Cloning arrays

clone :: MonadIO m => Array a -> m (Array a)

pinnedClone :: MonadIO m => Array a -> m (Array a)

Elimination

unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a

Return element at the specified index without checking the bounds.

Unsafe because it does not check the bounds of the array.

getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a

Return element at the specified index without checking the bounds.

byteLength :: Array a -> Int

O(1) Get the byte length of the array.

length :: Unbox a => Array a -> Int

O(1) Get the length of the array i.e. the number of elements in the array.

foldl' :: forall a b. Unbox a => (b -> a -> b) -> b -> Array a -> b

foldr :: Unbox a => (a -> b -> b) -> b -> Array a -> b

splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)

Create two slices of an array without copying the original array. The specified index i is the first index of the second slice.

toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a

toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a

toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a

toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a

toStream :: (Monad m, Unbox a) => Array a -> Stream m a

Deprecated: Please use read instead.

Same as read

toStreamRev :: (Monad m, Unbox a) => Array a -> Stream m a

Deprecated: Please use readRev instead.

Same as readRev

read :: (Monad m, Unbox a) => Array a -> Stream m a

Convert an Array into a stream.

Pre-release

readRev :: (Monad m, Unbox a) => Array a -> Stream m a

Convert an Array into a stream in reverse order.

Pre-release

readerRev :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a

Unfold an array into a stream in reverse order.

toList :: Unbox a => Array a -> [a]

Convert an Array into a list.

Folds

writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

writeN n folds a maximum of n elements from the input stream to an Array.

pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

Like fromListN but creates a pinned array.

writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

Like writeN but does not check the array bounds when writing. The fold driver must not call the step function more than n times otherwise it will corrupt the memory and crash. This function exists mainly because any conditional in the step function blocks fusion causing 10x performance slowdown.

pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)

data ArrayUnsafe a

Constructors

ArrayUnsafe !MutByteArray !Int !Int

pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)

pinnedWriteNAligned alignment n folds a maximum of n elements from the input stream to an Array aligned to the given size.

Pre-release

write :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)

Fold the whole input to a single array.

Caution! Do not use this on infinite streams.

pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)

Like write but creates a pinned array.

unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b

Fold "step" has a dependency on "initial", and each step is dependent on the previous invocation of step due to state passing, finally extract depends on the result of step, therefore, as long as the fold is driven in the correct order the operations would be correctly ordered. We need to ensure that we strictly evaluate the previous step completely before the next step.

To not share the same array we need to make sure that the result of "initial" is not shared. Existential type ensures that it does not get shared across different folds. However, if we invoke "initial" multiple times for the same fold, there is a possiblity of sharing the two because the compiler would consider it as a pure value. One such example is the chunksOf combinator, or using an array creation fold with foldMany combinator. Is there a proper way in GHC to tell it to not share a pure expression in a particular case?

For this reason array creation folds have a MonadIO constraint. Pure folds could be unsafe and dangerous. This is dangerous especially when used with foldMany like operations.

>>> unsafePureWrite = Array.unsafeMakePure Array.write

Streams of arrays

chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)

chunksOf n stream groups the elements in the input stream into arrays of n elements each.

Same as the following but may be more efficient:

>>> chunksOf n = Stream.foldMany (Array.writeN n)

Pre-release

pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)

Like chunksOf but creates pinned arrays.

bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))

flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a

Use the "read" unfold instead.

flattenArrays = unfoldMany read

We can try this if there are any fusion issues in the unfold.

flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a

Use the "readRev" unfold instead.

flattenArrays = unfoldMany readRev

We can try this if there are any fusion issues in the unfold.

Deprecated

unsafeIndex :: forall a. Unbox a => Int -> Array a -> a

Deprecated: Please use getIndexUnsafe instead

Construction

fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)

Create an Array from the first N elements of a stream. The array is allocated to size N, if the stream terminates before N elements then the array may hold less than N elements.

Pre-release

fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)

Create an Array from a stream. This is useful when we want to create a single array from a stream of unknown size. writeN is at least twice as efficient when the size is already known.

Note that if the input stream is too large memory allocation for the array may fail. When the stream size is not known, chunksOf followed by processing of indvidual arrays in the resulting stream should be preferred.

Pre-release

writeLastN :: (Storable a, Unbox a, MonadIO m) => Int -> Fold m a (Array a)

writeLastN n folds a maximum of n elements from the end of the input stream to an Array.

Unfolds

reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a

Unfold an array into a stream.

readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a

Unfold an array into a stream, does not check the end of the array, the user is responsible for terminating the stream within the array bounds. For high performance application where the end condition can be determined by a terminating fold.

Written in the hope that it may be faster than "read", however, in the case for which this was written, "read" proves to be faster even though the core generated with unsafeRead looks simpler.

Pre-release

producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a

Random Access

getIndex :: forall a. Unbox a => Int -> Array a -> Maybe a

O(1) Lookup the element at the given index. Index starts from 0.

getIndexRev :: forall a. Unbox a => Int -> Array a -> Maybe a

Like getIndex but indexes the array in reverse from the end.

Pre-release

last :: Unbox a => Array a -> Maybe a

>>> last arr = Array.getIndexRev arr 0

Pre-release

getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a

Given a stream of array indices, read the elements on those indices from the supplied Array. An exception is thrown if an index is out of bounds.

This is the most general operation. We can implement other operations in terms of this:

read =
     let u = lmap (arr -> (0, length arr - 1)) Unfold.enumerateFromTo
      in Unfold.lmap f (getIndices arr)

readRev =
     let i = length arr - 1
      in Unfold.lmap f (getIndicesFromThenTo i (i - 1) 0)

Pre-release

getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a

Unfolds (from, then, to, array) generating a finite stream whose first element is the array value from the index from and the successive elements are from the indices in increments of then up to to. Index enumeration can occur downwards or upwards depending on whether then comes before or after from.

getIndicesFromThenTo =
    let f (from, next, to, arr) =
            (Stream.enumerateFromThenTo from next to, arr)
     in Unfold.lmap f getIndices

Unimplemented

Size

null :: Array a -> Bool

>>> null arr = Array.byteLength arr == 0

Pre-release

Search

binarySearch :: a -> Array a -> Maybe Int

Given a sorted array, perform a binary search to find the given element. Returns the index of the element if found.

Unimplemented

findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int

Perform a linear search to find all the indices where a given element is present in an array.

Unimplemented

Casting

cast :: forall a b. Unbox b => Array a -> Maybe (Array b)

Cast an array having elements of type a into an array having elements of type b. The length of the array should be a multiple of the size of the target element otherwise Nothing is returned.

asBytes :: Array a -> Array Word8

Cast an Array a into an Array Word8.

castUnsafe :: Array a -> Array b

Cast an array having elements of type a into an array having elements of type b. The array size must be a multiple of the size of type b otherwise accessing the last element of the array may result into a crash or a random value.

Pre-release

asCStringUnsafe :: Array a -> (CString -> IO b) -> IO b

Convert an array of any type into a null terminated CString Ptr. If the array is unpinned it is first converted to a pinned array which requires a copy.

Unsafe

O(n) Time: (creates a copy of the array)

Pre-release

Subarrays

getSliceUnsafe

Arguments

:: forall a. Unbox a

=> Int

starting index

-> Int

length of the slice

-> Array a

-> Array a

O(1) Slice an array in constant time.

Caution: The bounds of the slice are not checked.

Unsafe

Pre-release

genSlicesFromLen

Arguments

:: forall m a. (Monad m, Unbox a)

=> Int

from index

-> Int

length of the slice

-> Unfold m (Array a) (Int, Int)

getSlicesFromLen

Arguments

:: forall m a. (Monad m, Unbox a)

=> Int

from index

-> Int

length of the slice

-> Unfold m (Array a) (Array a)

Generate a stream of slices of specified length from an array, starting from the supplied array index. The last slice may be shorter than the requested length.

Pre-release/

splitOn :: (Monad m, Unbox a) => (a -> Bool) -> Array a -> Stream m (Array a)

Split the array into a stream of slices using a predicate. The element matching the predicate is dropped.

Pre-release

Streaming Operations

streamTransform :: forall m a b. (MonadIO m, Unbox a, Unbox b) => (Stream m a -> Stream m b) -> Array a -> m (Array b)

Transform an array into another array using a stream transformation operation.

Pre-release

Folding

streamFold :: (Monad m, Unbox a) => (Stream m a -> m b) -> Array a -> m b

Fold an array using a stream fold operation.

Pre-release

fold :: forall m a b. (Monad m, Unbox a) => Fold m a b -> Array a -> m b

Fold an array using a Fold.

Pre-release

Serialization

encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8

serialize :: Serialize a => a -> Array Word8

Properties: 1. Identity: deserialize . serialize == id 2. Encoded equivalence: serialize a == serialize a

pinnedSerialize :: Serialize a => a -> Array Word8

Serialize a Haskell type to a pinned byte array. The array is allocated using pinned memory so that it can be used directly in OS APIs for writing to file or sending over the network.

Properties: 1. Identity: deserialize . pinnedSerialize == id 2. Encoded equivalence: pinnedSerialize a == pinnedSerialize a

deserialize :: Serialize a => Array Word8 -> a

Decode a Haskell type from a byte array containing its serialized representation.