Pipes.ByteString

This module provides pipes utilities for "byte streams", which are streams of strict ByteStrings chunks. Use byte streams to interact with both Handles and lazy ByteStrings.

To stream to or from Handles, use fromHandle or toHandle. For example, the following program copies data from one file to another:

import Pipes
import qualified Pipes.ByteString as P
import System.IO

main =
    withFile "inFile.txt"  ReadMode  $ \hIn  ->
    withFile "outFile.txt" WriteMode $ \hOut ->
    runEffect $ P.fromHandle hIn >-> P.toHandle hOut

You can stream to and from stdin and stdout using the predefined stdin and stdout pipes, like in the following "echo" program:

main = runEffect $ P.stdin >-> P.stdout

You can also translate pure lazy ByteStrings to and from pipes:

import qualified Data.ByteString.Lazy.Char8 as BL

main = runEffect $ P.fromLazy (BL.pack "Hello, world!\n") >-> P.stdout

In addition, this module provides many functions equivalent to lazy ByteString functions so that you can transform or fold byte streams. For example, to stream only the first three lines of stdin to stdout you would write:

import Lens.Family (over)
import Pipes
import qualified Pipes.ByteString as PB
import Pipes.Group (takes)

main = runEffect $ over PB.lines (takes 3) PB.stdin >-> PB.stdout

The above program will never bring more than one chunk (~ 32 KB) into memory, no matter how long the lines are.

Note that functions in this library are designed to operate on streams that are insensitive to chunk boundaries. This means that they may freely split chunks into smaller chunks and discard empty chunks. However, they will never concatenate chunks in order to provide strict upper bounds on memory usage.

Producers

fromLazy :: Monad m => ByteString -> Producer' ByteString m ()

Convert a lazy ByteString into a Producer of strict ByteStrings

stdin :: MonadIO m => Producer' ByteString m ()

Stream bytes from stdin

fromHandle :: MonadIO m => Handle -> Producer' ByteString m ()

Convert a Handle into a byte stream using a default chunk size

hGetSome :: MonadIO m => Int -> Handle -> Producer' ByteString m ()

Convert a handle into a byte stream using a maximum chunk size

hGetSome forwards input immediately as it becomes available, splitting the input into multiple chunks if it exceeds the maximum chunk size.

hGetNonBlocking :: MonadIO m => Int -> Handle -> Producer' ByteString m ()

Convert a handle into a byte stream using a fixed chunk size

Similar to hGet except that it will never block waiting for data to become available.

hGet :: MonadIO m => Int -> Handle -> Producer' ByteString m ()

Convert a handle into a byte stream using a fixed chunk size

hGet waits until exactly the requested number of bytes are available for each chunk.

hGetRange

Arguments

:: MonadIO m

=> Int

Offset

-> Int

Size

-> Handle

-> Producer' ByteString m ()

Like hGet but with an extra parameter specifying an initial handle offset

Servers

hGetSomeN :: MonadIO m => Handle -> Int -> Server' Int ByteString m ()

Like hGetSome, except you can vary the maximum chunk size for each request

hGetN :: MonadIO m => Handle -> Int -> Server' Int ByteString m ()

Like hGet, except you can vary the chunk size for each request

Consumers

stdout :: MonadIO m => Consumer' ByteString m ()

Stream bytes to stdout

Unlike toHandle, stdout gracefully terminates on a broken output pipe.

toHandle :: MonadIO m => Handle -> Consumer' ByteString m r

Convert a byte stream into a Handle

p >-> toHandle handle = for p (liftIO . hPutStr handle)

Pipes

map :: Monad m => (Word8 -> Word8) -> Pipe ByteString ByteString m r

Apply a transformation to each Word8 in the stream

concatMap :: Monad m => (Word8 -> ByteString) -> Pipe ByteString ByteString m r

Map a function over the byte stream and concatenate the results

take :: (Monad m, Integral n) => n -> Pipe ByteString ByteString m ()

(take n) only allows n bytes to pass

takeWhile :: Monad m => (Word8 -> Bool) -> Pipe ByteString ByteString m ()

Take bytes until they fail the predicate

filter :: Monad m => (Word8 -> Bool) -> Pipe ByteString ByteString m r

Only allows Word8s to pass if they satisfy the predicate

elemIndices :: (Monad m, Num n) => Word8 -> Pipe ByteString n m r

Stream all indices whose elements match the given Word8

findIndices :: (Monad m, Num n) => (Word8 -> Bool) -> Pipe ByteString n m r

Stream all indices whose elements satisfy the given predicate

scan :: Monad m => (Word8 -> Word8 -> Word8) -> Word8 -> Pipe ByteString ByteString m r

Strict left scan over the bytes

Folds

toLazy :: Producer ByteString Identity () -> ByteString

Fold a pure Producer of strict ByteStrings into a lazy ByteString

toLazyM :: Monad m => Producer ByteString m () -> m ByteString

Fold an effectful Producer of strict ByteStrings into a lazy ByteString

Note: toLazyM is not an idiomatic use of pipes, but I provide it for simple testing purposes. Idiomatic pipes style consumes the chunks immediately as they are generated instead of loading them all into memory.

toLazyM' :: Monad m => Producer ByteString m a -> m (ByteString, a)

Fold an effectful Producer of strict ByteStrings into a lazy ByteString alongside the return value

Note: toLazyM' is not an idiomatic use of pipes, but I provide it for simple testing purposes. Idiomatic pipes style consumes the chunks immediately as they are generated instead of loading them all into memory.

foldBytes :: Monad m => (x -> Word8 -> x) -> x -> (x -> r) -> Producer ByteString m () -> m r

Reduce the stream of bytes using a strict left fold

Note: It's more efficient to use folds from Control.Foldl.ByteString in conjunction with Pipes.Prelude.fold when possible

head :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Retrieve the first Word8

last :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Retrieve the last Word8

null :: Monad m => Producer ByteString m () -> m Bool

Determine if the stream is empty

length :: (Monad m, Num n) => Producer ByteString m () -> m n

Count the number of bytes

any :: Monad m => (Word8 -> Bool) -> Producer ByteString m () -> m Bool

Fold that returns whether Any received Word8s satisfy the predicate

all :: Monad m => (Word8 -> Bool) -> Producer ByteString m () -> m Bool

Fold that returns whether All received Word8s satisfy the predicate

maximum :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Return the maximum Word8 within a byte stream

minimum :: Monad m => Producer ByteString m () -> m (Maybe Word8)

Return the minimum Word8 within a byte stream

elem :: Monad m => Word8 -> Producer ByteString m () -> m Bool

Determine whether any element in the byte stream matches the given Word8

notElem :: Monad m => Word8 -> Producer ByteString m () -> m Bool

Determine whether all elements in the byte stream do not match the given Word8

find :: Monad m => (Word8 -> Bool) -> Producer ByteString m () -> m (Maybe Word8)

Find the first element in the stream that matches the predicate

index :: (Monad m, Integral n) => n -> Producer ByteString m () -> m (Maybe Word8)

Index into a byte stream

elemIndex :: (Monad m, Num n) => Word8 -> Producer ByteString m () -> m (Maybe n)

Find the index of an element that matches the given Word8

findIndex :: (Monad m, Num n) => (Word8 -> Bool) -> Producer ByteString m () -> m (Maybe n)

Store the first index of an element that satisfies the predicate

count :: (Monad m, Num n) => Word8 -> Producer ByteString m () -> m n

Store a tally of how many elements match the given Word8

Parsing

The following parsing utilities are single-byte analogs of the ones found in pipes-parse.

nextByte :: Monad m => Producer ByteString m r -> m (Either r (Word8, Producer ByteString m r))

Consume the first byte from a byte stream

next either fails with a Left if the Producer has no more bytes or succeeds with a Right providing the next byte and the remainder of the Producer.

drawByte :: Monad m => Parser ByteString m (Maybe Word8)

Draw one Word8 from the underlying Producer, returning Nothing if the Producer is empty

unDrawByte :: Monad m => Word8 -> Parser ByteString m ()

Push back a Word8 onto the underlying Producer

peekByte :: Monad m => Parser ByteString m (Maybe Word8)

peekByte checks the first Word8 in the stream, but uses unDrawByte to push the Word8 back

peekByte = do
    x <- drawByte
    case x of
        Nothing -> return ()
        Just w8 -> unDrawByte w8
    return x

isEndOfBytes :: Monad m => Parser ByteString m Bool

Check if the underlying Producer has no more bytes

Note that this will skip over empty ByteString chunks, unlike isEndOfInput from pipes-parse.

isEndOfBytes = liftM isNothing peekByte

Parsing Lenses

splitAt :: (Monad m, Integral n) => n -> Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Improper lens that splits a Producer after the given number of bytes

span :: Monad m => (Word8 -> Bool) -> Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Improper lens that splits after the longest consecutive group of bytes that satisfy the given predicate

break :: Monad m => (Word8 -> Bool) -> Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Improper lens that splits after the longest consecutive group of bytes that fail the given predicate

breakOn :: Monad m => ByteString -> Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Improper lens that splits at the first occurrence of the pattern.

groupBy :: Monad m => (Word8 -> Word8 -> Bool) -> Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Improper lens that splits after the first group of matching bytes, as defined by the given equality predicate

group :: Monad m => Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Like groupBy, where the equality predicate is (==)

word :: Monad m => Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Improper lens that splits a Producer after the first word

Unlike words, this does not drop leading whitespace

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the pipes-text library.

line :: Monad m => Lens' (Producer ByteString m x) (Producer ByteString m (Producer ByteString m x))

Improper lens that splits a Producer after the first line

Unlike lines, this does not consume the newline marker, which is stored within the inner Producer

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the pipes-text library.

Transforming Byte Streams

drop :: (Monad m, Integral n) => n -> Producer ByteString m r -> Producer ByteString m r

(drop n) drops the first n bytes

dropWhile :: Monad m => (Word8 -> Bool) -> Producer ByteString m r -> Producer ByteString m r

Drop bytes until they fail the predicate

intersperse :: Monad m => Word8 -> Producer ByteString m r -> Producer ByteString m r

Intersperse a Word8 in between the bytes of the byte stream

pack :: Monad m => Lens' (Producer Word8 m x) (Producer ByteString m x)

Improper lens from unpacked Word8s to packaged ByteStrings

unpack :: Monad m => Lens' (Producer ByteString m x) (Producer Word8 m x)

Improper lens from packed ByteStrings to unpacked Word8s

chunksOf' :: (Monad m, Integral n) => n -> Producer ByteString m r -> Producer ByteString m r

Group byte stream chunks into chunks of fixed length

Note: This is the only function in this API that concatenates ByteString chunks, which requires allocating new ByteStrings

FreeT Transformations

chunksOf :: (Monad m, Integral n) => n -> Lens' (Producer ByteString m x) (FreeT (Producer ByteString m) m x)

Split a byte stream into FreeT-delimited byte streams of fixed size

splitsWith :: Monad m => (Word8 -> Bool) -> Producer ByteString m x -> FreeT (Producer ByteString m) m x

Split a byte stream into groups separated by bytes that satisfy the predicate

splits :: Monad m => Word8 -> Lens' (Producer ByteString m x) (FreeT (Producer ByteString m) m x)

Split a byte stream into groups separated by the given byte

splitOn :: Monad m => ByteString -> Lens' (Producer ByteString m x) (FreeT (Producer ByteString m) m x)

Split a byte stream into groups separated by the given ByteString

groupsBy :: Monad m => (Word8 -> Word8 -> Bool) -> Lens' (Producer ByteString m x) (FreeT (Producer ByteString m) m x)

Isomorphism between a byte stream and groups of identical bytes using the supplied equality predicate

groups :: Monad m => Lens' (Producer ByteString m x) (FreeT (Producer ByteString m) m x)

Like groupsBy, where the equality predicate is (==)

lines :: Monad m => Lens' (Producer ByteString m x) (FreeT (Producer ByteString m) m x)

Improper lens between a bytestream and its lines

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the pipes-text library.

unlines :: Monad m => Lens' (FreeT (Producer ByteString m) m x) (Producer ByteString m x)

Improper lens between lines and a bytestream

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the pipes-text library.

words :: Monad m => Producer ByteString m x -> FreeT (Producer ByteString m) m x

Convert a bytestream to delimited words

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the pipes-text library.

unwords :: Monad m => FreeT (Producer ByteString m) m x -> Producer ByteString m x

Convert delimited words back to a byte stream

Note: This function is purely for demonstration purposes since it assumes a particular encoding. You should prefer the Text equivalent of this function from the pipes-text library.

Re-exports

Data.ByteString re-exports the ByteString type.

Data.Word re-exports the Word8 type.

Pipes.Parse re-exports Parser.

Pipes.Group re-exports concats, intercalates, and FreeT (the type).

module Data.ByteString

module Data.Word

module Pipes.Group

module Pipes.Parse