Control.Monad.Tokenizer.Text.Strict

A monad for writing pure tokenizers in an imperative-looking way.

Main idea: You walk through the input string like a turtle, and everytime you find a token boundary, you call emit. If some specific kinds of tokens should be suppressed, you can discard them instead (or filter afterwards).

This module supports is specialized for strict text. The module Control.Monad.Tokenizer provides more general types.

Example for a simple tokenizer, that splits words by whitespace and discards stop symbols:

tokenizeWords :: T.Text -> [T.Text]
tokenizeWords = runTokenizer $ untilEOT $ do
  c <- pop
  if isStopSym c
    then discard
    else if c `elem` ("  \t\r\n" :: [Char])
         then discard
         else do
           walkWhile (\c -> (c=='_') || not (isSpace c || isPunctuation' c))
           emit

Monad

type Tokenizer = Tokenizer Text

Tokenizer monad. Use runTokenizer or runTokenizerCS to run it

runTokenizer :: Tokenizer () -> Text -> [Text]

Split a string into tokens using the given tokenizer

runTokenizerCS :: Tokenizer () -> Text -> [Text]

Split a string into tokens using the given tokenizer, case sensitive version

untilEOT :: Tokenizer () -> Tokenizer ()

Repeat a given tokenizer as long as the end of text is not reached

Tests

peek :: Tokenizer Char

Peek the current character

isEOT :: Tokenizer Bool

Have I reached the end of the input text?

lookAhead :: [Char] -> Tokenizer Bool

Check if the next input chars agree with the given string

Movement

walk :: Tokenizer ()

Proceed to the next character

walkBack :: Tokenizer ()

Walk back to the previous character, unless it was discarded/emitted.

pop :: Tokenizer Char

Peek the current character and proceed

walkWhile :: (Char -> Bool) -> Tokenizer ()

Proceed as long as a given function succeeds

walkFold :: a -> (Char -> a -> Maybe a) -> Tokenizer ()

Proceed as long as a given fold returns Just (generalization of walkWhile)

Transactions

emit :: Tokenizer ()

Break at the current position and emit the scanned token

discard :: Tokenizer ()

Break at the current position and discard the scanned token

restore :: Tokenizer ()

Restore the state after the last emit/discard.