Data.Text.Fuzzy.Tokenize

The lightweight and multi-functional text tokenizer allowing different types of text tokenization depending on it's settings.

It may be used in different sutiations, for DSL, text markups or even for parsing simple grammars easier and sometimes faster than in case of usage mainstream parsing combinators or parser generators.

The primary goal of this package is to parse unstructured text data, however it may be used for parsing such data formats as CSV with ease.

Currently it supports the following types of entities: atoms, string literals (currently with the minimal set of escaped characters), punctuation characters and delimeters.

Examples

Simple CSV-like tokenization

>>> tokenize (delims ":") "aaa : bebeb : qqq ::::" :: [Text]
["aaa "," bebeb "," qqq "]

>>> tokenize (delims ":"<>sq<>emptyFields ) "aaa : bebeb : qqq ::::" :: [Text]
["aaa "," bebeb "," qqq ","","","",""]

>>> > tokenize (delims ":"<>sq<>emptyFields ) "aaa : bebeb : qqq ::::" :: [Maybe Text]
[Just "aaa ",Just " bebeb ",Just " qqq ",Nothing,Nothing,Nothing,Nothing]

>>> tokenize (delims ":"<>sq<>emptyFields ) "aaa : 'bebeb:colon inside' : qqq ::::" :: [Maybe Text]
[Just "aaa ",Just " ",Just "bebeb:colon inside",Just " ",Just " qqq ",Nothing,Nothing,Nothing,Nothing]

>>> let spec = sl<>delims ":"<>sq<>emptyFields<>noslits
>>> tokenize spec "   aaa :   'bebeb:colon inside' : qqq ::::" :: [Maybe Text]
[Just "aaa ",Just "bebeb:colon inside ",Just "qqq ",Nothing,Nothing,Nothing,Nothing]

>>> let spec = delims ":"<>sq<>emptyFields<>uw<>noslits
>>> tokenize spec "  a  b  c  : 'bebeb:colon inside' : qqq ::::"  :: [Maybe Text]
[Just "a b c",Just "bebeb:colon inside",Just "qqq",Nothing,Nothing,Nothing,Nothing]

Notes

About the delimeter tokens

This type of tokens appears during a "delimited" formats processing and disappears in results. Currenly you will never see it unless normalization is turned off by nn option.

The delimeters make sense in case of processing the CSV-like formats, but in this case you probably need only values in results.

This behavior may be changed later. But right now delimeters seem pointless in results. If you process some sort of grammar where delimeter character is important, you may use punctuation instead, i.e:

>>> let spec = delims " \t"<>punct ",;()" <>emptyFields<>sq
>>> tokenize spec "( delimeters , are , important, 'spaces are not');" :: [Text]
["(","delimeters",",","are",",","important",",","spaces are not",")",";"]

Other

For CSV-like formats it makes sense to split text to lines first, otherwise newline characters may cause to weird results

Documentation

data TokenizeSpec

Tokenization settings. Use mempty for an empty value and construction functions for changing the settings.

Instances

Eq TokenizeSpec

Defined in Data.Text.Fuzzy.Tokenize

Methods

(==) :: TokenizeSpec -> TokenizeSpec -> Bool

(/=) :: TokenizeSpec -> TokenizeSpec -> Bool

Ord TokenizeSpec

Defined in Data.Text.Fuzzy.Tokenize

Methods

compare :: TokenizeSpec -> TokenizeSpec -> Ordering

(<) :: TokenizeSpec -> TokenizeSpec -> Bool

(<=) :: TokenizeSpec -> TokenizeSpec -> Bool

(>) :: TokenizeSpec -> TokenizeSpec -> Bool

(>=) :: TokenizeSpec -> TokenizeSpec -> Bool

max :: TokenizeSpec -> TokenizeSpec -> TokenizeSpec

min :: TokenizeSpec -> TokenizeSpec -> TokenizeSpec

Show TokenizeSpec

Defined in Data.Text.Fuzzy.Tokenize

Methods

showsPrec :: Int -> TokenizeSpec -> ShowS

show :: TokenizeSpec -> String

showList :: [TokenizeSpec] -> ShowS

Semigroup TokenizeSpec

Defined in Data.Text.Fuzzy.Tokenize

Methods

(<>) :: TokenizeSpec -> TokenizeSpec -> TokenizeSpec

sconcat :: NonEmpty TokenizeSpec -> TokenizeSpec

stimes :: Integral b => b -> TokenizeSpec -> TokenizeSpec

Monoid TokenizeSpec

Defined in Data.Text.Fuzzy.Tokenize

Methods

mempty :: TokenizeSpec

mappend :: TokenizeSpec -> TokenizeSpec -> TokenizeSpec

mconcat :: [TokenizeSpec] -> TokenizeSpec

class IsToken a where

Typeclass for token values. Note, that some tokens appear in results only when nn option is set, i.e. sequences of characters turn out to text tokens or string literals and delimeter tokens are just removed from the results

Minimal complete definition

mkChar, mkSChar, mkPunct, mkText, mkStrLit, mkKeyword, mkEmpty

Methods

mkChar :: Char -> a

Create a character token

mkSChar :: Char -> a

Create a string literal character token

mkPunct :: Char -> a

Create a punctuation token

mkText :: Text -> a

Create a text chunk token

mkStrLit :: Text -> a

Create a string literal token

mkKeyword :: Text -> a

Create a keyword token

mkEmpty :: a

Create an empty field token

mkDelim :: a

Create a delimeter token

mkIndent :: Int -> a

Creates an indent token

mkEol :: a

Creates an EOL token

Instances

IsToken Text

Defined in Data.Text.Fuzzy.Tokenize

Methods

mkChar :: Char -> Text

mkSChar :: Char -> Text

mkPunct :: Char -> Text

mkText :: Text -> Text

mkStrLit :: Text -> Text

mkKeyword :: Text -> Text

mkEmpty :: Text

mkDelim :: Text

mkIndent :: Int -> Text

mkEol :: Text

IsToken (Maybe Text)

Defined in Data.Text.Fuzzy.Tokenize

Methods

mkChar :: Char -> Maybe Text

mkSChar :: Char -> Maybe Text

mkPunct :: Char -> Maybe Text

mkText :: Text -> Maybe Text

mkStrLit :: Text -> Maybe Text

mkKeyword :: Text -> Maybe Text

mkEmpty :: Maybe Text

mkDelim :: Maybe Text

mkIndent :: Int -> Maybe Text

mkEol :: Maybe Text

tokenize :: IsToken a => TokenizeSpec -> Text -> [a]

Tokenize a text

esc :: TokenizeSpec

Turn on character escaping inside string literals. Currently the following escaped characters are supported: [" ' t n r a b f v ]

addEmptyFields :: TokenizeSpec

Raise empty field tokens (note mkEmpty method) when no tokens found before a delimeter. Useful for processing CSV-like data in order to distingush empty columns

emptyFields :: TokenizeSpec

same as addEmptyFields

nn :: TokenizeSpec

Turns off token normalization. Makes the tokenizer generate character stream. Useful for debugging.

sq :: TokenizeSpec

Turns on single-quoted string literals. Character stream after '\'' character will be proceesed as single-quoted stream, assuming all delimeter, comment and other special characters as a part of the string literal until the next unescaped single quote character.

sqq :: TokenizeSpec

Enable double-quoted string literals support as sq for single-quoted strings.

noslits :: TokenizeSpec

Disable separate string literals.

Useful when processed delimeted data (csv-like formats). Normally, sequential text chunks are concatenated together, but consequent text and string literal will produce the two different tokens and it may cause weird results if data is in csv-like format, i.e:

>>> tokenize (delims ":"<>emptyFields<>sq ) "aaa:bebe:'qq' aaa:next::" :: [Maybe Text]
[Just "aaa",Just "bebe",Just "qq",Just " aaa",Just "next",Nothing,Nothing]

look: "qq" and " aaa" are turned into two separate tokens that makes the result of CSV processing looks improper, like it has an extra-column. This behavior may be avoided using this option, if you don't need to distinguish text chunks and string literals:

>>> tokenize (delims ":"<>emptyFields<>sq<>noslits) "aaa:bebe:'qq:foo' aaa:next::" :: [Maybe Text]
[Just "aaa",Just "bebe",Just "qq:foo aaa",Just "next",Nothing,Nothing]

sl :: TokenizeSpec

Strip spaces on left side of a token. Does not affect string literals, i.e string are processed normally. Useful mostly for processing CSV-like formats, otherwise delims may be used to skip unwanted spaces.

sr :: TokenizeSpec

Strip spaces on right side of a token. Does not affect string literals, i.e string are processed normally. Useful mostly for processing CSV-like formats, otherwise delims may be used to skip unwanted spaces.

uw :: TokenizeSpec

Strips spaces on right and left sides and transforms multiple spaces into the one. Name origins from unwords . words

Does not affect string literals, i.e string are processed normally. Useful mostly for processing CSV-like formats, otherwise delims may be used to skip unwanted spaces.

delims :: String -> TokenizeSpec

Specify the list of delimers (characters) to split the character stream into fields. Useful for CSV-like separated formats. Support for empty fields in token stream may be enabled by addEmptyFields function

comment :: Text -> TokenizeSpec

Specify the line comment prefix. All text after the line comment prefix will be ignored until the newline character appearance. Multiple line comments are supported.

punct :: Text -> TokenizeSpec

Specify the punctuation characters. Any punctuation character is handled as a separate token. Any token will be breaked on a punctiation character.

Useful for handling ... er... punctuaton, like

> function(a,b)

or

> (apply function 1 2 3)

>>> tokenize spec "(apply function 1 2 3)" :: [Text]
["(","apply","function","1","2","3",")"]

indent :: TokenizeSpec

Enable identation support

itabstops :: Int -> TokenizeSpec

Set tab expanding multiplier i.e. each tab extends into n spaces before processing. It also turns on the indentation. Only the tabs at the beginning of the string are expanded, i.e. before the first non-space character appears.

keywords :: [Text] -> TokenizeSpec

Specify the keywords list. Each keyword will be threated as a separate token.

eol :: TokenizeSpec

Turns on EOL token generation