NLP.Tokenize.Text

NLP Tokenizer, adapted to use Text instead of Strings from the tokenize package.

Documentation

newtype EitherList a b

The EitherList is a newtype-wrapped list of Eithers.

Constructors

E

Fields

• unE :: [Either a b]

Instances

Applicative (EitherList a)

Defined in NLP.Tokenize.Text

Methods

pure :: a0 -> EitherList a a0

(<*>) :: EitherList a (a0 -> b) -> EitherList a a0 -> EitherList a b

liftA2 :: (a0 -> b -> c) -> EitherList a a0 -> EitherList a b -> EitherList a c

(*>) :: EitherList a a0 -> EitherList a b -> EitherList a b

(<*) :: EitherList a a0 -> EitherList a b -> EitherList a a0

Functor (EitherList a)

Defined in NLP.Tokenize.Text

Methods

fmap :: (a0 -> b) -> EitherList a a0 -> EitherList a b

(<$) :: a0 -> EitherList a b -> EitherList a a0

Monad (EitherList a)

Defined in NLP.Tokenize.Text

Methods

(>>=) :: EitherList a a0 -> (a0 -> EitherList a b) -> EitherList a b

(>>) :: EitherList a a0 -> EitherList a b -> EitherList a b

return :: a0 -> EitherList a a0

type Tokenizer = Text -> EitherList Text Text

A Tokenizer is function which takes a list and returns a list of Eithers (wrapped in a newtype). Right Texts will be passed on for processing to tokenizers down the pipeline. Left Texts will be passed through the pipeline unchanged. Use a Left Texts in a tokenizer to protect certain tokens from further processing (e.g. see the uris tokenizer). You can define your own custom tokenizer pipelines by chaining tokenizers together:

tokenize :: Text -> [Text]

Split string into words using the default tokenizer pipeline

run :: Tokenizer -> Text -> [Text]

Run a tokenizer

defaultTokenizer :: Tokenizer

whitespace :: Tokenizer

Split string on whitespace. This is just a wrapper for Data.List.words

uris :: Tokenizer

Detect common uris and freeze them

punctuation :: Tokenizer

Split off initial and final punctuation

finalPunctuation :: Tokenizer

Split off word-final punctuation

initialPunctuation :: Tokenizer

Split off word-initial punctuation

allPunctuation :: Tokenizer

Split tokens on transitions between punctuation and non-punctuation characters. This tokenizer is not included in defaultTokenizer pipeline because dealing with word-internal punctuation is quite application specific.

contractions :: Tokenizer

Split common contractions off and freeze them. | Currently deals with: 'm, 's, 'd, 've, 'll

negatives :: Tokenizer

Split words ending in n't, and freeze n't