Data.Text.ICU.Spoof

String spoofing (confusability) checks for Unicode, implemented as bindings to the International Components for Unicode (ICU) uspoof library.

See UTR #36 and UTS #39 for detailed information about the underlying algorithms and databases used by this module.

Unicode spoof checking API

The spoofCheck, areConfusable, and getSkeleton functions analyze Unicode text for visually confusable (or "spoof") characters.

For example, Latin, Cyrillic, and Greek all contain unique Unicode values which appear nearly identical on-screen:

     A    0041    LATIN CAPITAL LETTER A
     Α    0391    GREEK CAPITAL LETTER ALPHA
     А    0410    CYRILLIC CAPITAL LETTER A
     Ꭺ    13AA    CHEROKEE LETTER GO
     ᴀ    1D00    LATIN LETTER SMALL CAPITAL A
     ᗅ    15C5    CANADIAN SYLLABICS CARRIER GHO
     Ａ    FF21    FULLWIDTH LATIN CAPITAL LETTER A
     𐊠    102A0   CARIAN LETTER A
     𝐀    1D400   MATHEMATICAL BOLD CAPITAL A

and so on. To check a string for visually confusable characters:

1. open an MSpoof
2. optionally configure it with setChecks, setRestrictionLevel, and/or setAllowedLocales, then
3. spoofCheck a single string, use areConfusable to check if two strings could be confused for each other, or use getSkeleton to precompute a "skeleton" string (similar to a hash code) which can be cached and re-used to quickly check (using Unicode string comparison) if two strings are confusable.

By default, these methods will use ICU's bundled copy of confusables.txt and confusablesWholeScript.txt, which could be out of date. To provide your own confusables databases, use openFromSource. (To avoid repeatedly parsing these databases, you can then serialize your configured MSpoof and later openFromSerialized to load the pre-parsed databases.)

data MSpoof

Configurable spoof checker wrapping an opaque handle and optionally wrapping a previously serialized instance.

data OpenFromSourceParseError

Exception thrown with openFromSource fails to parse one of the input files.

Constructors

OpenFromSourceParseError

Fields

• errFile :: OpenFromSourceParseErrorFile

  The file which could not be parsed.

• parseError :: ParseError

  Parse error encountered opening a spoof checker from source.

Instances

Show OpenFromSourceParseError

Defined in Data.Text.ICU.Spoof

Methods

showsPrec :: Int -> OpenFromSourceParseError -> ShowS

show :: OpenFromSourceParseError -> String

showList :: [OpenFromSourceParseError] -> ShowS

Exception OpenFromSourceParseError

Defined in Data.Text.ICU.Spoof

Methods

toException :: OpenFromSourceParseError -> SomeException

fromException :: SomeException -> Maybe OpenFromSourceParseError

displayException :: OpenFromSourceParseError -> String

NFData OpenFromSourceParseError

Defined in Data.Text.ICU.Spoof

Methods

rnf :: OpenFromSourceParseError -> ()

data SpoofCheck

Constructors

SingleScriptConfusable

Makes areConfusable report if both identifiers are both from the same script and are visually confusable. Does not affect spoofCheck.

MixedScriptConfusable

Makes areConfusable report if both identifiers are visually confusable and at least one identifier contains characters from more than one script.

Makes spoofCheck report if the identifier contains multiple scripts, and is confusable with some other identifier in a single script.

WholeScriptConfusable

Makes areConfusable report if each identifier is of a different single script, and the identifiers are visually confusable.

AnyCase

By default, spoof checks assume the strings have been processed through toCaseFold and only check lower-case identifiers. If this is set, spoof checks will check both upper and lower case identifiers.

RestrictionLevel

Checks that identifiers are no looser than the specified level passed to setRestrictionLevel.

Invisible

Checks the identifier for the presence of invisible characters, such as zero-width spaces, or character sequences that are likely not to display, such as multiple occurrences of the same non-spacing mark.

CharLimit

Checks whether the identifier contains only characters from a specified set (for example, via setAllowedLocales).

MixedNumbers

Checks that the identifier contains numbers from only a single script.

AllChecks

Enables all checks.

AuxInfo

Enables returning a RestrictionLevel in the SpoofCheckResult.

Instances

Bounded SpoofCheck

Defined in Data.Text.ICU.Spoof

Methods

minBound :: SpoofCheck

maxBound :: SpoofCheck

Enum SpoofCheck

Defined in Data.Text.ICU.Spoof

Methods

succ :: SpoofCheck -> SpoofCheck

pred :: SpoofCheck -> SpoofCheck

toEnum :: Int -> SpoofCheck

fromEnum :: SpoofCheck -> Int

enumFrom :: SpoofCheck -> [SpoofCheck]

enumFromThen :: SpoofCheck -> SpoofCheck -> [SpoofCheck]

enumFromTo :: SpoofCheck -> SpoofCheck -> [SpoofCheck]

enumFromThenTo :: SpoofCheck -> SpoofCheck -> SpoofCheck -> [SpoofCheck]

Eq SpoofCheck

Defined in Data.Text.ICU.Spoof

Methods

(==) :: SpoofCheck -> SpoofCheck -> Bool

(/=) :: SpoofCheck -> SpoofCheck -> Bool

Show SpoofCheck

Defined in Data.Text.ICU.Spoof

Methods

showsPrec :: Int -> SpoofCheck -> ShowS

show :: SpoofCheck -> String

showList :: [SpoofCheck] -> ShowS

data SpoofCheckResult

Constructors

CheckOK

The string passed all configured spoof checks.

CheckFailed [SpoofCheck]

The string failed one or more spoof checks.

CheckFailedWithRestrictionLevel

The string failed one or more spoof checks, and failed to pass the configured restriction level.

Fields

• failedChecks :: [SpoofCheck]

  The spoof checks which the string failed.

• failedLevel :: RestrictionLevel

  The restriction level which the string failed to pass.

Instances

Eq SpoofCheckResult

Defined in Data.Text.ICU.Spoof

Methods

(==) :: SpoofCheckResult -> SpoofCheckResult -> Bool

(/=) :: SpoofCheckResult -> SpoofCheckResult -> Bool

Show SpoofCheckResult

Defined in Data.Text.ICU.Spoof

Methods

showsPrec :: Int -> SpoofCheckResult -> ShowS

show :: SpoofCheckResult -> String

showList :: [SpoofCheckResult] -> ShowS

data RestrictionLevel

Constructors

ASCII

Checks that the string contains only Unicode values in the range ߝ inclusive.

SingleScriptRestrictive

Checks that the string contains only characters from a single script.

HighlyRestrictive

Checks that the string contains only characters from a single script, or from the combinations (Latin + Han + Hiragana + Katakana), (Latin + Han + Bopomofo), or (Latin + Han + Hangul).

ModeratelyRestrictive

Checks that the string contains only characters from the combinations (Latin + Cyrillic + Greek + Cherokee), (Latin + Han + Hiragana + Katakana), (Latin + Han + Bopomofo), or (Latin + Han + Hangul).

MinimallyRestrictive

Allows arbitrary mixtures of scripts.

Unrestrictive

Allows any valid identifiers, including characters outside of the Identifier Profile.

Instances

Bounded RestrictionLevel

Defined in Data.Text.ICU.Spoof

Methods

minBound :: RestrictionLevel

maxBound :: RestrictionLevel

Enum RestrictionLevel

Defined in Data.Text.ICU.Spoof

Methods

succ :: RestrictionLevel -> RestrictionLevel

pred :: RestrictionLevel -> RestrictionLevel

toEnum :: Int -> RestrictionLevel

fromEnum :: RestrictionLevel -> Int

enumFrom :: RestrictionLevel -> [RestrictionLevel]

enumFromThen :: RestrictionLevel -> RestrictionLevel -> [RestrictionLevel]

enumFromTo :: RestrictionLevel -> RestrictionLevel -> [RestrictionLevel]

enumFromThenTo :: RestrictionLevel -> RestrictionLevel -> RestrictionLevel -> [RestrictionLevel]

Eq RestrictionLevel

Defined in Data.Text.ICU.Spoof

Methods

(==) :: RestrictionLevel -> RestrictionLevel -> Bool

(/=) :: RestrictionLevel -> RestrictionLevel -> Bool

Show RestrictionLevel

Defined in Data.Text.ICU.Spoof

Methods

showsPrec :: Int -> RestrictionLevel -> ShowS

show :: RestrictionLevel -> String

showList :: [RestrictionLevel] -> ShowS

data SkeletonTypeOverride

Constructors

SkeletonSingleScript

By default, getSkeleton builds skeletons which catch visually confusable characters across multiple scripts. Pass this flag to override that behavior and build skeletons which catch visually confusable characters across single scripts.

SkeletonAnyCase

By default, getSkeleton assumes the input string has already been passed through toCaseFold and is lower-case. Pass this flag to override that behavior and allow upper and lower-case strings.

Instances

Bounded SkeletonTypeOverride

Defined in Data.Text.ICU.Spoof

Methods

minBound :: SkeletonTypeOverride

maxBound :: SkeletonTypeOverride

Enum SkeletonTypeOverride

Defined in Data.Text.ICU.Spoof

Methods

succ :: SkeletonTypeOverride -> SkeletonTypeOverride

pred :: SkeletonTypeOverride -> SkeletonTypeOverride

toEnum :: Int -> SkeletonTypeOverride

fromEnum :: SkeletonTypeOverride -> Int

enumFrom :: SkeletonTypeOverride -> [SkeletonTypeOverride]

enumFromThen :: SkeletonTypeOverride -> SkeletonTypeOverride -> [SkeletonTypeOverride]

enumFromTo :: SkeletonTypeOverride -> SkeletonTypeOverride -> [SkeletonTypeOverride]

enumFromThenTo :: SkeletonTypeOverride -> SkeletonTypeOverride -> SkeletonTypeOverride -> [SkeletonTypeOverride]

Eq SkeletonTypeOverride

Defined in Data.Text.ICU.Spoof

Methods

(==) :: SkeletonTypeOverride -> SkeletonTypeOverride -> Bool

(/=) :: SkeletonTypeOverride -> SkeletonTypeOverride -> Bool

Show SkeletonTypeOverride

Defined in Data.Text.ICU.Spoof

Methods

showsPrec :: Int -> SkeletonTypeOverride -> ShowS

show :: SkeletonTypeOverride -> String

showList :: [SkeletonTypeOverride] -> ShowS

Functions

open :: IO MSpoof

Open a spoof checker for checking Unicode strings for lookalike security issues with default options (all SpoofChecks except CharLimit).

openFromSerialized :: ByteString -> IO MSpoof

Open a spoof checker previously serialized to bytes using serialize. The returned MSpoof will retain a reference to the ForeignPtr inside the ByteString, so ensure its contents do not change for the lifetime of the lifetime of the returned value.

openFromSource :: (ByteString, ByteString) -> IO MSpoof

Open a spoof checker with custom rules given the UTF-8 encoded contents of the confusables.txt and confusablesWholeScript.txt files as described in Unicode UAX #39.

getSkeleton :: MSpoof -> Maybe SkeletonTypeOverride -> Text -> IO Text

Generates re-usable "skeleton" strings which can be used (via Unicode equality) to check if an identifier is confusable with some large set of existing identifiers.

If you cache the returned strings in storage, you must invalidate your cache any time the underlying confusables database changes (i.e., on ICU upgrade).

By default, assumes all input strings have been passed through toCaseFold and are lower-case. To change this, pass SkeletonAnyCase.

By default, builds skeletons which catch visually confusable characters across multiple scripts. Pass SkeletonSingleScript to override that behavior and build skeletons which catch visually confusable characters across single scripts.

getChecks :: MSpoof -> IO [SpoofCheck]

Get the checks performed by a spoof checker.

setChecks :: MSpoof -> [SpoofCheck] -> IO ()

Configure the checks performed by a spoof checker.

getRestrictionLevel :: MSpoof -> IO (Maybe RestrictionLevel)

Get the restriction level of a spoof checker.

setRestrictionLevel :: MSpoof -> RestrictionLevel -> IO ()

Configure the restriction level of a spoof checker.

getAllowedLocales :: MSpoof -> IO [String]

Get the list of locale names allowed to be used with a spoof checker. (We don't use LocaleName since the root and default locales have no meaning here.)

setAllowedLocales :: MSpoof -> [String] -> IO ()

Get the list of locale names allowed to be used with a spoof checker. (We don't use LocaleName since the root and default locales have no meaning here.)

areConfusable :: MSpoof -> Text -> Text -> IO SpoofCheckResult

Check if two strings could be confused with each other.

spoofCheck :: MSpoof -> Text -> IO SpoofCheckResult

Checks if a string could be confused with any other.

serialize :: MSpoof -> IO ByteString

Serializes the rules in this spoof checker to a byte array, suitable for re-use by openFromSerialized.

Only includes any data provided to openFromSource. Does not include any other state or configuration.