inf-backprop-0.1.0.0: Automatic differentiation and backpropagation.
Copyright(C) 2023 Alexey Tochin
LicenseBSD3 (see the file LICENSE)
MaintainerAlexey Tochin <Alexey.Tochin@gmail.com>
Safe HaskellSafe-Inferred
LanguageHaskell2010
Extensions
  • MonoLocalBinds
  • ScopedTypeVariables
  • TypeFamilies
  • GADTs
  • GADTSyntax
  • ConstraintKinds
  • InstanceSigs
  • DeriveFunctor
  • TypeSynonymInstances
  • FlexibleContexts
  • FlexibleInstances
  • ConstrainedClassMethods
  • MultiParamTypeClasses
  • KindSignatures
  • TupleSections
  • RankNTypes
  • ExplicitNamespaces
  • ExplicitForAll

InfBackprop

Description

Automatic differentiation and backpropagation. See Tutorial for details.

Synopsis

Base

Types

data Backprop cat input output Source #

Backprop morphism. Base type for an infinitely differentiable object. It depends on categorical type cat that is mostly common (->), see BackpropFunc which by it's definition is equivalent to

data BackpropFunc input output = forall cache. MkBackpropFunc {
 call     :: input -> output,
 forward  :: BackpropFunc input (output, cache),
 backward :: BackpropFunc (output, cache) input
}

The diagram below illustrates the how it works for the first derivative. Consider the role of function f in the derivative of the composition g(f(h(...))).

  h        ·                  f                   ·        g
           ·                                      ·
           ·               forward                ·
           · --- input  >-----+-----> output >--- ·
           ·                  V                   ·
 ...       ·                  |                   ·       ...
           ·                  | cache             ·
           ·                  |                   ·
           ·                  V                   ·
           · --< dInput <-----+-----< dOutput <-- ·
           ·               backward               ·

Notice that forward and backward are of type BackpropFunc but not (->). This is needed for further differentiation. However for the first derivative this difference can be ignored.

The return type of forward contains additional term cache. It is needed to save and transfer data calculated in the forward step to the backward step for reuse. See an example in

Differentiation with logging section .

Remark

Expand

Mathematically speaking we have to distinguish the types for forward and for backward methods because the second acts on the cotangent bundle. However, for simplicity and due to technical reasons we identify the types input and dInput as well as output and dOutput which is enough for our purposes because these types are usually real numbers or arrays of real numbers.

Constructors

forall cache. MkBackprop (cat input output) (Backprop cat input (output, cache)) (Backprop cat (output, cache) input) 

Instances

Instances details
(Isomorphism cat, CatBiFunctor (,) cat) => Category (Backprop cat :: Type -> Type -> Type) Source # 
Instance details

Defined in InfBackprop.Common

Methods

id :: forall (a :: k). Backprop cat a a #

(.) :: forall (b :: k) (c :: k) (a :: k). Backprop cat b c -> Backprop cat a b -> Backprop cat a c #

(Isomorphism cat, CatBiFunctor (,) cat) => CatBiFunctor (,) (Backprop cat) Source # 
Instance details

Defined in InfBackprop.Common

Methods

(***) :: Backprop cat a1 b1 -> Backprop cat a2 b2 -> Backprop cat (a1, a2) (b1, b2) Source #

first :: Backprop cat a b -> Backprop cat (a, c) (b, c) Source #

second :: Backprop cat a b -> Backprop cat (c, a) (c, b) Source #

(Isomorphism cat, CatBiFunctor (,) cat) => Isomorphism (Backprop cat) Source # 
Instance details

Defined in InfBackprop.Common

Methods

iso :: IsomorphicTo a b => Backprop cat a b Source #

type BackpropFunc = Backprop (->) Source #

Infinitely differentiable function. The definition of the type synonym is equivalent to

data BackpropFunc input output = forall cache. MkBackpropFunc {
   call     :: input -> output,
   forward  :: BackpropFunc input (output, cache),
   backward :: BackpropFunc (output, cache) input
 }

See Backprop for details.

Examples of usage

Expand
>>> import Prelude (fmap, Float)
>>> import InfBackprop (pow, call, derivative)
>>> myFunc = pow 2 :: BackpropFunc Float Float
>>> f = call myFunc :: Float -> Float
>>> fmap f [-3, -2, -1, 0, 1, 2, 3]
[9.0,4.0,1.0,0.0,1.0,4.0,9.0]
>>> df = derivative myFunc :: Float -> Float
>>> fmap df [-3, -2, -1, 0, 1, 2, 3]
[-6.0,-4.0,-2.0,0.0,2.0,4.0,6.0]

call :: Backprop cat input output -> cat input output Source #

Simple internal category object extraction.

forward :: Backprop cat input output -> Backprop cat input (output, cache) Source #

Returns forward category. In the case cat = (->), the method coincides with Backprop cat input output itself but the output contains an additional data term cache with some calculation result that can be reused on in backward.

backward :: Backprop cat input output -> Backprop cat (output, cache) input Source #

Returns backward category. In the case cat = (->), the method takes the additional data term cache that is calculated in forward.

derivative :: (Isomorphism cat, CatBiFunctor (,) cat, StartBackprop cat y) => Backprop cat x y -> cat x x Source #

Backpropagation derivative as categorical object. If cat is (->) the output is simply a function.

Examples of usage

Expand
>>> import InfBackprop (sin)
>>> import Prelude (Float)
>>> derivative sin (0 :: Float)
1.0

derivativeN :: (Isomorphism cat, CatBiFunctor (,) cat, StartBackprop cat x) => Natural -> Backprop cat x x -> cat x x Source #

Backpropagation derivative of order n as categorical object. If cat is (->) the output is simply a function.

Examples of usage

Expand
>>> import InfBackprop (pow, const)
>>> import Prelude (Float, fmap)
>>> myFunc = (pow 2) :: Backprop (->) Float Float
>>> fmap (derivativeN 0 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[9.0,4.0,1.0,0.0,1.0,4.0,9.0]
>>> fmap (derivativeN 1 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[-6.0,-4.0,-2.0,0.0,2.0,4.0,6.0]
>>> fmap (derivativeN 2 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[2.0,2.0,2.0,2.0,2.0,2.0,2.0]
>>> fmap (derivativeN 3 myFunc) [-3, -2, -1, 0, 1, 2, 3]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0]

Categorical Bifunctor

(***) :: CatBiFunctor p cat => cat a1 b1 -> cat a2 b2 -> cat (p a1 a2) (p b1 b2) Source #

Categorical generalization of

bimap :: (a1 -> b1) -> (a2 -> b2) -> (p a1 a2 -> p c1 c2)

borrowed from arrows.

first :: CatBiFunctor p cat => cat a b -> cat (p a c) (p b c) Source #

Categorical generalization of

first :: (a -> b) -> (p a c -> p c b)

borrowed from arrows.

second :: CatBiFunctor p cat => cat a b -> cat (p c a) (p c b) Source #

Categorical generalization of

second :: (a -> b) -> (p a c -> p c b)

borrowed from arrows.

Differentiable functions

Elementary functions

const :: forall c x. (Additive c, Additive x) => c -> BackpropFunc x c Source #

Infinitely differentiable constant function.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative, derivativeN)
>>> call (const 5) ()
5
>>> derivative (const (5 :: Float)) 42
0
>>> derivativeN 2 (const (5 :: Float)) 42
0.0

linear :: forall x. Distributive x => x -> BackpropFunc x x Source #

Linear differentiable function.

Examples of usage

Expand
>>> import Prelude (fmap, Float)
>>> import InfBackprop (pow, call, derivative)
>>> myFunc = linear 2 :: BackpropFunc Float Float
>>> f = call myFunc :: Float -> Float
>>> fmap f [-3, -2, -1, 0, 1, 2, 3]
[-6.0,-4.0,-2.0,0.0,2.0,4.0,6.0]
>>> df = derivative myFunc :: Float -> Float
>>> fmap df [-3, -2, -1, 0, 1, 2, 3]
[2.0,2.0,2.0,2.0,2.0,2.0,2.0]

(+) :: forall x. Additive x => BackpropFunc (x, x) x Source #

Summation differentiable binary operation.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call (+) (2, 3) :: Float
5.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> y = variable "y"
>>> derivative (+) (x, y)
(1,1)

(-) :: forall x. Subtractive x => BackpropFunc (x, x) x Source #

Subtraction differentiable binary operation.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call (-) (5, 3) :: Float
2.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> y = variable "y"
>>> derivative (-) (x, y)
(1,-(1))

negate :: forall x. Subtractive x => BackpropFunc x x Source #

Negate differentiable function.

Examples of usage

Expand
>>> import Prelude (Float, ($))
>>> import InfBackprop (call, derivative)
>>> call negate 42 :: Float
-42.0
>>> derivative negate 42 :: Float
-1.0

(*) :: Distributive x => BackpropFunc (x, x) x Source #

Product binnary operation

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call (*) (2, 3) :: Float
6.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> y = variable "y"
>>> derivative (*) (x, y)
(1·y,1·x)

(/) :: forall x. (Divisive x, Distributive x, Subtractive x) => BackpropFunc (x, x) x Source #

Division binary differentiable operation

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call (/) (6, 3) :: Float
2.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> y = variable "y"
>>> derivative (/) (x, y)
(1·(1/y),1·(-(x)·(1/(y·y))))

Tuple manipulations

dup :: forall x. Additive x => BackpropFunc x (x, x) Source #

Duplication differentiable operation.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call dup (42.0 :: Float)
(42.0,42.0)
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> derivative (dup >>> (*)) x
(1·x)+(1·x)

setFirst :: forall x y c. Additive c => c -> BackpropFunc (c, x) y -> BackpropFunc x y Source #

Transforms a 2-argument differentiable function into a single argument function by fixing its first argument.

>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call (setFirst 8 (/)) 4 :: Float
2.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> y = variable "y"
>>> derivative (setFirst x (*)) y
1·x

setSecond :: forall x y c. Additive c => c -> BackpropFunc (x, c) y -> BackpropFunc x y Source #

Transforms a 2-argument differentiable function into a single argument function by fixing its second argument.

>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call (setSecond 4 (/)) 8 :: Float
2.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> y = variable "y"
>>> derivative (setSecond y (*)) x
1·y

forget :: forall x. Additive x => BackpropFunc x () Source #

Transforms any function to unit (). It is not differentiable until StartBackprop is defined for (). However forget is useful if need to remove some data in the differentiable pipeline.

Examples of usage

Expand
>>> import InfBackprop (call, derivative)
>>> f = first forget >>> (iso :: BackpropFunc ((), a) a) :: Additive a => BackpropFunc (a, a) a
>>> call f (24, 42)
42
>>> derivative f (24, 42)
(0,1)

forgetFirst :: forall x y. Additive x => BackpropFunc (x, y) y Source #

Remove the first element of a tuple.

Examples of usage

Expand
>>> import InfBackprop (call, derivative)
>>> call forgetFirst (24, 42)
42
>>> derivative forgetFirst (24, 42)
(0,1)

forgetSecond :: forall x y. Additive y => BackpropFunc (x, y) x Source #

Remove the second element of a tuple.

Examples of usage

Expand
>>> import InfBackprop (call, derivative)
>>> call forgetSecond (24, 42)
24
>>> derivative forgetSecond (24, 42)
(1,0)

Exponential family functions

log :: ExpField x => BackpropFunc x x Source #

Natural logarithm differentiable function.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call log 10 :: Float
2.3025851
>>> derivative log 10 :: Float
0.1

logBase :: ExpField a => BackpropFunc (a, a) a Source #

Natural logarithm differentiable function.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call logBase (2, 8) :: Float
3.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> n = variable "n"
>>> derivative logBase (n, x)
((1·(-(log(x))·(1/(log(n)·log(n)))))·(1/n),(1·(1/log(n)))·(1/x))

exp :: forall x. ExpField x => BackpropFunc x x Source #

Natural logarithm differentiable function.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call exp 2
7.38905609893065
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> derivative exp x
1·exp(x)

(**) :: forall a. (ExpField a, FromIntegral a Integer) => BackpropFunc (a, a) a Source #

Power binary differentiable operation.

Examples of usage

Expand
>>> import Prelude (Float)
>>> import NumHask (half)
>>> import InfBackprop (call, derivative)
>>> call (**) (0.5, 9) :: Float
3.0
>>> import Debug.SimpleExpr.Expr (variable)
>>> x = variable "x"
>>> n = variable "n"
>>> derivative (**) (n, x)
(1·(n·(x^(n-1))),1·((x^n)·log(x)))

pow :: forall x. (Divisive x, Distributive x, Subtractive x, FromIntegral x Integer) => Integer -> BackpropFunc x x Source #

Integer power differentiable operation

Examples of usage

Expand
>>> import Prelude (Float)
>>> import InfBackprop (call, derivative)
>>> call (pow 3) 2 :: Float
8.0
>>> derivative (pow 3) 2 :: Float
12.0

Trigonometric functions

cos :: TrigField x => BackpropFunc x x Source #

Cosine differentiable function.

sin :: TrigField x => BackpropFunc x x Source #

Sine differentiable function

tan :: TrigField x => BackpropFunc x x Source #

Tangent differentiable function.

asin :: (TrigField x, ExpField x) => BackpropFunc x x Source #

Arcsine differentiable function.

acos :: (TrigField x, ExpField x) => BackpropFunc x x Source #

Arccosine differentiable function.

atan :: TrigField x => BackpropFunc x x Source #

Arctangent differentiable function.

atan2 :: TrigField a => BackpropFunc (a, a) a Source #

2-argument arctangent differentiable function.

sinh :: TrigField x => BackpropFunc x x Source #

Hyperbolic sine differentiable function.

cosh :: TrigField x => BackpropFunc x x Source #

Hyperbolic cosine differentiable function.

tanh :: TrigField x => BackpropFunc x x Source #

Hyperbolic tanget differentiable function.

asinh :: (TrigField x, ExpField x) => BackpropFunc x x Source #

Hyperbolic arcsine differentiable function.

acosh :: (TrigField x, ExpField x) => BackpropFunc x x Source #

Hyperbolic arccosine differentiable function.

atanh :: TrigField x => BackpropFunc x x Source #

Hyperbolic arctangent differentiable function.

Monadic differentiable functions

pureBackprop :: forall a b m. Monad m => Backprop (->) a b -> Backprop (Kleisli m) a b Source #

Lifts a backprop function morphism to the corresponding pure Kleisli morphism.

backpropExpr :: String -> BackpropFunc SimpleExpr SimpleExpr Source #

Returns symbolically differentiable Simple Expression.

Examples of usage

Expand
>>> import Control.Arrow (runKleisli)
>>> import Control.Monad.Logger (runStdoutLoggingT)
>>> import Debug.SimpleExpr.Expr (variable)
>>> import InfBackprop (call, derivative, backpropExpr)
>>> x = variable "x"
>>> f = backpropExpr "f"
>>> call f x
f(x)
>>> derivative f x
1·f'(x)

loggingBackpropExpr :: forall m. MonadLogger m => String -> Backprop (Kleisli m) SimpleExpr SimpleExpr Source #

Returns symbolically differentiable logging symbolic function.

Examples of usage

Expand
>>> import Control.Arrow (runKleisli)
>>> import Control.Monad.Logger (runStdoutLoggingT)
>>> import Debug.SimpleExpr.Expr (variable)
>>> import InfBackprop (call, derivative)
>>> x = variable "x"
>>> f = loggingBackpropExpr "f"
>>> runStdoutLoggingT $ runKleisli (call f) x
[Info] Calculating f of x => f(x)
f(x)
>>> runStdoutLoggingT $ runKleisli (derivative f) x
[Info] Calculating f of x => f(x)
[Info] Calculating f' of x => f'(x)
[Info] Calculating multiplication of 1 and f'(x) => 1·f'(x)
1·f'(x)

Tools

pureKleisli :: Monad m => (a -> b) -> Kleisli m a b Source #

Returns pure Kleisli morphism given a map.

Examples of usage

Expand
>>> import Control.Arrow (runKleisli)
>>> import Control.Monad.Logger (runStdoutLoggingT)
>>> loggingDup = pureKleisli (\x -> (x, x))
>>> runStdoutLoggingT $ runKleisli loggingDup 42
(42,42)

simpleDifferentiable :: forall x. Distributive x => (x -> x) -> BackpropFunc x x -> BackpropFunc x x Source #

Returns a differentiable morphism given forward function and backpropagation derivative differential morphism.

Examples of usage

Expand
>>> import qualified NumHask as NH
>>> cos = simpleDifferentiable NH.cos (sin >>> negate)