http-conduit-2.3.0: HTTP client package with conduit interface and HTTPS support.

Safe HaskellNone
LanguageHaskell98

Network.HTTP.Conduit

Contents

Description

Simpler API

The API below is rather low-level. The Network.HTTP.Simple module provides a higher-level API with built-in support for things like JSON request and response bodies. For most users, this will be an easier place to start. You can read the tutorial at:

https://haskell-lang.org/library/http-client

Lower-level API

This module contains everything you need to initiate HTTP connections. If you want a simple interface based on URLs, you can use simpleHttp. If you want raw power, http is the underlying workhorse of this package. Some examples:

-- Just download an HTML document and print it.
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L

main = simpleHttp "http://www.haskell.org/" >>= L.putStr

This example uses interleaved IO to write the response body to a file in constant memory space.

import Data.Conduit.Binary (sinkFile) -- Exported from the package conduit-extra
import Network.HTTP.Conduit
import qualified Data.Conduit as C
import Control.Monad.Trans.Resource (runResourceT)

main :: IO ()
main = do
     request <- parseRequest "http://google.com/"
     manager <- newManager tlsManagerSettings
     runResourceT $ do
         response <- http request manager
         responseBody response C.$$+- sinkFile "google.html"

The following headers are automatically set by this module, and should not be added to requestHeaders:

  • Cookie
  • Content-Length
  • Transfer-Encoding

Note: In previous versions, the Host header would be set by this module in all cases. Starting from 1.6.1, if a Host header is present in requestHeaders, it will be used in place of the header this module would have generated. This can be useful for calling a server which utilizes virtual hosting.

Use cookieJar If you want to supply cookies with your request:

{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import Network
import Data.Time.Clock
import Data.Time.Calendar
import qualified Control.Exception as E
import Network.HTTP.Types.Status (statusCode)

past :: UTCTime
past = UTCTime (ModifiedJulianDay 56200) (secondsToDiffTime 0)

future :: UTCTime
future = UTCTime (ModifiedJulianDay 562000) (secondsToDiffTime 0)

cookie :: Cookie
cookie = Cookie { cookie_name = "password_hash"
                , cookie_value = "abf472c35f8297fbcabf2911230001234fd2"
                , cookie_expiry_time = future
                , cookie_domain = "example.com"
                , cookie_path = "/"
                , cookie_creation_time = past
                , cookie_last_access_time = past
                , cookie_persistent = False
                , cookie_host_only = False
                , cookie_secure_only = False
                , cookie_http_only = False
                }

main = do
     request' <- parseRequest "http://example.com/secret-page"
     manager <- newManager tlsManagerSettings
     let request = request' { cookieJar = Just $ createCookieJar [cookie] }
     fmap Just (httpLbs request manager) `E.catch`
             (\ex -> case ex of
                 HttpExceptionRequest _ (StatusCodeException res _) ->
                     if statusCode (responseStatus res) == 403
                       then (putStrLn "login failed" >> return Nothing)
                       else return Nothing
                 _ -> E.throw ex)

Cookies are implemented according to RFC 6265.

Note that by default, the functions in this package will throw exceptions for non-2xx status codes. If you would like to avoid this, you should use checkStatus, e.g.:

import Data.Conduit.Binary (sinkFile)
import Network.HTTP.Conduit
import qualified Data.Conduit as C
import Network

main :: IO ()
main = do
     request' <- parseRequest "http://www.yesodweb.com/does-not-exist"
     let request = request' { checkStatus = \_ _ _ -> Nothing }
     manager <- newManager tlsManagerSettings
     res <- httpLbs request manager
     print res

By default, when connecting to websites using HTTPS, functions in this package will throw an exception if the TLS certificate doesn't validate. To continue the HTTPS transaction even if the TLS cerficate validation fails, you should use mkManagerSetttings as follows:

import Network.Connection (TLSSettings (..))
import Network.HTTP.Conduit

main :: IO ()
main = do
    request <- parseRequest "https://github.com/"
    let settings = mkManagerSettings (TLSSettingsSimple True False False) Nothing
    manager <- newManager settings
    res <- httpLbs request manager
    print res

For more information, please be sure to read the documentation in the Network.HTTP.Client module.

Synopsis

Perform a request

simpleHttp :: MonadIO m => String -> m ByteString Source #

Download the specified URL, following any redirects, and return the response body.

This function will throwIO an HttpException for any response with a non-2xx status code (besides 3xx redirects up to a limit of 10 redirects). It uses parseUrlThrow to parse the input. This function essentially wraps httpLbs.

Note: Even though this function returns a lazy bytestring, it does not utilize lazy I/O, and therefore the entire response body will live in memory. If you want constant memory usage, you'll need to use the conduit package and http directly.

Note: This function creates a new Manager. It should be avoided in production code.

httpLbs :: MonadIO m => Request -> Manager -> m (Response ByteString) Source #

Download the specified Request, returning the results as a Response.

This is a simplified version of http for the common case where you simply want the response data as a simple datatype. If you want more power, such as interleaved actions on the response body during download, you'll need to use http directly. This function is defined as:

httpLbs = lbsResponse <=< http

Even though the Response contains a lazy bytestring, this function does not utilize lazy I/O, and therefore the entire response body will live in memory. If you want constant memory usage, you'll need to use conduit packages's Source returned by http.

This function will throwIO an HttpException for any response with a non-2xx status code (besides 3xx redirects up to a limit of 10 redirects). This behavior can be modified by changing the checkStatus field of your request.

Note: Unlike previous versions, this function will perform redirects, as specified by the redirectCount setting.

http :: MonadResource m => Request -> Manager -> m (Response (ConduitM i ByteString m ())) Source #

Datatypes

data Proxy :: * #

Define a HTTP proxy, consisting of a hostname and port number.

Constructors

Proxy 

Fields

Instances

data RequestBody :: * #

When using one of the RequestBodyStream / RequestBodyStreamChunked constructors, you must ensure that the GivesPopper can be called multiple times. Usually this is not a problem.

The RequestBodyStreamChunked will send a chunked request body. Note that not all servers support this. Only use RequestBodyStreamChunked if you know the server you're sending to supports chunked request bodies.

Since 0.1.0

Constructors

RequestBodyLBS ByteString 
RequestBodyBS ByteString 
RequestBodyBuilder Int64 Builder 
RequestBodyStream Int64 (GivesPopper ()) 
RequestBodyStreamChunked (GivesPopper ()) 
RequestBodyIO (IO RequestBody)

Allows creation of a RequestBody inside the IO monad, which is useful for making easier APIs (like setRequestBodyFile).

Since: 0.4.28

Instances

Request

data Request :: * #

All information on how to connect to a host and what should be sent in the HTTP request.

If you simply wish to download from a URL, see parseRequest.

The constructor for this data type is not exposed. Instead, you should use either the defaultRequest value, or parseRequest to construct from a URL, and then use the records below to make modifications. This approach allows http-client to add configuration options without breaking backwards compatibility.

For example, to construct a POST request, you could do something like:

initReq <- parseRequest "http://www.example.com/path"
let req = initReq
            { method = "POST"
            }

For more information, please see http://www.yesodweb.com/book/settings-types.

Since 0.1.0

Instances

method :: Request -> Method #

HTTP request method, eg GET, POST.

Since 0.1.0

secure :: Request -> Bool #

Whether to use HTTPS (ie, SSL).

Since 0.1.0

host :: Request -> ByteString #

Requested host name, used for both the IP address to connect to and the host request header.

Since 0.1.0

port :: Request -> Int #

The port to connect to. Also used for generating the host request header.

Since 0.1.0

path :: Request -> ByteString #

Everything from the host to the query string.

Since 0.1.0

queryString :: Request -> ByteString #

Query string appended to the path.

Since 0.1.0

requestHeaders :: Request -> RequestHeaders #

Custom HTTP request headers

The Content-Length and Transfer-Encoding headers are set automatically by this module, and shall not be added to requestHeaders.

If not provided by the user, Host will automatically be set based on the host and port fields.

Moreover, the Accept-Encoding header is set implicitly to gzip for convenience by default. This behaviour can be overridden if needed, by setting the header explicitly to a different value. In order to omit the Accept-Header altogether, set it to the empty string "". If you need an empty Accept-Header (i.e. requesting the identity encoding), set it to a non-empty white-space string, e.g. " ". See RFC 2616 section 14.3 for details about the semantics of the Accept-Header field. If you request a content-encoding not supported by this module, you will have to decode it yourself (see also the decompress field).

Note: Multiple header fields with the same field-name will result in multiple header fields being sent and therefore it's the responsibility of the client code to ensure that the rules from RFC 2616 section 4.2 are honoured.

Since 0.1.0

requestBody :: Request -> RequestBody #

Request body to be sent to the server.

Since 0.1.0

proxy :: Request -> Maybe Proxy #

Optional HTTP proxy.

Since 0.1.0

hostAddress :: Request -> Maybe HostAddress #

Optional resolved host address. May not be used by all backends.

Since 0.1.0

rawBody :: Request -> Bool #

If True, a chunked and/or gzipped body will not be decoded. Use with caution.

Since 0.1.0

decompress :: Request -> ByteString -> Bool #

Predicate to specify whether gzipped data should be decompressed on the fly (see alwaysDecompress and browserDecompress). Argument is the mime type. Default: browserDecompress.

Since 0.1.0

redirectCount :: Request -> Int #

How many redirects to follow when getting a resource. 0 means follow no redirects. Default value: 10.

Since 0.1.0

checkResponse :: Request -> Request -> Response BodyReader -> IO () #

Check the response immediately after receiving the status and headers. This can be useful for throwing exceptions on non-success status codes.

In previous versions of http-client, this went under the name checkStatus, but was renamed to avoid confusion about the new default behavior (doing nothing).

Since: 0.5.0

responseTimeout :: Request -> ResponseTimeout #

Number of microseconds to wait for a response. If Nothing, will wait indefinitely. Default: use managerResponseTimeout (which by default is 30 seconds).

Since 0.1.0

cookieJar :: Request -> Maybe CookieJar #

A user-defined cookie jar. If Nothing, no cookie handling will take place, "Cookie" headers in requestHeaders will be sent raw, and responseCookieJar will be empty.

Since 0.1.0

requestVersion :: Request -> HttpVersion #

HTTP version to send to server.

Default: HTTP 1.1

Since 0.4.3

setQueryString :: [(ByteString, Maybe ByteString)] -> Request -> Request #

Set the query string to the given key/value pairs.

Since 0.3.6

Request body

requestBodySource :: Int64 -> ConduitM () ByteString (ResourceT IO) () -> RequestBody Source #

requestBodySourceChunked :: ConduitM () ByteString (ResourceT IO) () -> RequestBody Source #

requestBodySourceIO :: Int64 -> ConduitM () ByteString IO () -> RequestBody Source #

requestBodySourceChunkedIO :: ConduitM () ByteString IO () -> RequestBody Source #

Response

data Response body :: * -> * #

A simple representation of the HTTP response.

Since 0.1.0

Instances

Functor Response 

Methods

fmap :: (a -> b) -> Response a -> Response b #

(<$) :: a -> Response b -> Response a #

Foldable Response 

Methods

fold :: Monoid m => Response m -> m #

foldMap :: Monoid m => (a -> m) -> Response a -> m #

foldr :: (a -> b -> b) -> b -> Response a -> b #

foldr' :: (a -> b -> b) -> b -> Response a -> b #

foldl :: (b -> a -> b) -> b -> Response a -> b #

foldl' :: (b -> a -> b) -> b -> Response a -> b #

foldr1 :: (a -> a -> a) -> Response a -> a #

foldl1 :: (a -> a -> a) -> Response a -> a #

toList :: Response a -> [a] #

null :: Response a -> Bool #

length :: Response a -> Int #

elem :: Eq a => a -> Response a -> Bool #

maximum :: Ord a => Response a -> a #

minimum :: Ord a => Response a -> a #

sum :: Num a => Response a -> a #

product :: Num a => Response a -> a #

Traversable Response 

Methods

traverse :: Applicative f => (a -> f b) -> Response a -> f (Response b) #

sequenceA :: Applicative f => Response (f a) -> f (Response a) #

mapM :: Monad m => (a -> m b) -> Response a -> m (Response b) #

sequence :: Monad m => Response (m a) -> m (Response a) #

Eq body => Eq (Response body) 

Methods

(==) :: Response body -> Response body -> Bool #

(/=) :: Response body -> Response body -> Bool #

Show body => Show (Response body) 

Methods

showsPrec :: Int -> Response body -> ShowS #

show :: Response body -> String #

showList :: [Response body] -> ShowS #

responseStatus :: Response body -> Status #

Status code of the response.

Since 0.1.0

responseVersion :: Response body -> HttpVersion #

HTTP version used by the server.

Since 0.1.0

responseHeaders :: Response body -> ResponseHeaders #

Response headers sent by the server.

Since 0.1.0

responseBody :: Response body -> body #

Response body sent by the server.

Since 0.1.0

responseCookieJar :: Response body -> CookieJar #

Cookies set on the client after interacting with the server. If cookies have been disabled by setting cookieJar to Nothing, then this will always be empty.

Since 0.1.0

Manager

data Manager :: * #

Keeps track of open connections for keep-alive.

If possible, you should share a single Manager between multiple threads and requests.

Since 0.1.0

Instances

newManager :: ManagerSettings -> IO Manager #

Create a Manager. The Manager will be shut down automatically via garbage collection.

Creating a new Manager is a relatively expensive operation, you are advised to share a single Manager between requests instead.

The first argument to this function is often defaultManagerSettings, though add-on libraries may provide a recommended replacement.

Since 0.1.0

closeManager :: Manager -> IO () #

Close all connections in a Manager.

Note that this doesn't affect currently in-flight connections, meaning you can safely use it without hurting any queries you may have concurrently running.

Since 0.1.0

Settings

data ManagerSettings :: * #

Settings for a Manager. Please use the defaultManagerSettings function and then modify individual settings. For more information, see http://www.yesodweb.com/book/settings-types.

Since 0.1.0

tlsManagerSettings :: ManagerSettings #

Default TLS-enabled manager settings

mkManagerSettings :: TLSSettings -> Maybe SockSettings -> ManagerSettings #

Create a TLS-enabled ManagerSettings with the given TLSSettings and SockSettings

managerConnCount :: ManagerSettings -> Int #

Number of connections to a single host to keep alive. Default: 10.

Since 0.1.0

managerResponseTimeout :: ManagerSettings -> ResponseTimeout #

Default timeout to be applied to requests which do not provide a timeout value.

Default is 30 seconds

Since: 0.5.0

managerTlsConnection :: ManagerSettings -> IO (Maybe HostAddress -> String -> Int -> IO Connection) #

Create a TLS connection. Default behavior: throw an exception that TLS is not supported.

Since 0.1.0

Response timeout

data ResponseTimeout :: * #

How to deal with timing out a response

Since: 0.5.0

Instances

responseTimeoutMicro :: Int -> ResponseTimeout #

Specify a response timeout in microseconds

Since: 0.5.0

responseTimeoutNone :: ResponseTimeout #

Do not have a response timeout

Since: 0.5.0

responseTimeoutDefault :: ResponseTimeout #

Use the default response timeout

When used on a Request, means: use the manager's timeout value

When used on a ManagerSettings, means: default to 30 seconds

Since: 0.5.0

Cookies

data Cookie :: * #

Constructors

Cookie 

Fields

Instances

data CookieJar :: * #

Instances

createCookieJar :: [Cookie] -> CookieJar #

destroyCookieJar :: CookieJar -> [Cookie] #

Utility functions

parseUrl :: MonadThrow m => String -> m Request #

Deprecated synonym for parseUrlThrow. You probably want parseRequest or parseRequest_ instead.

Since: 0.1.0

parseUrlThrow :: MonadThrow m => String -> m Request #

Same as parseRequest, except will throw an HttpException in the event of a non-2XX response.

Since: 0.4.30

parseRequest :: MonadThrow m => String -> m Request #

Convert a URL into a Request.

This defaults some of the values in Request, such as setting method to GET and requestHeaders to [].

Since this function uses MonadThrow, the return monad can be anything that is an instance of MonadThrow, such as IO or Maybe.

You can place the request method at the beginning of the URL separated by a space, e.g.:

@@ parseRequest "POST http://httpbin.org/post" @@

Note that the request method must be provided as all capital letters.

Request created by this function won't cause exceptions on non-2XX response status codes.

To create a request which throws on non-2XX status codes, see parseUrlThrow

Since: 0.4.30

parseRequest_ :: String -> Request #

Same as parseRequest, but in the cases of a parse error generates an impure exception. Mostly useful for static strings which are known to be correctly formatted.

defaultRequest :: Request #

A default request value

Since: 0.4.30

applyBasicAuth :: ByteString -> ByteString -> Request -> Request #

Add a Basic Auth header (with the specified user name and password) to the given Request. Ignore error handling:

 applyBasicAuth "user" "pass" $ parseRequest_ url

NOTE: The function applyDigestAuth is provided by the http-client-tls package instead of this package due to extra dependencies. Please use that package if you need to use digest authentication.

Since 0.1.0

addProxy :: ByteString -> Int -> Request -> Request #

Add a proxy to the Request so that the Request when executed will use the provided proxy.

Since 0.1.0

lbsResponse :: Monad m => Response (ConduitM () ByteString m ()) -> m (Response ByteString) Source #

getRedirectedRequest :: Request -> ResponseHeaders -> CookieJar -> Int -> Maybe Request #

If a request is a redirection (status code 3xx) this function will create a new request from the old request, the server headers returned with the redirection, and the redirection code itself. This function returns Nothing if the code is not a 3xx, there is no location header included, or if the redirected response couldn't be parsed with parseRequest.

If a user of this library wants to know the url chain that results from a specific request, that user has to re-implement the redirect-following logic themselves. An example of that might look like this:

myHttp req man = do
   (res, redirectRequests) <- (`runStateT` []) $
        'httpRedirect'
            9000
            (\req' -> do
               res <- http req'{redirectCount=0} man
               modify (\rqs -> req' : rqs)
               return (res, getRedirectedRequest req' (responseHeaders res) (responseCookieJar res) (W.statusCode (responseStatus res))
               )
            'lift'
            req
   applyCheckStatus (checkStatus req) res
   return redirectRequests

Decompression predicates

alwaysDecompress :: ByteString -> Bool #

Always decompress a compressed stream.

browserDecompress :: ByteString -> Bool #

Decompress a compressed stream unless the content-type is 'application/x-tar'.

Request bodies

Network.HTTP.Client.MultipartFormData provides an API for building form-data request bodies.

urlEncodedBody :: [(ByteString, ByteString)] -> Request -> Request #

Add url-encoded parameters to the Request.

This sets a new requestBody, adds a content-type request header and changes the method to POST.

Since 0.1.0

Exceptions

data HttpException :: * #

An exception which may be generated by this library

Since: 0.5.0

Constructors

HttpExceptionRequest Request HttpExceptionContent

Most exceptions are specific to a Request. Inspect the HttpExceptionContent value for details on what occurred.

Since: 0.5.0

InvalidUrlException String String

A URL (first field) is invalid for a given reason (second argument).

Since: 0.5.0

Instances

data HttpExceptionContent :: * #

Constructors

StatusCodeException (Response ()) ByteString

Generated by the parseUrlThrow function when the server returns a non-2XX response status code.

May include the beginning of the response body.

Since: 0.5.0

TooManyRedirects [Response ByteString]

The server responded with too many redirects for a request.

Contains the list of encountered responses containing redirects in reverse chronological order; including last redirect, which triggered the exception and was not followed.

Since: 0.5.0

OverlongHeaders

Either too many headers, or too many total bytes in a single header, were returned by the server, and the memory exhaustion protection in this library has kicked in.

Since: 0.5.0

ResponseTimeout

The server took too long to return a response. This can be altered via responseTimeout or managerResponseTimeout.

Since: 0.5.0

ConnectionTimeout

Attempting to connect to the server timed out.

Since: 0.5.0

ConnectionFailure SomeException

An exception occured when trying to connect to the server.

Since: 0.5.0

InvalidStatusLine ByteString

The status line returned by the server could not be parsed.

Since: 0.5.0

InvalidHeader ByteString

The given response header line could not be parsed

Since: 0.5.0

InternalException SomeException

An exception was raised by an underlying library when performing the request. Most often, this is caused by a failing socket action or a TLS exception.

Since: 0.5.0

ProxyConnectException ByteString Int Status

A non-200 status code was returned when trying to connect to the proxy server on the given host and port.

Since: 0.5.0

NoResponseDataReceived

No response data was received from the server at all. This exception may deserve special handling within the library, since it may indicate that a pipelining has been used, and a connection thought to be open was in fact closed.

Since: 0.5.0

TlsNotSupported

Exception thrown when using a Manager which does not have support for secure connections. Typically, you will want to use tlsManagerSettings from http-client-tls to overcome this.

Since: 0.5.0

WrongRequestBodyStreamSize Word64 Word64

The request body provided did not match the expected size.

Provides the expected and actual size.

Since: 0.4.31

ResponseBodyTooShort Word64 Word64

The returned response body is too short. Provides the expected size and actual size.

Since: 0.5.0

InvalidChunkHeaders

A chunked response body had invalid headers.

Since: 0.5.0

IncompleteHeaders

An incomplete set of response headers were returned.

Since: 0.5.0

InvalidDestinationHost ByteString

The host we tried to connect to is invalid (e.g., an empty string).

HttpZlibException ZlibException

An exception was thrown when inflating a response body.

Since: 0.5.0

InvalidProxyEnvironmentVariable Text Text

Values in the proxy environment variable were invalid. Provides the environment variable name and its value.

Since: 0.5.0

ConnectionClosed

Attempted to use a Connection which was already closed

Since: 0.5.0

InvalidProxySettings Text

Proxy settings are not valid (Windows specific currently) @since 0.5.7

Instances