http-conduit-downloader: HTTP downloader tailored for web-crawler needs.
HTTP/HTTPS downloader built on top of http-client
and used in https://bazqux.com crawler.
Previously it was based on http-conduit
(hence the name) but since
all the necessary parts are in http-client
now http-conduit
is no
longer used.
Handles all possible http-client exceptions and returns human readable error messages.
Handles some web server bugs (returning
deflate
data instead ofgzip
, invalidgzip
encoding).Uses OpenSSL instead of
tls
package (sincetls
doesn't handle all sites and works slower than OpenSSL).Ignores invalid SSL sertificates.
Receives data in 32k chunks internally to reduce memory fragmentation on many parallel downloads.
Download timeout.
Total download size limit.
Returns HTTP headers for subsequent redownloads and handles 'Not modified' results.
Can be used with external DNS resolver (e.g.
concurrent-dns-cache
).
Downloads
- http-conduit-downloader-1.1.5.tar.gz [browse] (Cabal source package)
- Package description (as included in the package)
Maintainer's Corner
For package maintainers and hackage trustees
Candidates
Versions [RSS] | 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.0.9, 1.0.10, 1.0.11, 1.0.12, 1.0.13, 1.0.14, 1.0.15, 1.0.16, 1.0.17, 1.0.18, 1.0.19, 1.0.20, 1.0.21, 1.0.22, 1.0.23, 1.0.24, 1.0.25, 1.0.30, 1.0.31, 1.0.32, 1.0.33, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5 |
---|---|
Dependencies | base (>=4 && <5), bytestring, data-default, HsOpenSSL (>=0.11.5), http-client (>=0.7.16 && <0.7.17), http-client-openssl (>=0.3.3 && <0.3.4), http-types, network (>=2.6), network-uri (>=2.6), text, time (>=1.5.0), zlib [details] |
License | BSD-3-Clause |
Author | Vladimir Shabanov <dev@vshabanov.com> |
Maintainer | Vladimir Shabanov <dev@vshabanov.com> |
Category | Web |
Home page | https://github.com/bazqux/http-conduit-downloader |
Source repo | head: git clone https://github.com/bazqux/http-conduit-downloader |
Uploaded | by VladimirShabanov at 2024-02-14T21:15:18Z |
Distributions | |
Reverse Dependencies | 2 direct, 0 indirect [details] |
Downloads | 22882 total (55 in the last 30 days) |
Rating | 2.0 (votes: 1) [estimated by Bayesian average] |
Your Rating | |
Status | Docs available [build log] Last success reported on 2024-02-14 [all 1 reports] |