Network.AWS.S3.StreamingUpload

Documentation

streamUpload

Arguments

:: (MonadUnliftIO m, MonadAWS m)

=> Maybe ChunkSize

Optional chunk size

-> CreateMultipartUpload

Upload location

-> ConduitT ByteString Void m (Either (AbortMultipartUploadResponse, SomeException) CompleteMultipartUploadResponse)

Given a CreateMultipartUpload, creates a Sink which will sequentially upload the data streamed in in chunks of at least minimumChunkSize and return either the CompleteMultipartUploadResponse, or if an exception is thrown, AbortMultipartUploadResponse and the exception as SomeException. If aborting the upload also fails then the exception caused by the call to abort will be thrown.

ListMultipartUploads can be used to list any pending uploads - it is important to abort multipart uploads because you will be charged for storage of the parts until it is completed or aborted. See the AWS documentation for more details.

May throw Error

type ChunkSize = Int

minimumChunkSize :: ChunkSize

Minimum size of data which will be sent in a single part, currently 6MB

type NumThreads = Int

concurrentUpload

Arguments

:: MonadAWS m

=> Maybe ChunkSize

Optional chunk size

-> Maybe NumThreads

Optional number of threads to upload with

-> UploadLocation

Whether to upload a file on disk or a ByteString that's already in memory.

-> CreateMultipartUpload

Description of where to upload.

-> m CompleteMultipartUploadResponse

Allows a file or ByteString to be uploaded concurrently, using the async library. The chunk size may optionally be specified, but will be at least minimumChunkSize, and may be made larger than if the ByteString or file is larger enough to cause more than 10,000 chunks.

Files are mmapped into chunkSize chunks and each chunk is uploaded in parallel. This considerably reduces the memory necessary compared to reading the contents into memory as a strict ByteString. The usual caveats about mmaped files apply: if the file is modified during this operation, the data may become corrupt.

May throw Error, or IOError; an attempt is made to cancel the multipart upload on any error, but this may also fail if, for example, the network connection has been broken. See abortAllUploads for a crude cleanup method.

data UploadLocation

Specifies whether to upload a file or 'ByteString

Constructors

FP FilePath

A file to be uploaded

BS ByteString

A strict ByteString

abortAllUploads :: MonadAWS m => BucketName -> m ()

Aborts all uploads in a given bucket - useful for cleaning up.