从ASP.NET Core Web API将多GB文件流式传输到AWS S3

我希望从ASP.NET Core Web API在AWS S3存储桶中创建一个大型(多GB)文件.该文件足够大,我希望在将Stream上传到AWS S3之前不将其加载到内存中.

使用PutObjectAsync()我强制在将Stream传递给AWS SDK之前预先填充Stream,如下图所示：

var putObjectRequest = new PutObjectRequest
{
    BucketName = "my-s3-bucket",Key = "my-file-name.txt",InputStream = stream
};
var putObjectResponse = await amazonS3Client.PutObjectAsync(putObjectRequest);

我理想的模式将涉及AWS SDK返回StreamWriter(各种类型)我可以多次写入(),然后在完成后返回Finalize().

关于我的挑战的两个问题：

>我是否误以为在调用PutObjectAsync()之前必须预先填充Stream？
>我应该如何上传我的大(多GB)文件？

解决方法

对于这种情况,AWS docs提供了两种选择：

> Using the AWS .NET SDK for Multipart Upload (High-Level API)
> Using the AWS .NET SDK for Multipart Upload (Low-Level API)

高级API只是建议您创建一个指定了PartSize的TransferUtilityUploadRequest,这样类本身就可以上传文件而无需自己维护上传.在这种情况下,您可以通过订阅StreamTransferProgress事件获得分段上传的进度.您可以上载文件,流或目录.

显然,Low-level API更复杂,但更灵活 – 您可以启动上传,之后您会在循环中上传文件的下一部分.文档中的示例代码：

var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1);

// List to store upload part responses.
var uploadResponses = new List<UploadPartResponse>();

// 1. Initialize.
var initiateRequest = new InitiateMultipartUploadRequest
    {
        BucketName = existingBucketName,Key = keyName
    };

var initResponse = s3Client.InitiateMultipartUpload(initRequest);

// 2. Upload Parts.
var contentLength = new FileInfo(filePath).Length;
var partSize = 5242880; // 5 MB

try
{
    long filePosition = 0;
    for (var i = 1; filePosition < contentLength; ++i)
    {
        // Create request to upload a part.
        var uploadRequest = new UploadPartRequest
            {
                BucketName = existingBucketName,Key = keyName,UploadId = initResponse.UploadId,PartNumber = i,PartSize = partSize,FilePosition = filePosition,FilePath = filePath
            };

       // Upload part and add response to our list.
       uploadResponses.Add(s3Client.UploadPart(uploadRequest));

       filePosition += partSize;
   }

   // Step 3: complete.
   var completeRequest = new CompleteMultipartUploadRequest
       {
           BucketName = existingBucketName,};

   // add ETags for uploaded files
   completeRequest.AddPartETags(uploadResponses);

   var completeUploadResponse = s3Client.CompleteMultipartUpload(completeRequest);     
}
catch (Exception exception)
{
    Console.WriteLine("Exception occurred: {0}",exception.ToString());
    var abortmpuRequest = new AbortMultipartUploadRequest
        {
            BucketName = existingBucketName,UploadId = initResponse.UploadId
        };
    s3Client.AbortMultipartUpload(abortmpuRequest);
}

Asynchronous version of UploadPart也可用,因此如果您需要完全控制上传,则应调查该路径.