Post a file stream from S3 to an API with Node

Ever needed to send a file that lives on S3 to an API endpoint? You could download the file and then send it, but that requires multiple steps and involves saving the file to the local file system. A better approach is to use streams, where you can pipe the data from S3 directly to the API.

First, install @aws-sdk/client-s3 and create an S3 client:

npm i @aws-sdk/client-s3

const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");

const s3 = new S3Client({
  credentials: {
    accessKeyId: "accessKeyId",
    secretAccessKey: "secretAccessKey"
  },
  region: "region"
});

Now use the GetObjectCommand to retrieve a handle to the object stream:

const bucket = "mybucket";
const key = "path/to/somefile.zip";

const response = await s3.send(new GetObjectCommand({ 
  Bucket: bucket, 
  Key: key 
}));

The command’s response Body will be a stream:

const stream = response.Body;

Next, create a FormData object to attach the stream as a file:

const form = new FormData();
form.append("file", stream);

Then post the form data. Note that I am using Axios here, but it should work with other libraries although there are apparently issues with native fetch so beware if using fetch.

const result = await axios.post("https://someapi.com/upload", form, {
  headers: form.getHeaders()
});

And that’s all it takes.

There is however one gotcha that I ran into which caused some probems on the receiving API – the AWS SDK adds the x-id=GetObject query parameter to the file name. Depending on how your API handles the file it may or may not be a problem. A simple workaround I found that works is to modify the request path of the stream before making the post request to remove the query params from the name:

// Strip any query params that the AWS SDK adds to the file name 
if (stream.req.path.indexOf("?") > -1) {
  stream.req.path = stream.req.path.split("?")[0];
}

Now the file name on the receiving API will not have any query parameters.

Here is the full code:

const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");

const s3 = new S3Client({
  credentials: {
    accessKeyId: "accessKeyId",
    secretAccessKey: "secretAccessKey"
  },
  region: "region"
});

const bucket = "mybucket";
const key = "path/to/somefile.zip";

const response = await s3.send(new GetObjectCommand({ 
  Bucket: bucket, 
  Key: key 
}));

const stream = response.Body;

// Strip any query params that the AWS SDK adds to the file name 
if (stream.req.path.indexOf("?") > -1) {
  stream.req.path = stream.req.path.split("?")[0];
}

const form = new FormData();
form.append("file", stream);

const result = await axios.post("https://someapi.com/upload", form, {
  headers: form.getHeaders()
});