Streamed file(s) upload - callback not fired


#1

I am attempting to upload one or more files and then stream the uploaded content directly to Kraken.io for optimisation.

Here is my current code:

  async upload ({ request }) {
    try {
      request.multipart.file('files', {}, async (file) => {
        let krakenResponse = await axios.post('https://api.kraken.io/v1/upload', {
          auth: {
            api_key: Env.get('KRAKEN_API_KEY'),
            api_secret: Env.get('KRAKEN_SECRET')
          },
          file: file.stream
        })
        console.log(krakenResponse)
      })
      await request.multipart.process()
      return { status: 'success' }
    } catch (error) {
      console.log(error)
      return {
        status: 'error',
        error: {
          code: error.code,
          message: error.message
        }
      }
    }
  }

The file upload itself appears to be working fine - it correctly shows the full buffer and if I switch this to a regular upload the file is saved under /tmp correctly.

The problem is that the console.log(krakenResponse) does not fire and my dashboard within Kraken.io shows no hits so I can only presume the async (file) => { ... } callback is not being fired. The return { status: 'success' } however does fire, so my frontend shows that the upload was successful when this is not the case - the stream does not go anywhere.

Is there anything I am doing wrong here?


#2

Couple of things here.

  1. Have u turned off autoProcess value inside config/bodyParser.js file, to manually handle the stream?
  2. The best to know whether the callback has been fired or not, is to use console.log over checking the kraken.io dashboard.
  3. Also does kraken API expects the stream to be uploaded or do they expect the real file?

#3
  1. Yes
  2. As per my code above I am already using a console.log(krakenResponse). I have also tried adding a console.log(file) as the very first line in the callback but nothing gets logged. The Kraken.io dashboard was just another check to see if they received anything.
  3. They support either option. I figured sending a stream would be better in this case to avoid saving a temporary copy of the file and wasting server resources.

#4

Also just found that kraken.io only works with fs streams and not generic streams https://github.com/kraken-io/kraken-node/pull/22

So the only option is to stream it from the tmpPath path. Again the problem with tmpPath file is that the file extension is set to .tmp and kraken doesn’t allow specifying a file name.

Finally, the only option is to

  1. Set autoProcess = true
  2. Use the tmpPath to create a fs stream.
  3. Use request module to upload the file. The reason request module is required, so that we can pass a custom file name for the stream, which is not possible with kraken SDK.
const Helpers = use('Helpers')
const fs = use('fs')
const request = use('request')
const postRequest = Helpers.promisify(request.post.bind(request))

Route.post('/', async ({ request }) => {
  // uploaded file
  const file = request.file('file')

  // request file node
  const fileNode = {
    value: fs.createReadStream(file.tmpPath),
    options: {
      filename: file._clientName,
      contentType: file._headers['content-type']
    }
  }

  // other data to be sent with form data
  const data = JSON.stringify({
    wait: true,
    auth: {
      api_key: 'API_KEY',
      api_secret: 'API_SECRET'
    }
  })

  // Making the final request
  try {
    return await postRequest({
      url: 'https://api.kraken.io/v1/upload',
      json: true,
      strictSSL: false,
      formData: {
        file: fileNode,
        data: data
      }
    })
  } catch (error) {
    return error
  }
})

#5

Thank you for the above - very helpful.

I’m not entirely certain in this instance that Kraken is at fault though. If I drop my code to just the following I still do not get anything logged but the success message returns:

request.multipart.file('files', {}, async (file) => {
  console.log('Method firing?')
})

await request.multipart.process()

return { status: 'success' }

#6

Can u share how do u submit the file? CURL request for same will be helpful


#7

For testing purposes I am just using a regular setup:

<form enctype="multipart/form-data" novalidate>
  <input type="file" name="files" accept="image/png, image/jpg, image/jpeg" multiple>
</form>

Once one or more files are selected they are POSTed to an API endpoint, with the POST request looking like this:


#8

For what it’s worth, the initial upload into my app works fine if I turn on autoProcess and do something like:

const uploadedPics = request.file('files', {
  types: ['png', 'jpg', 'jpeg'],
  size: '1mb'
})

await uploadedPics.move(Helpers.tmpPath('uploads'))

if (!uploadedPics.moved()) {
  return uploadedPics.error()
}

response.json({ status: 'success'})

As such I do not believe I am doing anything incorrectly on the client-side.