Upload/Download file in Database


#1

Hello,

I am new to Adonisjs (and NodeJs), which I really like, I have a PHP background.

I am developing a system which holds scenarios into a MySQL database. Each scenario can have attachments which are stored into a db table (file content into a blob), not on the filesystem.

I have managed to upload attachments through the following:

     const scenario = new Scenario()
     scenario.title = request.input('title')
     scenario.description = request.input('description')
     scenario.institution_id = request.input('institution_id')
     await scenario.save()

     //save the file attachments
    const scenarioFiles = request.file('scenario_files')
    const scenarioFilesarray = scenarioFiles.all()
    scenarioFilesarray.forEach(async function (scenariofile, index) {
    const filestore = new Filestore()
    filestore.name  = scenariofile.clientName
    filestore.filecontents = await Drive.get(scenariofile.tmpPath)
    filestore.size = scenariofile.size
    filestore.type = scenariofile.type
    filestore.scenario_id = scenario.id
    await filestore.save()
  1. I am not sure this is the best way but the information is there, although I am not sure that the file content is correct in the blob.

  2. I am trying to let users download the attachments when viewing scenarios with attachments
    That means:
    a) displaying link on the page with attachment name (have done that)
    b) let the browser diplay/download the file when the user clicks on the link

  3. I haven’t managed that. Any ideas?
    I was trying the following:

    async download({ params, response }) {
    const filestore = await Filestore.find(params.id)

    response.header('Content-disposition', 'attachment')
    response.header('content-type', filestore.type)
    response.header('content-length', filestore.size)
    response.attachment(filestore.name)
    response.send(params.filecontents)

but I get - Corrupted Content Error

Not sure if that is because the upload is not correct or the download doen’t work.

Thanks


#2

Since you are not uploading files to the file-system, there is no point is processing a single file for multiple times.

For example, the bodyparser will move the file to the tmp directory and then you will read the file from the tmp directory into a Buffer and finally moving it to the database.

AdonisJs let you optionally disable the bodyparser automatic processing, so that you can read the file stream manually, which means stream is only read by your code and never written to the tmp directory.

Let’s see how it looks.

First disable autoProcessing of files inside config/bodyParser.js.

For all file uploads

autoProcess: false

For single route

autoProcess: true,
processManually: ['your-route']

Next, access the multipart object on the request object as follows.

const getStream = use('get-stream')

async uploadFile ({ request }) {
  const scenarioFiles = []

  request.multipart.file('scenario_files[]', {}, async function (file) {
  	const fileContent = await getStream.buffer(file.stream)

  	scenarioFiles.push({
  	  filecontents: fileContent,
  	  name: file.clientName,
  	  type: `${file.type}/${this.subtype}`
  	})
  })

  await request.multipart.process()

  // now all files have been processed
  await use('Database').table('files').insert(scenarioFiles)
}

In order to download the file, you just need to read the file from the DB and set right headers.

const file = await Database.table('files').where('id', whatEverId).first()
response.header('content-type', file.tyle)
response.header('content-length', Buffer.byteLength(file.contents))
response.send(file.contents)

NOTE

The field name scenario_files[] passed to request.multipart.file has to be exactly the same as the HTML form submits it. Since you are dealing with the raw HTTP request, AdonisJs does not normalize field names.


File upload (s3) with text values
#3

Hello Virk,

thanks for your quick answer.
I tried the upload as you have discribed it but I get an exception:
EBADCSRFTOKEN: Invalid CSRF token

If I change the autoProcess to true, I do not get the exception but no upload.

Any ideas?


#4

Yes CSRF has to be disabled for that route, since bodyparser is not parsing the request body, and instead you are doing it inside your controller now


#5

Can I disable the CSRF protection only for that route (only when storing the attachments)?


#6

Yup check the config file


#7

I haven’t seen how to disable only one route from CSRF in the shield.js file, so I have disabled CSRF completely. I don’t know where/how to disable CSRF for a single route.

Uploading the file resulted to the other parameters of the form not being saved in the database:
const scenario = new Scenario()
scenario.title = request.input(‘title’)
scenario.description = request.input(‘description’)
scenario.institution_id = request.input(‘institution_id’)
await scenario.save()

    const scenarioFiles = []

    request.multipart.file('scenario_files[]', {}, async function (file) {
        const fileContent = await getStream.buffer(file.stream)

        scenarioFiles.push({
            filecontents: fileContent,
            name: file.clientName,
            size: file.size,
            type: '${file.type}/${this.subtype}',
            scenario_id: scenario.id
        }) 
    })

    await request.multipart.process()

    // now all files have been processed
    await Database.table('filestores').insert(scenarioFiles)

ALL the scenario info is NULL.

Downloading the file results in ‘File not found’ although the ID passed is correct and the table is filled:
async download({ params, response }) {

    const file = await Database.table('filestores').where('id', params.id).first()
    response.header('content-type', file.type)
    response.header('content-length', Buffer.byteLength(file.contents))
    response.send(file.contents)

File upload (s3) with text values
#8

Here it is https://github.com/adonisjs/adonis-fullstack-app/blob/develop/config/shield.js#L137


#9

Hi @virk and @spyrosaba

Did you figure out a way to solve this?

Uploading the file resulted to the other parameters of the form not being saved in the database:
const scenario = new Scenario()
scenario.title = request.input(‘title’)
scenario.description = request.input(‘description’)
scenario.institution_id = request.input(‘institution_id’)
await scenario.save()


#10

First disable autoProcessing of files inside config/bodyParser.js .

autoProcess: false,
processManually: ['your-route-upload-in:start/routes.js','your-route-donwload-in:start/routes.js']

In Controller

const mongodb = require("mongodb")
const Database = use("Database")

class UploadController {
  
  // sample route post: /upload
  async dbStore({ request, response }) {
    //https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93

    //important
    response.implicitEnd = false
    
    let responseData = {
      success: false
    }

    const mongoClient = await Database.connect()

   //read aditional params in body
    let body = {}
    request.multipart.field((name, value) => {
      body[name] = value
    })

    request.multipart.file("htmlInputFileName", {}, async file => {

       //if validate here...

       //customize filename
      var newFileName = "customize your filename.jpeg"

      let bucket = new mongodb.GridFSBucket(mongoClient, {
        bucketName: "images"
      })
      
      const optionFile = {
        metadata: {
          uid: "userid",
          name: file.clientName
        },
        contentType: file.headers["content-type"]
      }

      let uploadStream = bucket.openUploadStream(newFileName, optionFile)
      file.stream.pipe(uploadStream)

      uploadStream.on("error", () => {
        responseData.message = "error descriptions"
        return res.status(500).send(responseData)
      })

      uploadStream.on("finish", (fileInfo) => {
        responseData.success = true
        responseData.message = ""
        responseData.id = uploadStream.id.toString()
        return response.status(201).send(responseData)
      })
    })

    await request.multipart.process()
  }

 //route  get: /download?oid=5ba054955fe0bd50837dc98c
  async dbDownload({ request, response, params }) {
   //important
    response.implicitEnd = false

    try {
      var uriParams = request.get()
      var fileId = uriParams.oid
      var fileID = new mongodb.ObjectID(fileId)
    } catch (err) {
      return response.status(400).json({
        message:
          "Invalid oid in URL parameter. "
      })
    }

    const mongoClient = await Database.connect()
    const fileMetadata = await mongoClient.collection("images.files").findOne({ _id: fileID }) 
    let bucket = new mongodb.GridFSBucket(mongoClient, {
      bucketName: "images"
    })

    
      response.header("content-type", fileMetadata.contentType)
    

    let downloadStream = bucket.openDownloadStream(trackID)
    downloadStream.on("data", chunk => {
      response.send(chunk)
    })

    downloadStream.on("error", () => {
      response.sendStatus(404)
    })

    downloadStream.on("end", () => {
      response.end()
    })
  }
}
module.exports = UploadController

References:
https://nodejs.org/api/stream.html#stream_class_stream_passthrough

https://docs.mongodb.com/manual/core/gridfs/



#11

Thank you for the information.