Uploading multiple pdfs to Flask ( Sql alchmey) with additional variables

[KirkmanJ]

i have tried to explain this in a helpful clean way but it is quite long, so get snacks before starting ( id recommend jelly babies for that energy burst), ive tried to break the problem down into: desired outcome, what i have, where i think the solution lies, what i need help with.

My language: Before i start when i mention the variable 'doctype' i mean a category in which you might class a document by, e.g. user manual.
When i mention filename/filetype i mean the type of file it is e.g. '.pdf'. or the filename: 'python.pdf.'
Just dodging that question before it is asked :P.

Desired outcome:

Files are to be uploaded to a sqlalchemy database table. What I would like is for a user to be able to upload multiple files at 'the same time,'(loosely as i dont think the same time is strictly possible, more like one at a time but selecting multiple from the file input.) Linking variables, product_serial_number and doctype to the file in columns on the same row as the file.

So sql table would look something like:

filename data product_serial_number doctype


What i have so far:

Currently i have the html which will all multiple file selection:

<form method=POST enctype=multipart/form-data action='/upload_pdf'>
   <input type='file' name='inputFile' multiple="multiple" class='mp_FileInput'></input>
   <button class='pdf01_upload_button' type='submit' value='Upload' ><b>Upload</b></button>
</form>

html which uses JSON to allow a user to select a doctype and serial number:

<p> serial_number: <input type='text' id='serial_number_entry' name='serial_number_entry'></p>
<select type='text' name='doctype_entry' id='doctype_entry' placeholder='Select a Doc Type...'>
    <option value='' disabled selected>Select a Doc Type...</option>
    <option>Build Log</option>
    <option>Calibration Certificate</option>
    <option>Factory Acceptance Test</option>
    <option>Performance Verification Certificate</option>
    <option>Procedure Checklist</option>
    <option>System Configuration Tracking</option>
    <option>Test Certificate</option>
</select>
<button id="add-pdf">Upload PDF</button>
<p id='result'> ... </p>

<script>
  $(function() {
    $('#add-pdf').on('click', function() {
      $.getJSON('{{ url_for('test_domain.test_process') }}',{
        serial_number: $('input[name="serial_number_entry"]').val(),
        doctype: $('select[name="doctype_entry"]').val(),
      }, function(data) {
        $("#result").text(data.result);
      });
      return false;
    });
  });
</script>

And the python for a single pdf upload ( ive striped down to the bare bones so if there is a random 'else' or 'if' then its not a codding mistake, just me cutting things out.):

def upload_file_pdf():
    product_serial_number_store = request.form['product_serial_number']
    Doc_Type_Store = request.form['Doc_Type_Input']

    if Tempfile and allowed_file(Tempfile.filename):
      filename = secure_filename(Tempfile.filename)
      newFile = pdf_test_equipment_store(related_to=related_to, name=Tempfile.filename,  product_serial_number= product_serial_number_store,doctype=Doc_Type_Store, date_time=datetime.utcnow(), data=Tempfile.read(), place_of_procedure=Tester.location, uploaded_by=Tester.username)
      db.session.add(newFile)
      db.session.commit()
      flash(' Saved ' + Doc_Type_Store + ' for ' + product_serial_number_store + ' to the test equipment pdf store!')
      return render_template('upload_pdf.html', title='upload_pdf', filename=filename)
               

And i can combine both html documents in 1 form, lose the javascript and use the python in order to allow for 1 a single pdf to be uploaded.

What needs to happen:
User clicks upload button and selects ( lets say 10) pdfs to upload.
I would like my website to select the first pdf of the list, ask the user to input a doctype and product_serial_number , then query the information and upload to table. Then select pdf 2 from the list and ask the user to input doctype and product_serial_number. And repeat.

What i need help with.

So i can do the script for 1 pdf. and i assume by using the JSON and the use of a prompt box i can loop the script and just get the next file in the list. I need help implementing my thoughts though, or being corrected if my theory is incorrect.

To anyone that is still reading, thanks for sticking with me and hope i have explained in enough detail that you might be able to help me.

---

I have just put together a script ( shown below) which uploads multiple files to a database, so now I only need help with linking the the doctype and product_serial_number to each pdf.

if request.method == 'POST':
   uploaded_files =request.files.getlist("file[]")
   print uploaded_files
   for file in uploaded_files:
       filename = secure_filename(file.filename)
       newFile = pdf_test_equipment_store(name=file.filename, date_time=datetime.utcnow(), data=file.read(), place_of_procedure=Tester.location, uploaded_by=Tester.username)
       db.session.add(newFile)
       db.session.commit()

   flash(' Saved to the pdf store!')
   return render_template('upload_pdf.html', title='upload_pdf', filename=filename)
else:
   return render_template('upload_pdf.html', title='upload_pdf')

[gontajones]

What about after the multiple selection of files you redirect user to a new form with each file name and text fields for doctype and serial number?

Another approach could be spamming multiple modals but I think this way will be annoying for the user.

[KirkmanJ]

Thats not a bad suggestion, had not thought about that approach.
So what im now thinking i could do is have a select button that allows a user to select a bunch of pdfs.
Then on 'submit' it used javascript to display a form bellow which included a table with serial_number and doctype inputs.

I can 'probably' get most of that working :) however what would be the easiest way of ensuring the pdf data was transferred correctly between the first form, second form and 'upload ( completion)'. Would I have to upload the pdfs to the table then require them ?

[gontajones]

I think you can store the pdf files info in a temp list after the submit of the first form then after the second form is submitted with valid info, you write everything in DB.

[KirkmanJ]

Hello again, continuing with this project, i 'think' i have almost cracked how to complete this task. However i can not figure out how to process the pdfs when i need to call them again. I am pretty sure the bellow code wont loop properly but im hoping its at least a start.
Current issue is 'calling' the pdf list back in a way which makes them usable.
Stores pdf as temp list and pulls html asking for variable input:

@upload_pdf_blueprint.route('/pdf_information', methods=['GET', 'POST'])

def pdf_information():
    ALLOWED_EXTENSIONS = set(['pdf'])
    ALERT_EXTENSIONS = set(['py', 'html', 'php'])
    uploaded_files =request.files.getlist("file[]")
    filenames = []

    for file in uploaded_files:
        if allowed_file(file.filename):
            temp_filename = secure_filename(file.filename)
            filenames.append(temp_filename)
        else:
            flash('Please ensure all files selected are pdfs')
            return render_template('upload_pdf.html')
    print(filenames)
    set_pdf_values(uploaded_files)

    return render_template('upload_pdf_information.html', filenames=filenames)

Processes the pdfs and uploads to sqlalchemy table.

@upload_pdf_blueprint.route('/upload_pdf', methods=['GET', 'POST'])

def upload_file_pdf():
    print (pdf_values)
    Tester = user_info.query.filter_by(username=current_user.username).first()
    product_serial_number_store = request.form['product_serial_number']
    Doc_Type_Store = request.form['Doc_Type_Input']
    related_to = request.form['related_to']
    related_to = related_to.lower()
    if request.method == 'POST':
        print(Doc_Type_Store)
        print(product_serial_number_store)
        print(related_to)
        for file in pdf_values:
            print ('HELLOLOOOOO')
            if related_to == 'test equipment':
                filename = secure_filename(file.filename)
                newFile = pdf_test_equipment_store(name=file.filename,related_to = related_to, product_serial_number=product_serial_number_store, doctype=Doc_Type_Store, date_time=datetime.utcnow(), data=file.read(), location_of_procedure=Tester.location, uploaded_by=Tester.username)
                db.session.add(newFile)
                db.session.commit()

            elif related_to == 'product':
                filename = secure_filename(file.filename)
                newFile = pdf_products_store(name=file.filename, related_to = related_to, product_serial_number=product_serial_number_store, doctype=Doc_Type_Store, date_time=datetime.utcnow(), data=file.read(), location_of_procedure=Tester.location, uploaded_by=Tester.username)
                db.session.add(newFile)
                db.session.commit()

            else:
                pass
        flash(' Saved to the pdf store!')
        return render_template('upload_pdf.html')
    else:
        return render_template('upload_pdf.html')

I currently store the pdfs as a single variable and when i try to run the 2nd code i get :

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/jack/1st-dpe-t-tintin/app/upload_pdf/views.py", line 77, in upload_file_pdf
    newFile = pdf_products_store(name=file.filename, related_to = related_to, product_serial_number=product_serial_number_store, doctype=Doc_Type_Store, date_time=datetime.utcnow(), data=file.read(), location_of_procedure=Tester.location, uploaded_by=Tester.username)
  File "/usr/lib/python2.7/tempfile.py", line 603, in read
    return self._file.read(*args)
ValueError: I/O operation on closed file

[gontajones]

What are you doing inside pdf_test_equipment_store() and pdf_products_store()?

According to the error your code is trying to write/read the file.filename, but it is closed.

[KirkmanJ]

Both are supposed to take a document from the list and depending on which option was selected from a dropdown save it in the appropriate table.

Yes i had figured out that the error message had meant it was closed =) . However i could never get my head round 'with open' which is my best guess at a solution.

The exact bit inside pdf_test_equipment_store() is the query used to add the row to the table.

[gontajones]

Are you trying to write the pdf file content inside a field of your DB?
I'm thinking here, maybe your code is trying to find the file in its local path.

---

If you could post the content of your pdf_test_equipment_store() function, it will be better.

[KirkmanJ]

pdf_test_equipmennt_store() is the table name. The line you refer to simply takes the values and populates the row in the table. nothing fancy.

And yes i do want to store the pdf as a BLOB inside a sqlalchemy table

[gontajones]

Check what is coming in data=file.read().
Maybe you will need to store the data of each file inside pdf_information(), like you did with the filenames.

I have this Django project where I use fd to read the file content (data) from a uploaded CSV file.
Just for reference.

class PerformanceView(View):

    def post(self, request, *args, **kwargs):
        if(request.method == "POST" and request.FILES):
            fd = request.FILES['csv_file']

In my template:

<form action="/performance/" method="post" enctype="multipart/form-data">
    { % csrf_token % }
    <a class="btn btn-app btn-file">
        <i class="fa fa-paperclip"> </i> <div id="fileupload_name" > {{filename_upload}} </div>
        <input id="fileupload" type="file" name="csv_file" />
    </a>
    <input class="btn btn-app" type="submit" value="Upload" >
</form>