I need to read data from hundreds of PDF forms. These forms have all text entry boxes, the forms are not editable. I have been trying to use Python and PyPDF2 to read these forms to a CSV file (since the ultimate goal is an excel database.
I have tried using acrobats export as csv function, but this is extremely slow as each form has 4 embedded images that export as plaintext. I have the following code,
from PyPDF2 import PdfFileReader
infile = "FormSample.pdf"
pdf_reader = PdfFileReader(open(infile, "rb"))
with open('exportharvest.csv','w') as exportharvestcsv:
dictionary = pdf_reader.getFields(fileobj = exportharvestcsv)
textfields = pdf_reader.getFormTextFields()
dest = pdf_reader.getNamedDestinations()
print(dest)
The issue with the above code is as follows: the getFields command only gets the ~4 digital signature fields in the form (form has ~300 entries). Is there some way to instruct python to look through all the fields? I know the field names in the document as they are listed when I export to pdf.
getFormTextFields() returns a dictionary of {}
getNamedDestinations() returns a dictionary of {}
Thanks for any help.
From my experience pyPDF is slow as well.
this here should do what you want:
from PyPDF2 import PdfFileReader
from pprint import pprint
pdf_file_name = 'formdocument.pdf'
f = PdfFileReader(pdf_file_name)
fields = f.getFields()
fdfinfo = dict((k, v.get('/V', '')) for k, v in fields.items())
pprint(fdfinfo)
with open('test.csv', 'w') as f2:
for key in fdfinfo.keys():
if type(key)==type("string") and type(str(fdfinfo[key]))==type("string"):
f2.write('"'+key+'","'+fdfinfo[key]+'"\n')
Related
My project involves reading text from a bunch of PDF form files for which I'm using PyPDF2 open source library. There is no issue in getting the text data as follows:
reader = PdfReader("data/test.pdf")
cnt = len(reader.pages)
print("reading pdf (%d pages)" % cnt)
page = reader.pages[cnt-1]
lines = page.extract_text().splitlines()
print("%d lines extracted..." % len(lines))
However, this text doesn't contain the checked statuses of the radio and checkboxes. I just get normal text (like Yes No or Check-1 Check-2 Check-3 for example) instead of these values.
I also tried the reader.get_fields() and reader.get_form_text_fields() methods as described in their documentation but they return empty values. I also tried reading it through annotations but no "/Annots" found on the page. When I open the PDF in a notepad++ to see its meta data, this is what I get:
%PDF-1.4
%²³´µ
%Generated by ExpertPdf v9.2.2
It appears to me that these checkboxes aren't usual form fields used in PDF but appear similar to HTML elements. Is there any way to extract these fields using python?
Finally, I've also tried pdfminer.six, the other popular pdf library for python with same results.
It seems like the PDF you have contains XFA forms, which are not supported by the pdfminer.six library.
You may try this tool to extract the XFA forms.
If your PDF does not contain XFA forms, you can extract the form field and value data using the following snippet from pdfminer.six docs
from pdfminer.pdfparser import PDFParser
from pdfminer.pdfdocument import PDFDocument
from pdfminer.pdftypes import resolve1
from pdfminer.psparser import PSLiteral, PSKeyword
from pdfminer.utils import decode_text
data = {}
def decode_value(value):
# decode PSLiteral, PSKeyword
if isinstance(value, (PSLiteral, PSKeyword)):
value = value.name
# decode bytes
if isinstance(value, bytes):
value = decode_text(value)
return value
with open(file_path, 'rb') as fp:
parser = PDFParser(fp)
doc = PDFDocument(parser)
res = resolve1(doc.catalog)
if 'AcroForm' not in res:
raise ValueError("No AcroForm Found")
fields = resolve1(doc.catalog['AcroForm'])['Fields'] # may need further resolving
for f in fields:
field = resolve1(f)
name, values = field.get('T'), field.get('V')
# decode name
name = decode_text(name)
# resolve indirect obj
values = resolve1(values)
# decode value(s)
if isinstance(values, list):
values = [decode_value(v) for v in values]
else:
values = decode_value(values)
data.update({name: values})
print(name, values)
If the above code throws No AcroForm found error, that means your PDF most probably contains XFA forms.
At the moment I use a script to populate a template for each of the entries in our database and generate a docx file for each entry. Following that I convert that docx file to a pdf and mail it to the user.
For this I use following code :
from docxtpl import DocxTemplate
from docx2pdf import convert
pathToTemplate='template.docx'
outputPath='output.docx'
template = DocxTemplate(pathToTemplate)
context = person.get_context(short) # gets the context used to render the document
template.render(context)
template.save(outputPath)
pdfpath = outputPath[:-4]+'pdf'
convert(outputPath, pdfpath)
This part of the code is embedded in a loop and when measuring the time needed to generate the context from the database (in the person.get_context(short) function) and generating the docx file it gives me a result between 0.5s - 1.0s. When measuring the time needed to convert this docx to pdf it gives me a time of 5.0s - 7.0s.
Because the loop has to loop over > 1000 users, this is the difference can add up. Does anyone have an idea how the DocxTemplate kan save to pdf directly (and how fast this is) or if there is a faster way to generate the pdf files?
as far as I know you just can't do it with the docx library itself, but I have found an alternate way to achieve this, we can convert the docx to pdf using the following code
from docxtpl import DocxTemplate
import pandas as pd
df = pd.read_excel("Data.xlsx")
import time
import os
from win32com import client
word_app = client.Dispatch("Word.Application")
for i , j in df.iterrows():
Name = j["Party_Name"]
tpl = DocxTemplate("Invoice_Template.docx")
dicty = df.to_dict()
x = df.to_dict(orient="records")
context = x
tpl.render(context[i])
tpl.save("hello.docx")
rod = os.path.dirname(os.path.abspath(__file__))
print(rod)
time.sleep(2)
#converting to pdf
doc = word_app.Documents.Open(rod + "\\1.docx")
doc.SaveAs(rod + "\\hello.pdf", FileFormat=17)
I am trying to use this class to append a page from one PDF to another by specifying the page position.
Does anyone have experience with this? I couldn't find any example of using PdfFileMerger.merge over internet
with open(orig_pdf, 'rb') as orig, open(amend_pdf, 'rb') as new:
pdf = PdfFileMerger()
pdf.merge(2, new)
pdf.write('.pdf')
Consider using merge and passing the position, which is the page number you wish to add the pdf file
There are (probably) many ways of achieving the same results. Heres a basic working example:
from PyPDF2 import PdfFileMerger, PdfFileReader
orig_pdf = r'C:\temp\old.pdf'
amend_pdf = r'C:\temp\new.pdf'
with open(orig_pdf, 'rb') as orig, open(amend_pdf, 'rb') as new:
merger = PdfFileMerger()
merger.append(PdfFileReader(orig_pdf))
# Add amend_pdf after page 2
merger.merge(2, PdfFileReader(amend_pdf))
merger.write("results.pdf")
For more info, have a look at the official documentation https://pythonhosted.org/PyPDF2/PdfFileMerger.html
When I print a PDF from any of my source PDFs, the file size drops and removes the text boxes presents in form. In short, it flattens the file.
This is behavior I want to achieve.
The following code to create a PDF using another PDF as a source (the one I want to flatten), it writes the text boxes form as well.
Can I get a PDF without the text boxes, flatten it? Just like Adobe does when I print a PDF as a PDF.
My other code looks something like this minus some things:
import os
import StringIO
from pyPdf import PdfFileWriter, PdfFileReader
from reportlab.pdfgen import canvas
from reportlab.lib.pagesizes import letter
directory = os.path.join(os.getcwd(), "source") # dir we are interested in
fif = [f for f in os.listdir(directory) if f[-3:] == 'pdf'] # get the PDFs
for i in fif:
packet = StringIO.StringIO()
can = canvas.Canvas(packet, pagesize=letter)
can.rotate(-90)
can.save()
packet.seek(0)
new_pdf = PdfFileReader(packet)
fname = os.path.join('source', i)
existing_pdf = PdfFileReader(file(fname, "rb"))
output = PdfFileWriter()
nump = existing_pdf.getNumPages()
page = existing_pdf.getPage(0)
for l in range(nump):
output.addPage(existing_pdf.getPage(l))
page.mergePage(new_pdf.getPage(0))
outputStream = file("out-"+i, "wb")
output.write(outputStream)
outputStream.close()
print fName + " written as", i
Summing up: I have a pdf, I add a text box to it, covering up info and adding new info, and then I print a pdf from that pdf. The text box becomes not editable or moveable any longer. I wanted to automate that process but everything I tried still allowed that text box to be editable.
If installing an OS package is an option, then you could use pdftk with its python wrapper pypdftk like this:
import pypdftk
pypdftk.fill_form('filled.pdf', out_file='flattened.pdf', flatten=True)
You would also need to install the pdftk package, which on Ubuntu could be done like this:
sudo apt-get install pdftk
The pypdftk library can by downloaded from PyPI:
pip install pypdftk
Update: pdftk was briefly removed from Ubuntu in version 18.04, but it seems it is back since 20.04.
A simple but more of a round about way it to covert the pdf to images than to put those image into a pdf.
You'll need pdf2image and PIL
Like So
from pdf2image import convert_from_path
from PIL import Image
images = convert_from_path('temp.pdf')
im1 = images[0]
images.pop(0)
pdf1_filename = "flattened.pdf"
im1.save(pdf1_filename, "PDF" ,resolution=100.0, save_all=True, append_images=images)
Edit:
I created a library to do this called fillpdf
pip install fillpdf
from fillpdf import fillpdfs
fillpdfs.flatten_pdf('input.pdf', 'newflat.pdf')
Per the Adobe Docs, you can change the Bit Position of the Editable Form Fields to 1 to make the field ReadOnly. I provided a full solution here, but it uses Django:
https://stackoverflow.com/a/55301804/8382028
Adobe Docs (page 552):
https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdfs/pdf_reference_archives/PDFReference.pdf
Use PyPDF2 to fill the fields, then loop through the annotations to change the bit position:
from io import BytesIO
from PyPDF2 import PdfFileReader, PdfFileWriter
from PyPDF2.generic import BooleanObject, NameObject, NumberObject
# open the pdf
input_stream = open("YourPDF.pdf", "rb")
reader = PdfFileReader(input_stream, strict=False)
if "/AcroForm" in reader.trailer["/Root"]:
reader.trailer["/Root"]["/AcroForm"].update(
{NameObject("/NeedAppearances"): BooleanObject(True)}
)
writer = PdfFileWriter()
writer.set_need_appearances_writer()
if "/AcroForm" in writer._root_object:
# Acro form is form field, set needs appearances to fix printing issues
writer._root_object["/AcroForm"].update(
{NameObject("/NeedAppearances"): BooleanObject(True)}
)
data_dict = dict() # this is a dict of your form values
writer.addPage(reader.getPage(0))
page = writer.getPage(0)
# update form fields
writer.updatePageFormFieldValues(page, data_dict)
for j in range(0, len(page["/Annots"])):
writer_annot = page["/Annots"][j].getObject()
for field in data_dict:
if writer_annot.get("/T") == field:
# make ReadOnly:
writer_annot.update({NameObject("/Ff"): NumberObject(1)})
output_stream = BytesIO()
writer.write(output_stream)
# output_stream is your flattened PDF
A solution that goes for Windows as well, converts many pdf pages and flatens the chackbox values as well. For some reason #ViaTech code did not work in my pc (Windows7 python 3.8)
Followed #ViaTech indications and used extensively #hchillon code from this post
from PyPDF2 import PdfFileReader, PdfFileWriter
from PyPDF2.generic import BooleanObject, NameObject, IndirectObject, TextStringObject, NumberObject
def set_need_appearances_writer(writer):
try:
catalog = writer._root_object
# get the AcroForm tree and add "/NeedAppearances attribute
if "/AcroForm" not in catalog:
writer._root_object.update({
NameObject("/AcroForm"): IndirectObject(len(writer._objects), 0, writer)})
need_appearances = NameObject("/NeedAppearances")
writer._root_object["/AcroForm"][need_appearances] = BooleanObject(True)
return writer
except Exception as e:
print('set_need_appearances_writer() catch : ', repr(e))
return writer
class PdfFileFiller(object):
def __init__(self, infile):
self.pdf = PdfFileReader(open(infile, "rb"), strict=False)
if "/AcroForm" in self.pdf.trailer["/Root"]:
self.pdf.trailer["/Root"]["/AcroForm"].update(
{NameObject("/NeedAppearances"): BooleanObject(True)})
# newvals and newchecks have keys have to be filled. '' is not accepted
def update_form_values(self, outfile, newvals=None, newchecks=None):
self.pdf2 = MyPdfFileWriter()
trailer = self.pdf.trailer['/Root'].get('/AcroForm', None)
if trailer:
self.pdf2._root_object.update({
NameObject('/AcroForm'): trailer})
set_need_appearances_writer(self.pdf2)
if "/AcroForm" in self.pdf2._root_object:
self.pdf2._root_object["/AcroForm"].update(
{NameObject("/NeedAppearances"): BooleanObject(True)})
for i in range(self.pdf.getNumPages()):
self.pdf2.addPage(self.pdf.getPage(i))
self.pdf2.updatePageFormFieldValues(self.pdf2.getPage(i), newvals)
for j in range(0, len(self.pdf.getPage(i)['/Annots'])):
writer_annot = self.pdf.getPage(i)['/Annots'][j].getObject()
for field in newvals:
writer_annot.update({NameObject("/Ff"): NumberObject(1)})
self.pdf2.updatePageFormCheckboxValues(self.pdf2.getPage(i), newchecks)
with open(outfile, 'wb') as out:
self.pdf2.write(out)
class MyPdfFileWriter(PdfFileWriter):
def __init__(self):
super().__init__()
def updatePageFormCheckboxValues(self, page, fields):
for j in range(0, len(page['/Annots'])):
writer_annot = page['/Annots'][j].getObject()
for field in fields:
writer_annot.update({NameObject("/Ff"): NumberObject(1)})
origin = ## Put input pdf path here
destination = ## Put output pdf path here, even if the file does not exist yet
newchecks = {} # A dict with all checkbox values that need to be changed
newvals = {'':''} # A dict with all entry values that need to be changed
# newvals dict has to be equal to {'':''} in case that no changes are needed
c = PdfFileFiller(origin)
c.update_form_values(outfile=destination, newvals=newvals, newchecks=newchecks)
print('PDF has been created\n')
I had trouble flattening a form that I had entered content into using pdfrw (How to Populate Fillable PDF's with Python) and found that I had to add an additional step using generate_fdf (pdftk flatten loses fillable field data).
os.system('pdftk '+outtemp+' generate_fdf output '+outfdf)
os.system('pdftk '+outtemp+' fill_form '+outfdf+' output '+outpdf)
I came to this solution because I was able to flatten a file just fine using ghostscript's pdf2ps followed by ps2pdf on my Mac, but the quality had low resolution when I ran it on an Amazon Linux instance. I couldn't figure out why that was the case and so moved to the pdftk solution.
Is anybody has experience merging two page of PDF file into one using python lib PyPDF2.
When I try page1.mergePage(page2) it results with page2 overlayed page1. How to make it to add page2 to the bottom of the page1?
As I'm searching the web for python pdf merging solution, I noticed that there's a general misconception with merging versus appending.
Most people call the appending action a merge but it's not. What you're describing in your question is really the intended use of mergePage which should be called applyPageOnTopOfAnother but that's a little long. What you are (were) looking for is really appending two files/pages into a new file.
Appending PDF files
Using the PdfFileMerger class and its append method.
Identical to the merge() method, but assumes you want to concatenate
all pages onto the end of the file instead of specifying a position.
Here's one way to do it taken from pypdf Merging multiple pdf files into one pdf:
from PyPDF2 import PdfFileMerger, PdfFileReader
# ...
merger = PdfFileMerger()
merger.append(PdfFileReader(file(filename1, 'rb')))
merger.append(PdfFileReader(file(filename2, 'rb')))
merger.write("document-output.pdf")
Appending specific PDF pages
And to append specific pages of different PDF files, use the PdfFileWriter class with the addPage method.
Adds a page to this PDF file. The page is usually acquired from a
PdfFileReader instance.
file1 = PdfFileReader(file(filename1, "rb"))
file2 = PdfFileReader(file(filename2, "rb"))
output = PdfFileWriter()
output.addPage(file1.getPage(specificPageIndex))
output.addPage(file2.getPage(specificPageIndex))
outputStream = file("document-output.pdf", "wb")
output.write(outputStream)
outputStream.close()
Merging two pages into one page
Using mergePage
Merges the content streams of two pages into one. Resource references
(i.e. fonts) are maintained from both pages. The mediabox/cropbox/etc
of this page are not altered. The parameter page’s content stream will
be added to the end of this page’s content stream, meaning that it
will be drawn after, or “on top” of this page.
file1 = PdfFileReader(file(filename1, "rb"))
file2 = PdfFileReader(file(filename2, "rb"))
output = PdfFileWriter()
page = file1.getPage(specificPageIndex)
page.mergePage(file2.getPage(specificPageIndex))
output.addPage(page)
outputStream = file("document-output.pdf", "wb")
output.write(outputStream)
outputStream.close()
If the 2 PDFs do not exist on your local machine, and instead are normally accessed/download via a URL (i.e. http://foo/bar.pdf & http://bar/foo.pdf), we can fetch both PDFs from remote locations and merge them together in memory in one-fell-swoop.
This eliminates the assumed step of downloading the PDF to begin with, and allows us to generalize beyond the simple case of both PDFs existing on disk. Specifically, it generalizes the solution to any HTTP-accessible PDF.
The example:
from PyPDF2 import PdfFileMerger, PdfFileReader
pdf_content_1 = requests.get('http://foo/bar.pdf').content
pdf_content_2 = requests.get('http://bar/foo.pdf').content
# Write to in-memory file-like buffers
pdf_buffer_1 = StringIO.StringIO().write(pdf_content_1)
pdf_buffer_2 = StringIO.StringIO().write(pdf_content_2)
pdf_merged_buffer = StringIO.StringIO()
merger = PdfFileMerger()
merger.append(PdfFileReader(pdf_buffer_1))
merger.append(PdfFileReader(pdf_buffer_2))
merger.write(pdf_merged_buffer)
# Option 1:
# Return the content of the buffer in an HTTP response (Flask example below)
response = make_response(pdf_merged_buffer.getvalue())
# Set headers so web-browser knows to render results as PDF
response.headers['Content-Type'] = 'application/pdf'
response.headers['Content-Disposition'] = \
'attachment; filename=%s.pdf' % 'Merged PDF'
return response
# Option 2: Write to disk
with open("merged_pdf.pdf", "w") as fp:
fp.write(pdf_merged_buffer.getvalue())
Did it this way:
reader = PyPDF2.PdfFileReader(open("input.pdf",'rb'))
NUM_OF_PAGES = reader.getNumPages()
page0 = reader.getPage(0)
h = page0.mediaBox.getHeight()
w = page0.mediaBox.getWidth()
newpdf_page = PyPDF2.pdf.PageObject.createBlankPage(None, w, h*NUM_OF_PAGES)
for i in range(NUM_OF_PAGES):
next_page = reader.getPage(i)
newpdf_page.mergeScaledTranslatedPage(next_page, 1, 0, h*(NUM_OF_PAGES-i-1))
writer = PdfFileWriter()
writer.addPage(newpdf_page)
with open('output.pdf', 'wb') as f:
writer.write(f)
It works when every page has the same height and width. Otherwise, it needs some modifications.
Maybe Emile Bergeron solution is better. Didn't try it.
The pdfrw library can do this. There is a 4up example in the examples directory that places 4 input pages on every output page, and a booklet example that takes 8.5x11 input and creates 11x17 output. Disclaimer -- I am the pdfrw author.
The code posted in this following link accomplished your objective.
Using PyPDF2 to merge files into multiple output files
I believe the trick is:
merger.append(input)