We Keep Coding

Python Data Extraction from an Encrypted PDF

I am an recent graduate in pure mathematics who only has taken few basic programming courses. I am doing an internship and I have an internal data analysis project. I have to analyze the internal PDFs of the last years. The PDFs are "secured." In other words, they are encrypted. We do not have PDF passwords, even more, we are not sure if passwords exist. But, we have all these documents and we can read them manually. We can print them as well. The goal is to read them with Python because is the language that we have some idea.
First, I tried to read the PDFs with some Python libraries. However, the Python libraries that I found do not read encrypted PDFs. At that time, I could not export the information using Adobe Reader either.
Second, I decided to decrypt the PDFs. I was successful using the Python library pykepdf. Pykepdf works very well! However, the decrypted PDFs cannot be read as well with the Python libraries of the previous point (PyPDF2 and Tabula). At this time, we have made some improvement because using Adobe Reader I can export the information from the decrypted PDFs, but the goal is to do everything with Python.
The code that I am showing works perfectly with unencrypted PDFs, but not with encrypted PDFs. It is not working with the decrypted PDFs that were gotten with pykepdf as well.
I did not write the code. I found it in the documentation of the Python libraries Pykepdf and Tabula. The PyPDF2 solution was written by Al Sweigart in his book, "Automate the Boring Stuff with Python," that I highly recommend. I also checked that the code is working fine, with the limitations that I explained before.
First question,
why I cannot read the decrypted files, if the programs work with files that never have been encrypted?
Second question,
Can we read with Python the decrypted files somehow? Which library can do it or is impossible? Are all decrypted PDFs extractable?
Thank you for your time and help!!!
I found these results using Python 3.7, Windows 10, Jupiter Notebooks, and Anaconda 2019.07.
Python
import pikepdf
with pikepdf.open("encrypted.pdf") as pdf:
num_pages = len(pdf.pages)
del pdf.pages[-1]
pdf.save("decrypted.pdf")
import tabula
tabula.read_pdf("decrypted.pdf", stream=True)
import PyPDF2
pdfFileObj=open("decrypted.pdf", "rb")
pdfReader=PyPDF2.PdfFileReader(pdfFileObj)
pdfReader.numPages
pageObj=pdfReader.getPage(0)
pageObj.extractText()
With Tabula, I am getting the message "the output file is empty."
With PyPDF2, I am getting only '/n'
UPDATE 10/3/2019 Pdfminer.six (Version November 2018)
I got better results using the solution posted by DuckPuncher. For the decrypted file, I got the labels, but not the data. Same happens with the encrypted file. For the file that has never been encrypted works perfect. As I need the data and the labels of encrypted or decrypted files, this code does not work for me. For that analysis, I used pdfminer.six that is Python library that was released in November 2018. Pdfminer.six includes a library pycryptodome. According to their documentation "PyCryptodome is a self-contained Python package of low-level cryptographic primitives.."
The code is in the stack exchange question:
Extracting text from a PDF file using PDFMiner in python?
I would love if you want to repeat my experiment. Here is the description:
1) Run the codes mention in this question with any PDF that never has been encrypted.
2) Do the same with a PDF "Secure" (this is a term that Adobe uses), I am calling it the encrypted PDF. Use a generic form that you can find using Google. After you download it, you need to fill the fields. Otherwise, you would be checking for labels, but not fields. The data is in the fields.
3) Decrypt the encrypted PDF using Pykepdf. This will be the decrypted PDF.
4) Run the codes again using the decrypted PDF.
UPDATE 10/4/2019 Camelot (Version July 2019)
I found the Python library Camelot. Be careful that you need camelot-py 0.7.3.
It is very powerful, and works with Python 3.7. Also, it is very easy to use. First, you need also to install Ghostscript. Otherwise, it will not work.
You need also to install Pandas. Do not use pip install camelot-py. Instead use pip install camelot-py[cv]
The author of the program is Vinayak Mehta. Frank Du shares this code in a YouTube video "Extract tabular data from PDF with Camelot Using Python."
I checked the code and it is working with unencrypted files. However, it does not work with encrypted and decrypted files, and that is my goal.
Camelot is oriented to get tables from PDFs.
Here is the code:
Python
import camelot
import pandas
name_table = camelot.read_pdf("uncrypted.pdf")
type(name_table)
#This is a Pandas dataframe
name_table[0]
first_table = name_table[0]
#Translate camelot table object to a pandas dataframe
first_table.df
first_table.to_excel("unencrypted.xlsx")
#This creates an excel file.
#Same can be done with csv, json, html, or sqlite.
#To get all the tables of the pdf you need to use this code.
for table in name_table:
print(table.df)
UPDATE 10/7/2019
I found one trick. If I open the secured pdf with Adobe Reader, and I print it using Microsoft to PDF, and I save it as a PDF, I can extract the data using that copy. I also can convert the PDF file to JSON, Excel, SQLite, CSV, HTML, and another formats. This is a possible solution to my question. However, I am still looking for an option to do it without that trick because the goal is to do it 100% with Python. I am also concerned that if a better method of encryption is used the trick maybe would not work. Sometimes you need to use Adobe Reader several times to get an extractable copy.
UPDATE 10/8/2019. Third question.
I have now a third question. Do all secured/encrypted pdf are password protected? Why pikepdf is not working? My guess is that the current version of pikepdf can break some type of encryptions but not all of them.
#constt mentioned that PyPDF2 can break some type of protection. However, I replied to him that I found an article that PyPDF2 can break encryptions made with Adobe Acrobat Pro 6.0, but no with posterior versions.

LAST UPDATED 10-11-2019
I'm unsure if I understand your question completely. The code below can be refined, but it reads in either an encrypted or unencrypted PDF and extracts the text. Please let me know if I misunderstood your requirements.
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.converter import TextConverter
from pdfminer.layout import LAParams
from pdfminer.pdfpage import PDFPage
from io import StringIO
def extract_encrypted_pdf_text(path, encryption_true, decryption_password):
output = StringIO()
resource_manager = PDFResourceManager()
laparams = LAParams()
device = TextConverter(resource_manager, output, codec='utf-8', laparams=laparams)
pdf_infile = open(path, 'rb')
interpreter = PDFPageInterpreter(resource_manager, device)
page_numbers = set()
if encryption_true == False:
for page in PDFPage.get_pages(pdf_infile, page_numbers, maxpages=0, caching=True, check_extractable=True):
interpreter.process_page(page)
elif encryption_true == True:
for page in PDFPage.get_pages(pdf_infile, page_numbers, maxpages=0, password=decryption_password, caching=True, check_extractable=True):
interpreter.process_page(page)
text = output.getvalue()
pdf_infile.close()
device.close()
output.close()
return text
results = extract_encrypted_pdf_text('encrypted.pdf', True, 'password')
print (results)
I noted that your pikepdf code used to open an encrypted PDF was missing a password, which should have thrown this error message:
pikepdf._qpdf.PasswordError: encrypted.pdf: invalid password
import pikepdf
with pikepdf.open("encrypted.pdf", password='password') as pdf:
num_pages = len(pdf.pages)
del pdf.pages[-1]
pdf.save("decrypted.pdf")
You can use tika to extract the text from the decrypted.pdf created by pikepdf.
from tika import parser
parsedPDF = parser.from_file("decrypted.pdf")
pdf = parsedPDF["content"]
pdf = pdf.replace('\n\n', '\n')
Additionally, pikepdf does not currently implement text extraction this includes the latest release v1.6.4.
I decided to run a couple of test using various encrypted PDF files.
I named all the encrypted files 'encrypted.pdf' and they all used the same encryption and decryption password.
Adobe Acrobat 9.0 and later - encryption level 256-bit AES
pikepdf was able to decrypt this file
PyPDF2 could not extract the text correctly
tika could extract the text correctly
Adobe Acrobat 6.0 and later - encryption level 128-bit RC4
pikepdf was able to decrypt this file
PyPDF2 could not extract the text correctly
tika could extract the text correctly
Adobe Acrobat 3.0 and later - encryption level 40-bit RC4
pikepdf was able to decrypt this file
PyPDF2 could not extract the text correctly
tika could extract the text correctly
Adobe Acrobat 5.0 and later - encryption level 128-bit RC4
created with Microsoft Word
pikepdf was able to decrypt this file
PyPDF2 could extract the text correctly
tika could extract the text correctly
Adobe Acrobat 9.0 and later - encryption level 256-bit AES
created using pdfprotectfree
pikepdf was able to decrypt this file
PyPDF2 could extract the text correctly
tika could extract the text correctly
PyPDF2 was able to extract text from decrypted PDF files not created with Adobe Acrobat.
I would assume that the failures have something to do with embedded formatting in the PDFs created by Adobe Acrobat. More testing is required to confirm this conjecture about the formatting.
tika was able to extract text from all the documents decrypted with pikepdf.
import pikepdf
with pikepdf.open("encrypted.pdf", password='password') as pdf:
num_pages = len(pdf.pages)
del pdf.pages[-1]
pdf.save("decrypted.pdf")
from PyPDF2 import PdfFileReader
def text_extractor(path):
with open(path, 'rb') as f:
pdf = PdfFileReader(f)
page = pdf.getPage(1)
print('Page type: {}'.format(str(type(page))))
text = page.extractText()
print(text)
text_extractor('decrypted.pdf')
PyPDF2 cannot decrypt Acrobat PDF files => 6.0
This issue has been open with the module owners, since September 15, 2015. It unclear in the comments related to this issue when this problem will be fixed by the project owners. The last commit was June 25, 2018.
PyPDF4 decryption issues
PyPDF4 is the replacement for PyPDF2. This module also has decryption issues with certain algorithms used to encrypt PDF files.
test file: Adobe Acrobat 9.0 and later - encryption level 256-bit AES
PyPDF2 error message: only algorithm code 1 and 2 are supported
PyPDF4 error message: only algorithm code 1 and 2 are supported. This PDF uses code 5
UPDATE SECTION 10-11-2019
This section is in response to your updates on 10-07-2019 and 10-08-2019.
In your update you stated that you could open a 'secured pdf with Adobe Reader' and print the document to another PDF, which removes the 'SECURED' flag. After doing some testing, I believe that have figured out what is occurring in this scenario.
Adobe PDFs level of security
Adobe PDFs have multiple types of security controls that can be enabled by the owner of the document. The controls can be enforced with either a password or a certificate.
Document encryption (enforced with a document open password)
Encrypt all document contents (most common)
Encrypt all document contents except metadata => Acrobat 6.0
Encrypt only file attachments => Acrobat 7.0
Restrictive editing and printing (enforced with a permissions password)
Printing Allowed
Changes Allowed
The image below shows an Adobe PDF being encrypted with 256-Bit AES encryption. To open or print this PDF a password is required. When you open this document in Adobe Reader with the password, the title will state SECURED
This document requires a password to open with the Python modules that are mentioned in this answer. If you attempt to open an encrypted PDF with Adobe Reader. You should see this:
If you don't get this warning then the document either has no security controls enable or only has the restrictive editing and printing ones enabled.
The image below shows restrictive editing being enabled with a password in a PDF document. Note printing is enabled. To open or print this PDF a password is not required. When you open this document in Adobe Reader without a password, the title will state SECURED This is the same warning as the encrypted PDF that was opened with a password.
When you print this document to a new PDF the SECURED warning is removed, because the restrictive editing has been removed.
All Adobe products enforce the restrictions set by the permissions password. However, if third-party products do not support these settings, document recipients are able to bypass some or all of the restrictions set.
So I assume that the document that you are printing to PDF has restrictive editing enabled and does not have a password required to open enabled.
Concerning breaking PDF encryption
Neither PyPDF2 or PyPDF4 are designed to break the document open password function of a PDF document. Both the modules will throw the following error if they attempt to open an encrypted password protected PDF file.
PyPDF2.utils.PdfReadError: file has not been decrypted
The opening password function of an encrypted PDF file can be bypassed using a variety of methods, but a single technique might not work and some will not be acceptable because of several factors, including password complexity.
PDF encryption internally works with encryption keys of 40, 128, or 256 bit depending on the PDF version. The binary encryption key is derived from a password provided by the user. The password is subject to length and encoding constraints.
For example, PDF 1.7 Adobe Extension Level 3 (Acrobat 9 - AES-256) introduced Unicode characters (65,536 possible characters) and bumped the maximum length to 127 bytes in the UTF-8 representation of the password.
The code below will open a PDF with restrictive editing enabled. It will save this file to a new PDF without the SECURED warning being added. The tika code will parse the contents from the new file.
from tika import parser
import pikepdf
# opens a PDF with restrictive editing enabled, but that still
# allows printing.
with pikepdf.open("restrictive_editing_enabled.pdf") as pdf:
pdf.save("restrictive_editing_removed.pdf")
# plain text output
parsedPDF = parser.from_file("restrictive_editing_removed.pdf")
# XHTML output
# parsedPDF = parser.from_file("restrictive_editing_removed.pdf", xmlContent=True)
pdf = parsedPDF["content"]
pdf = pdf.replace('\n\n', '\n')
print (pdf)
This code checks if a password is required for opening the file. This code be refined and other functions can be added. There are several other features that can be added, but the documentation for pikepdf does not match the comments within the code base, so more research is required to improve this.
# this would be removed once logging is used
############################################
import sys
sys.tracebacklimit = 0
############################################
import pikepdf
from tika import parser
def create_pdf_copy(pdf_file_name):
with pikepdf.open(pdf_file_name) as pdf:
new_filename = f'copy_{pdf_file_name}'
pdf.save(new_filename)
return new_filename
def extract_pdf_content(pdf_file_name):
# plain text output
# parsedPDF = parser.from_file("restrictive_editing_removed.pdf")
# XHTML output
parsedPDF = parser.from_file(pdf_file_name, xmlContent=True)
pdf = parsedPDF["content"]
pdf = pdf.replace('\n\n', '\n')
return pdf
def password_required(pdf_file_name):
try:
pikepdf.open(pdf_file_name)
except pikepdf.PasswordError as error:
return ('password required')
except pikepdf.PdfError as results:
return ('cannot open file')
filename = 'decrypted.pdf'
password = password_required(filename)
if password != None:
print (password)
elif password == None:
pdf_file = create_pdf_copy(filename)
results = extract_pdf_content(pdf_file)
print (results)

You can try to handle the error these files produce when you open these files without a password.
import pikepdf
def open_pdf(pdf_file_path, pdf_password=''):
try:
pdf_obj = pikepdf.Pdf.open(pdf_file_path)
except pikepdf._qpdf.PasswordError:
pdf_obj = pikepdf.Pdf.open(pdf_file_path, password=pdf_password)
finally:
return pdf_obj
You can use the returned pdf_obj for your parsing work.
Also, you can provide the password in case you have an encrypted PDF.

For tabula-py, you can try password option with read_pdf. It depends on tabula-java's function so I'm not sure which encryption is supported though.

Python & MS Word: Convert .doc to .docx?

I found several questions that were similar to mine, but none of the answers came close to what I need.
Specifications: I'm working with Python 3 and do not have MS Word. My programming machine is running OS X and cloud machine is linux/ubuntu too.
I'm using python-docx to extract values from a .doc file that is sent to me nightly. However, python-docx only works with .docx files, so I need to convert the file to that extension first.
So, I've got a .doc file that I need to convert to .docx. This script might have to run in the cloud so I can't install any kind of Office or Office-like software. Can this be done?

You are working with Linux/ubuntu, you can use LibreOffice’s inbuilt converter.
SYNTAX
lowriter --convert-to docx *.doc
Example
lowriter --convert-to docx testdoc.doc
This will convert all doc files to docx and save in the same folder itself.

You could use unoconv - Universal Office Converter. Convert between any document format supported by LibreOffice/OpenOffice.
unoconv -d document --format=docx *.doc
subprocess.call(['unoconv', '-d', 'document', '--format=docx', filename])

Aspose.Words Cloud SDK for Python can convert DOC to DOCX. The package can open, generate, edit, split, merge, compare and convert a Word document in Python on any platform without depending on MS Word.
It is a paid product, but the free plan provides 150 free monthly API calls.
P.S: I'm developer evangelist at Aspose.
# Import module
import asposewordscloud
import asposewordscloud.models.requests
from shutil import copyfile
# Get your credentials from https://dashboard.aspose.cloud (free registration is required).
words_api = asposewordscloud.WordsApi(app_sid='xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx',app_key='xxxxxxxxxxxxxxxxxxxxxxxxx')
words_api.api_client.configuration.host = 'https://api.aspose.cloud'
filename = 'C:/Temp/02_pages.doc'
dest_name = 'C:/Temp/02_pages.docx'
#Convert RTF to text
request = asposewordscloud.models.requests.ConvertDocumentRequest(document=open(filename, 'rb'), format='docx')
result = words_api.convert_document(request)
copyfile(result, dest_name)

import aspose.words as aw
path1="doc file path"
path2="path to save converted file"
file2=file.rsplit('.',1)[0]+'.docx'
filename1=os.path.join(path2,file2)
filename=os.path.join(path1,file)
doc = aw.Document(filename)
doc.save(filename1)

First you will need to be using Windows. If that is an acceptable barrier then please read on....
Next you need to install the Microsoft Office Compatibility Pack.
Now download and install the Microsoft Office Migration Planning Manager.
To run the tool you need to create a .ini file that controls the program. An example .ini file and further information is available on this blog post.
There is more detailed information from Microsoft here.

Python: How to replace text in pdf

I have a pdf file and i want to replace some text in pdf file and generate new pdf. How can i do that in python?
I have tried reportlab , reportlab does not have any fucntion to search text and replace it. What other module can i use?

You can try Aspose.PDF Cloud SDK for Python, Aspose.PDF Cloud is a REST API PDF Processing solution. It is paid API and its free package plan provides 50 credits per month.
I'm developer evangelist at Aspose.
import os
import asposepdfcloud
from asposepdfcloud.apis.pdf_api import PdfApi
# Get App key and App SID from https://cloud.aspose.com
pdf_api_client = asposepdfcloud.api_client.ApiClient(
app_key='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
app_sid='xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx')
pdf_api = PdfApi(pdf_api_client)
filename = '02_pages.pdf'
remote_name = '02_pages.pdf'
copied_file= '02_pages_new.pdf'
#upload PDF file to storage
pdf_api.upload_file(remote_name,filename)
#upload PDF file to storage
pdf_api.copy_file(remote_name,copied_file)
#Replace Text
text_replace = asposepdfcloud.models.TextReplace(old_value='origami',new_value='polygami',regex='true')
text_replace_list = asposepdfcloud.models.TextReplaceListRequest(text_replaces=[text_replace])
response = pdf_api.post_document_text_replace(copied_file, text_replace_list)
print(response)

Have a look in THIS thread for one of the many ways to read text from a PDF. Then you'll need to create a new pdf, as they will, as far as I know, not retrieve any formatting for you.

The CAM::PDF Perl Library can output text that's not too hard to parse (it seems to fairly randomly split lines of text). I couldn't be bothered to learn too much Perl, so I wrote these really basic Perl command line scripts, one that reads a single page pdf to a text file perl read.pl pdfIn.pdf textOut.txt and one that writes the text (that you can modify in the meantime) to a pdf perl write.pl pdfIn.pdf textIn.txt pdfOut.pdf.
#!/usr/bin/perl
use Module::Load;
load "CAM::PDF";
$pdfIn = $ARGV[0];
$textOut = $ARGV[1];
$pdf = CAM::PDF->new($pdfIn);
$page = $pdf->getPageContent(1);
open(my $fh, '>', $textOut);
print $fh $page;
close $fh;
exit;
and
#!/usr/bin/perl
use Module::Load;
load "CAM::PDF";
$pdfIn = $ARGV[0];
$textIn = $ARGV[1];
$pdfOut = $ARGV[2];
$pdf = CAM::PDF->new($pdfIn);
my $page;
open(my $fh, '<', $textIn) or die "cannot open file $filename";
{
local $/;
$page = <$fh>;
}
close($fh);
$pdf->setPageContent(1, $page);
$pdf->cleanoutput($pdfOut);
exit;
You can call these with python either side of doing some regex etc stuff on the outputted text file.
If you're completely new to Perl (like I was), you need to make sure that Perl and CPAN are installed, then run sudo cpan, then in the prompt install "CAM::PDF";, this will install the required modules.
Also, I realise that I should probably be using stdout etc, but I was in a hurry :-)
Also also, any ideas what the format CAM-PDF outputs is? is there any doc for it?

Python convert Excel File (xls or xlsx) to/from ODS

I've been scouring the net to find a Python library or tool that can converts an Excel file to/from ODS format, but haven't been able to come across anything.
I need the ability to input and output data in either format. We don't need to worry about merged cells, formulas or anything non-straightforward.

If you have libreoffice installed, you can do a python execution wrapper around its headless mode:
$ /usr/bin/libreoffice --headless --invisible -convert-to ods /home/cwgem/Downloads/QTL_Sample_data.xls
convert /home/cwgem/Downloads/QTL_Sample_data.xls -> /home/cwgem/QTL_Sample_data.ods using OpenDocument Spreadsheet Flat XML
$ /usr/bin/libreoffice --headless --invisible -convert-to xls /home/cwgem/QTL_Sample_data.ods
convert /home/cwgem/QTL_Sample_data.ods -> /home/cwgem/QTL_Sample_data.xls using
Which would be a bit easier than trying to do it through the library route.

I succeeded to convert an xlsx file to an ods file with this method :
install pyexcel-xlsx
install pyexcel-ods3
And use the following code :
from pyexcel_ods3 import save_data
from pyexcel_xlsx import get_data
dataXlsx = get_data("file.xlsx")
save_data("file.ods", dataXlsx)
Attention : the colors/design of the xlsx file is removed in the ouput ods file...so that it is not a real success.

AppDailySales: Works, but the downloaded gzip file is corrupted

I am trying to use the appdailysales.py module to download daily our iPhone apps. I am a .NET developer, so I tried running this using IronPython in a C# solution using the following code:
using IronPython.Hosting;
var ipy = Python.CreateRuntime();
dynamic appSales = ipy.UseFile("appdailysales.py");
appSales.main();
Because I didn't have gzip, I took out the references to that module. I was going to use the GZipStream C# class to decompress the file (Apple, provides their downloads as .gz files). So, I commented out lines 75 and 429-435.
I have tried executing appdailysales.py in my C# solution, directly from IronPython and using Python 2.7 (installed ActivePython last night); all with the same results: When I try to open the .gz file using 7zip, I get the following error:
CRC Failed ... file is broken
When I try using the GZipStream class I get:
The CRC in GZip footer does not match the CRC calculated from the decompressed data
If I download the .gz file manually, I can decompress the file just fine using 7Zip or GZipStream.
I am fluent in C#, but new to Python. Any help you can provide would be much appreciated.
Thanks for your time.

Looks like line 444 is the problem. Here are lines 444-446:
downloadFile = open(filename, 'w')
downloadFile.write(filebuffer)
downloadFile.close()
At this stage, IF you have deleted lines 429-435 OR selected not to unzip, then filebuffer refers to the raw gzipped stream that you got from the web. The output file is opened in TEXT mode, and you are on Windows, so every \n in the BINARY gzipped stream will be converted to \r\n -- CORRUPTION, like the error message said.
So: for the module to be used portably on both Windows and other platforms, the open mode must be "wb" (b for binary). If the gunzipped result file is also a binary file, "wb" can be hardcoded in the open call. However if the gunzipped file is a text file (meant to be capable of being opened in a text editor), then you need just "w" for that purpose, and you should set a variable mode to either "wb" or "w" as appropriate, and use mode in the open call.
Big question: I understand why you removed the gzip references for IronPython usage. Did you remove those lines for Python 2.7? Or did you run it under Python 2.7 with those lines still in, but set options.unzipFile to False?