In Python I'm using pdfminer to read the text from a pdf with the code below this message. I now get an error message saying:
File "/usr/local/lib/python2.7/dist-packages/pdfminer/pdfpage.py", line 124, in get_pages
raise PDFTextExtractionNotAllowed('Text extraction is not allowed: %r' % fp)
PDFTextExtractionNotAllowed: Text extraction is not allowed: <cStringIO.StringO object at 0x7f79137a1
ab0>
When I open this pdf with Acrobat Pro it turns out it is secured (or "read protected"). From this link however, I read that there's a multitude of services which can disable this read-protection easily (for example pdfunlock.com. When diving into the source of pdfminer, I see that the error above is generated on these lines.
if check_extractable and not doc.is_extractable:
raise PDFTextExtractionNotAllowed('Text extraction is not allowed: %r' % fp)
Since there's a multitude of services which can disable this read-protection within a second, I presume it is really easy to do. It seems that .is_extractable is a simple attribute of the doc, but I don't think it is as simple as changing .is_extractable to True..
Does anybody know how I can disable the read protection on a pdf using Python? All tips are welcome!
================================================
Below you will find the code with which I currently extract the text from non-read protected.
def getTextFromPDF(rawFile):
resourceManager = PDFResourceManager(caching=True)
outfp = StringIO()
device = TextConverter(resourceManager, outfp, codec='utf-8', laparams=LAParams(), imagewriter=None)
interpreter = PDFPageInterpreter(resourceManager, device)
fileData = StringIO()
fileData.write(rawFile)
for page in PDFPage.get_pages(fileData, set(), maxpages=0, caching=True, check_extractable=True):
interpreter.process_page(page)
fileData.close()
device.close()
result = outfp.getvalue()
outfp.close()
return result
I had some issues trying to get qpdf to behave in my program. I found a useful library, pikepdf, that is based on qpdf and automatically converts pdfs to be extractable.
The code to use this is pretty straightforward:
import pikepdf
pdf = pikepdf.open('unextractable.pdf')
pdf.save('extractable.pdf')
As far as I know, in most cases the full content of the PDF is actually encrypted, using the password as the encryption key, and so simply setting .is_extractable to True isn't going to help you.
Per this thread:
Does a library exist to remove passwords from PDFs programmatically?
I would recommend removing the read-protection with a command-line tool such as qpdf (easily installable, e.g. on Ubuntu use apt-get install qpdf if you don't have it already):
qpdf --password=PASSWORD --decrypt SECURED.pdf UNSECURED.pdf
Then open the unlocked file with pdfminer and do your stuff.
For a pure-Python solution, you can try using PyPDF2 and its .decrypt() method, but it doesn't work with all types of encryption, so really, you're better off just using qpdf - see:
https://github.com/mstamy2/PyPDF2/issues/53
I used below code using pikepdf and able to overwrite.
import pikepdf
pdf = pikepdf.open('filepath', allow_overwriting_input=True)
pdf.save('filepath')
In my case there was no password, but simply setting check_extractable=False circumvented the PDFTextExtractionNotAllowed exception for a problematic file (that opened fine in other viewers).
Full disclosure, I am one of the maintainers of pdfminer.six. It is a community-maintained version of pdfminer for python 3.
This issue was fixed in 2020 by disabling the check_extractable by default. It now shows a warning instead of raising an error.
Similar question and answer here.
The 'check_extractable=True' argument is by design.
Some PDFs explicitly disallow to extract text, and PDFMiner follows the directive. You can override it (giving check_extractable=False), but do it at your own risk.
If you want to unlock all pdf files in a folder without renaming them, you may use this code:
import glob, os, pikepdf
p = os.getcwd()
for file in glob.glob('*.pdf'):
file_path = os.path.join(p, file).replace('\\','/')
init_pdf = pikepdf.open(file_path)
new_pdf = pikepdf.new()
new_pdf.pages.extend(init_pdf.pages)
new_pdf.save(str(file))
In pikepdf library it is impossible to overwrite the existing file by saving it with the same name. In contrast, you would like to copy the pages to the newly created empty pdf file, and save it.
I too faced the same problem of parsing the secured pdf but it has got resolved using pikepdf library. I tried this library on my jupyter notebbok and on windows os but it gave errors but it worked smoothly on Ubuntu
If you've forgotten the password to your PDF, below is a generic script which tries a LOT of password combinations on the same PDF. It uses pikepdf, but you can update the function check_password to use something else.
Usage example:
I used this when I had forgotten a password on a bank PDF. I knew that my bank always encrypts these kind of PDFs with the same password-structure:
Total length = 8
First 4 characters = an uppercase letter.
Last 4 characters = a number.
I call script as follows:
check_passwords(
pdf_file_path='/Users/my_name/Downloads/XXXXXXXX.pdf',
combination=[
ALPHABET_UPPERCASE,
ALPHABET_UPPERCASE,
ALPHABET_UPPERCASE,
ALPHABET_UPPERCASE,
NUMBER,
NUMBER,
NUMBER,
NUMBER,
]
)
Password-checking script:
(Requires Python3.8, with libraries numpy and pikepdf)
from typing import *
from itertools import product
import time, pikepdf, math, numpy as np
from pikepdf import PasswordError
ALPHABET_UPPERCASE: Sequence[str] = tuple('ABCDEFGHIJKLMNOPQRSTUVWXYZ')
ALPHABET_LOWERCASE: Sequence[str] = tuple('abcdefghijklmnopqrstuvwxyz')
NUMBER: Sequence[str] = tuple('0123456789')
def as_list(l):
if isinstance(l, (list, tuple, set, np.ndarray)):
l = list(l)
else:
l = [l]
return l
def human_readable_numbers(n, decimals: int = 0):
n = round(n)
if n < 1000:
return str(n)
names = ['', 'thousand', 'million', 'billion', 'trillion', 'quadrillion']
n = float(n)
idx = max(0,min(len(names)-1,
int(math.floor(0 if n == 0 else math.log10(abs(n))/3))))
return f'{n/10**(3*idx):.{decimals}f} {names[idx]}'
def check_password(pdf_file_path: str, password: str) -> bool:
## You can modify this function to use something other than pike pdf.
## This function should throw return True on success, and False on password-failure.
try:
pikepdf.open(pdf_file_path, password=password)
return True
except PasswordError:
return False
def check_passwords(pdf_file_path, combination, log_freq: int = int(1e4)):
combination = [tuple(as_list(c)) for c in combination]
print(f'Trying all combinations:')
for i, c in enumerate(combination):
print(f"{i}) {c}")
num_passwords: int = np.product([len(x) for x in combination])
passwords = product(*combination)
success: bool | str = False
count: int = 0
start: float = time.perf_counter()
for password in passwords:
password = ''.join(password)
if check_password(pdf_file_path, password=password):
success = password
print(f'SUCCESS with password "{password}"')
break
count += 1
if count % int(log_freq) == 0:
now = time.perf_counter()
print(f'Tried {human_readable_numbers(count)} ({100*count/num_passwords:.1f}%) of {human_readable_numbers(num_passwords)} passwords in {(now-start):.3f} seconds ({human_readable_numbers(count/(now-start))} passwords/sec). Latest password tried: "{password}"')
end: float = time.perf_counter()
msg: str = f'Tried {count} passwords in {1000*(end-start):.3f}ms ({count/(end-start):.3f} passwords/sec). '
msg += f"Correct password: {success}" if success is not False else f"All {num_passwords} passwords failed."
print(msg)
Comments
Obviously, don't use this to break into PDFs which are not your own. I hold no responsibility over how you use this script or any consequences of using it.
A lot of optimizations can be made.
Right now check_password uses pikepdf, which loads the file from disk for every "check". This is really slow, ideally it should run against an in-memory copy. I haven't figured out a way to do that, though.
You can probably speed this up a LOT by calling qpdf directly using C++, which is much better than Python for this kind of stuff.
I would avoid multi-processing here, since we're calling the same qpdf binary (which is normally a system-wide installation), which might become the bottleneck.
Related
The php code is
'''
$input_file = "a.txt";
$source = file_get_contents($input_file);
$source = gzcompress($source);
file_put_contents("php.txt",$source)
'''
The python code is
'''
testFile = "a.txt"
content = None
with open(testFile,"rb") as f:
content = f.read()
outContent = zlib.compress(content)
with open("py.txt","wb") as f:
f.write(outContent)
'''
The python3 version is [Python 3.6.9]
The php version is [PHP 7.2.17]
I need the same result for same md5.
The problem is not in PHP or Python, but rather in your "need". You cannot expect to get the same result, unless the two environments happen to be using the same version of the same compression code with the same settings. Since you do not have control of the version of code being used, your "need" can never be guaranteed to be met.
You should instead be doing your md5 on the decompressed data, not the compressed data.
I find the solution.
The code is
compress = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION,zlib.DEFLATED, 15, 9)
outContent = compress.compress(content)
outContent += compress.flush()
The python zlib provide a interface "zlib.compressobj",which returns a compressobj,and the parameters decide the result.
You can adjust parameters to make sure the python's result is same with php's
I need to read the details for a file in Windows so that I can interrogate the file's 'File version' as displayed in the Details tab of the File Properties window.
I haven't found anything in the standard library that makes this very easy to accomplish but figured if I could find the right windows function, I could probably accomplish it using ctypes.
Does anyone have any exemplary code or can they point me to a Windows function that would let me read this info. I took a look a GetFileAttributes already, but that wasn't quite right as far as I could tell.
Use the win32 api Version Information functions from ctypes. The api is a little fiddly to use, but I also want this so have thrown together a quick script as an example.
usage: version_info.py [-h] [--lang LANG] [--codepage CODEPAGE] path
Can also use as a module, see the VersionInfo class. Checked with Python 2.7 and 3.6 against a few files.
import array
from ctypes import *
def get_file_info(filename, info):
"""
Extract information from a file.
"""
# Get size needed for buffer (0 if no info)
size = windll.version.GetFileVersionInfoSizeA(filename, None)
# If no info in file -> empty string
if not size:
return ''
# Create buffer
res = create_string_buffer(size)
# Load file informations into buffer res
windll.version.GetFileVersionInfoA(filename, None, size, res)
r = c_uint()
l = c_uint()
# Look for codepages
windll.version.VerQueryValueA(res, '\\VarFileInfo\\Translation',
byref(r), byref(l))
# If no codepage -> empty string
if not l.value:
return ''
# Take the first codepage (what else ?)
codepages = array.array('H', string_at(r.value, l.value))
codepage = tuple(codepages[:2].tolist())
# Extract information
windll.version.VerQueryValueA(res, ('\\StringFileInfo\\%04x%04x\\'
+ info) % codepage, byref(r), byref(l))
return string_at(r.value, l.value)
So I'm trying to write a basic disassembler for a school project using Python. I'm using the pydasm and capstone libraries. What I don't understand is how I can actually access the assembly instructions of a program using Python. These libraries allow me to disassemble instructions, but I can't figure out how to access a program's instructions in Python. Could anyone give me some direction?
Thanks.
You should be cautious about the code section as the base of the code section might not contain only code (imports or read only data might be present at this location).
The best way to start a disassembly is by looking at the AddressOfEntryPoint field in the IMAGE_OPTIONAL_HEADER which indicates the first executed byte in the PE file (except if TLS is present but that's another subject).
A very good library for browsing PE files in python is pefile.
Here's an example to get the first 10 bytes at the program entry point:
#!/usr/local/bin/python2
# -*- coding: utf8 -*-
from __future__ import print_function
import sys
import os.path
import pefile
def find_entry_point_section(pe, eop_rva):
for section in pe.sections:
if section.contains_rva(eop_rva):
return section
return None
def main(file_path):
print("Opening {}".format(file_path))
try:
pe = pefile.PE(file_path, fast_load=True)
# AddressOfEntryPoint if guaranteed to be the first byte executed.
eop = pe.OPTIONAL_HEADER.AddressOfEntryPoint
code_section = find_entry_point_section(pe, eop)
if not code_section:
return
print("[+] Code section found at offset: "
"{:#x} [size: {:#x}]".format(code_section.PointerToRawData,
code_section.SizeOfRawData))
# get first 10 bytes at entry point and dump them
code_at_oep = code_section.get_data(eop, 10)
print("[*] Code at EOP:\n{}".
format(" ".join("{:02x}".format(ord(c)) for c in code_at_oep)))
except pefile.PEFormatError as pe_err:
print("[-] error while parsing file {}:\n\t{}".format(file_path,
pe_err))
if __name__ == '__main__':
if len(sys.argv) < 2:
print("[*] {} <PE_Filename>".format(sys.argv[0]))
else:
file_path = sys.argv[1]
if os.path.isfile(file_path):
main(file_path)
else:
print("[-] {} is not a file".format(file_path))
Simply pass the name of your PE file as the first argument.
In the above code the code_at_oep variable holds the first few bytes of the entry point. From there you can pass this bytes to the capstone engine.
Note that these first bytes might simply be a jmp or call instruction, so you'll have to follow the code execution in order to disassemble correctly. Disassembling correctly a program is still an open problem in computer science...
This depends on what OS you're using. You have some other questions about Linux, so I'm assuming you're using that. On Linux, executables are typically in ELF format, so you'll need a python library to read that or else to use some other tool to extract the part of the ELF file that you want.
The actual instructions are stored in the .text section, so if you extract that section's contents, those should be the raw bytes you want to disassemble.
so here is my code that disassembles your exe file and gives you an output in x86 assembly language I used pefile and capstone library
#!/usr/bin/python
import pefile
from capstone import *
# load the target PE file
pe = pefile.PE("/file/path/code.exe")
# get the address of the program entry point from the program header
entrypoint = pe.OPTIONAL_HEADER.AddressOfEntryPoint
# compute memory address where the entry code will be loaded into memory
entrypoint_address = entrypoint+pe.OPTIONAL_HEADER.ImageBase
# get the binary code from the PE file object
binary_code = pe.get_memory_mapped_image()[entrypoint:entrypoint+100]
# initialize disassembler to disassemble 32 bit x86 binary code
disassembler = Cs(CS_ARCH_X86, CS_MODE_32)
# disassemble the code
for instruction in disassembler.disasm(binary_code, entrypoint_address):
print("%s\t%s" %(instruction.mnemonic, instruction.op_str))
make sure to change and give the correct path of your exe file, also you need to specify how many instructions you want to print out at line number 15. here [entrypoint:entrypoint+100] I specify only 100 instructions but you can change it.
What would be a solid tool to use for generating PDF reports? Particularly, we are interested in creating interactive PDFs that have video, like the example found here.
Right now we are using Python and reportlab to generate PDFs, but have not explored the library completely (mostly because the license pricing is a little prohibitive)
We have been looking at the Adobe's SDK and iText libraries but it's hard to say what the capabilities of either are.
The ability to generate a document from a template PDF would be a plus.
Any pointers or comments will be appreciated.
Thanks,
Recently, I needed to create PDF reports for a Django application; a ReportLab license was available, but I ended up choosing LaTeX. The benefit of this approach is that we could use Django templates to generate the LaTeX source, and not get over encumbered writing lots of code for the many reports we needed to create. Plus, we could take advantage of the relatively much more concise LaTeX syntax (which does have it's many quirks and is not suitable for every purpose).
This snippet provides a general overview of the approach. I found it necessary to make some changes, which I have provided at the end of this question. The main addition is detection for Rerun LaTeX messages, which indicates an additional pass is required. Usage is as simple as:
def my_view(request):
pdf_stream = process_latex(
'latex_template.tex',
context=RequestContext(request, {'context_obj': context_obj})
)
return HttpResponse(pdf_stream, content_type='application/pdf')
It is possible to embed videos in LaTeX generated PDFs, however I do not have any experience with it. Here is a top Google result.
This solution does require spawning a new process (pdflatex), so if you want a pure Python solution keep looking.
import os
from subprocess import Popen, PIPE
from tempfile import NamedTemporaryFile
from django.template import loader, Context
class LaTeXException(Exception):
pass
def process_latex(template, context={}, type='pdf', outfile=None):
"""
Processes a template as a LaTeX source file.
Output is either being returned or stored in outfile.
At the moment only pdf output is supported.
"""
t = loader.get_template(template)
c = Context(context)
r = t.render(c)
tex = NamedTemporaryFile()
tex.write(r)
tex.flush()
base = tex.name
names = dict((x, '%s.%s' % (base, x)) for x in (
'log', 'aux', 'pdf', 'dvi', 'png'))
output = names[type]
stdout = None
if type == 'pdf' or type == 'dvi':
stdout = pdflatex(base, type)
elif type == 'png':
stdout = pdflatex(base, 'dvi')
out, err = Popen(
['dvipng', '-bg', '-transparent', names['dvi'], '-o', names['png']],
cwd=os.path.dirname(base), stdout=PIPE, stderr=PIPE
).communicate()
os.remove(names['log'])
os.remove(names['aux'])
# pdflatex appears to ALWAYS return 1, never returning 0 on success, at
# least on the version installed from the Ubuntu apt repository.
# so instead of relying on the return code to determine if it failed,
# check if it successfully created the pdf on disk.
if not os.path.exists(output):
details = '*** pdflatex output: ***\n%s\n*** LaTeX source: ***\n%s' % (
stdout, r)
raise LaTeXException(details)
if not outfile:
o = file(output).read()
os.remove(output)
return o
else:
os.rename(output, outfile)
def pdflatex(file, type='pdf'):
out, err = Popen(
['pdflatex', '-interaction=nonstopmode', '-output-format', type, file],
cwd=os.path.dirname(file), stdout=PIPE, stderr=PIPE
).communicate()
# If the output tells us to rerun, do it by recursing over ourself.
if 'Rerun LaTeX.' in out:
return pdflatex(file, type)
else:
return out
I suggest to use https://github.com/mreiferson/py-wkhtmltox to render HTML to PDF.
And use any tool you choose to render reports as HTML. I like http://www.makotemplates.org/
Basically i am trying to find out what version of ArcGIS the user currently has installed, i looked through the registry and couldn't find anything related to a version string. However i know it is stored, within the .exe.
I've done a fair bit of googling, and can't find anything really worth it. I tried using the GetFileVersionInfo, and i seem to get a random mishmash of stuff.
Any ideas?
EDIT
Sigh....
Turns out pywin32 is not always installed on all machines. Does anyone know if its possible to do the same thing via ctypes?
Also this is only for windows.
If you prefer not to do this using pywin32, you would be able to do this with ctypes, for sure.
The trick will be decoding that silly file version structure that comes back.
There's one old mailing list post that is doing what you're asking. Unfortunately, I don't have a windows box handy to test this myself, right now. But if it doesn't work, it should at least give you a good start.
Here's the code, in case those 2006 archives vanish sometime:
import array
from ctypes import *
def get_file_info(filename, info):
"""
Extract information from a file.
"""
# Get size needed for buffer (0 if no info)
size = windll.version.GetFileVersionInfoSizeA(filename, None)
# If no info in file -> empty string
if not size:
return ''
# Create buffer
res = create_string_buffer(size)
# Load file informations into buffer res
windll.version.GetFileVersionInfoA(filename, None, size, res)
r = c_uint()
l = c_uint()
# Look for codepages
windll.version.VerQueryValueA(res, '\\VarFileInfo\\Translation',
byref(r), byref(l))
# If no codepage -> empty string
if not l.value:
return ''
# Take the first codepage (what else ?)
codepages = array.array('H', string_at(r.value, l.value))
codepage = tuple(codepages[:2].tolist())
# Extract information
windll.version.VerQueryValueA(res, ('\\StringFileInfo\\%04x%04x\\'
+ info) % codepage, byref(r), byref(l))
return string_at(r.value, l.value)
print get_file_info(r'C:\WINDOWS\system32\calc.exe', 'FileVersion')
--
Ok - back near a windows box. Have actually tried this code now. "Works for me".
>>> print get_file_info(r'C:\WINDOWS\system32\calc.exe', 'FileVersion')
6.1.7600.16385 (win7_rtm.090713-1255)
there's a gnu linux utility called 'strings' that prints the printable characters in any file(binary or non-binary), try using that and look for a version number like pattern
on windows, you can get strings here http://unxutils.sourceforge.net/