We Keep Coding

Why python-magic returns wrong mime-type if file size is too small?

In case of the file size is under 5000 bytes (InMemoryUploadedFile).
This code doesn't work
mime_type = magic.from_buffer(file.read(), mime=True)
It returns wrong mime_type.
For example, I have a file cv.docx with 4074 bytes size.
It returns a mime_type:
'application/x-empty'
instead of
'application/vnd.openxmlformats-officedocument.wordprocessingml.document'
Could you please suggest me any advices to solve this case?

I had this problem as well. It's very likely not to do with the file size, because I have tested magic.from_buffer on 90 byte text/plain files as well and it returned the right value.
The problem is that the file has somehow become empty. In my case, this is because the file was a stream and I had already read from the stream (remember if you read from a stream and read again, the second read will start where the first read finished -- unlike reading from the start of a file each time).
This example is from flask
mime_type1 = magic.from_buffer(request.stream.read(2048), mime=True) // returns text/plain
mime_type = magic.from_buffer(request.files["file"].stream.read(2048), mime=True) // returns application/x-empty because the stream has already been read from
It's hard to exactly diagnose without seeing your earlier code but check where else you are working with the file and comment those out.
You might need to do something like
file.seek(0)
mime_type = magic.from_buffer(file.read(), mime=True)

PyPDF2 PdfReadError: Could not read Boolean object

I am getting the following error, when reading certain PDF files using PyPDF2. Due to the confidential nature of these documents, I can't share them, but I can try and provide information which can help solve this problem.
Stacktrace -
inputpdf = PdfFileReader(open(pdfpath, "rb"), strict=False)
File "/home/tata/.virtualenvs/obu/local/lib/python2.7/site-packages/PyPDF2/pdf.py", line 1084, in __init__
self.read(stream)
File "/home/tata/.virtualenvs/obu/local/lib/python2.7/site-packages/PyPDF2/pdf.py", line 1732, in read
num = readObject(stream, self)
File "/home/tata/.virtualenvs/obu/local/lib/python2.7/site-packages/PyPDF2/generic.py", line 74, in readObject
return BooleanObject.readFromStream(stream)
File "/home/tata/.virtualenvs/obu/local/lib/python2.7/site-packages/PyPDF2/generic.py", line 137, in readFromStream
raise utils.PdfReadError('Could not read Boolean object')
PdfReadError: Could not read Boolean object
The exception seems to be raised from the following function, in generic.py:
def readFromStream(stream):
word = stream.read(4)
if word == b_("true"):
return BooleanObject(True)
elif word == b_("fals"):
stream.read(1)
return BooleanObject(False)
else:
raise utils.PdfReadError('Could not read Boolean object')
Printing the variable word prints the string trai, but I am not sure what this string represents.
Since the PyPDF2 project seems unmaintained, can someone help me figure out a solution for this?
Note : Please note that these PDFs are not password protected.

It seems as if all pdfs are encrypted in some way. Using the solution cited in this issue #53 in PyPDF2's github repository, I used the following command to generate another pdf ( The Decrypted version of the original pdf ) -
qpdf --password= --decrypt input.pdf output.pdf
and then reading output.pdf worked for me. I am not sure as to how I can determine beforehand, whether a pdf is encrypted ( or in this particular state ) or not. But this solution temporarily solves the problem.

Parsing binary Stanford polygon files (PLY) with Pyparsing

For a larger project, I'm currently in the process of writing a Stanford polygon file (PLY) parser. The example at Github Gists is currently capable of parsing ASCII-format PLY files into a data abstraction Mesh. It also contains a description of the actual grammar, for those inclined.
However the format definition (PLY - Polygon File Format) also includes two binary formats (little and big endian). Since those two formats are much more common (and storage-space efficient), I would like to be able to parse those files with pyparsing as well.
I'm grateful for some advice on how to do that, if at all possible.
The idea of the binary PLY files is that, the header portion consists of an ASCII description of the actual data of the file, and the body contains the actual data. An example (data in brackets are hex bytes):
ply
format binary_little_endian 1.0
element vertex 1
property float x
property float y
property float z
property uchar red
property uchar green
property uchar blue
property uchar alpha
end_header
[84 72 F1 C1 D8 FD 9F C1 00 00 00 00 3B 45 CB FF]
My first approach was to just load the input file in binary format (using bytes instead of str), and adapt the parser accordingly, but this somehow throws pyparsing off track. Also, I don't really know how to tell pyparsing how to grok byte groups.
File "components.py", line 338, in create
mesh = PlyParser.create().load(mesh_path)
File "model_parser.py", line 120, in create
property_position = aggregate_property("position", b"x", b"y", b"z")
File "model_parser.py", line 113, in aggregate_property
aggregates.append(pp.Group(property_simple_prefix + keyword_or(*keywords)("name")))
File "model_parser.py", line 87, in keyword_or
return pp.Or(pp.CaselessKeyword(literal) for literal in keywords)
File "pyparsing.py", line 3418, in __init__
super(Or,self).__init__(exprs, savelist)
File "pyparsing.py", line 3222, in __init__
exprs = list(exprs)
File "model_parser.py", line 87, in <genexpr>
return pp.Or(pp.CaselessKeyword(literal) for literal in keywords)
File "pyparsing.py", line 2496, in __init__
super(CaselessKeyword,self).__init__( matchString, identChars, caseless=True )
File "pyparsing.py", line 2422, in __init__
self.matchLen = len(matchString)
TypeError: object of type 'int' has no len()

What you might want to try is to open the file as text, use pyparsing to parse the header and capture the end position of the "end header" token. Use the structure information extracted from the header to build a Python struct reader that will process the binary content. Then reopen the file as binary, seek to the position, and use the struct reader to load the binary content. Probably simpler than twisting pyparsing to be both text and binary.

There is already a module for parsing binary PLY files: python-plyfile.
You could either use this or at least look at the source code to get an idea how it works.
It uses numpy.fromfile - which is described as a "highly efficient way of reading binary data with a known data-type" - to to the binary data reading.

Python poplib error_proto: line too long

Recently, without changes to codes/libs, I started getting python error_proto: line too long error when reading email (poplib.retr) from hotmail inbox. I am using Python version 2.7.8. I understand that a long line may be caused this error. But is there a way to go around this or a certain version I need to put in place. Thank you for any advice/direction anyone can give.
Here is a traceback error:
"/opt/rh/python27/root/usr/lib64/python2.7/poplib.py", line 232, in retr\n return self._longcmd(\'RETR %s\' % which)\n',
' File "/opt/rh/python27/root/usr/lib64/python2.7/poplib.py", line 167, in _longcmd\n return self._getlongresp()\n',
' File "/opt/rh/python27/root/usr/lib64/python2.7/poplib.py", line 152, in _getlongresp\n line, o = self._getline()\n',
' File "/opt/rh/python27/root/usr/lib64/python2.7/poplib.py", line 377, in _getline\n raise error_proto(\'line too long\')\n',
'error_proto: line too long\n'

A python bug report exists for this issue here: https://bugs.python.org/issue16041
The work around I put inplace was as follows:
import poplib
poplib._MAXLINE=20480
I thought this was a better idea, rather than editing the poplib.py library file directly.
Woody

Are you sure you've not updated poplib? Have a look at the most recent diff, committed last night:
# Added:
...
# maximal line length when calling readline(). This is to prevent
# reading arbitrary length lines. RFC 1939 limits POP3 line length to
# 512 characters, including CRLF. We have selected 2048 just to be on
# the safe side.
_MAXLINE = 2048
...
# in_getline()...
if len(self.buffer) > _MAXLINE:
raise error_proto('line too long')
...it looks suspiciously similar to your problem.
So if you roll back to the previous version, it will probably be OK.

Python Overwrite Dictionary to Text File doesn't work... why?

I've been previously writing code for a quiz program with a text file that stores all of the participants' results. The code that converts the text file to a dictionary and the text file itself are shown below:
Code:
import collections
from collections import defaultdict
scores_guessed = collections.defaultdict(lambda: collections.deque(maxlen=4))
with open('GuessScores.txt') as f:
for line in f:
name,val = line.split(":")
scores_guessed[name].appendleft(int(val))
for k in sorted(scores_guessed):
print("\n"+k," ".join(map(str,scores_guessed[k])))
writer = open('GuessScores.txt', 'wb')
for key, value in scores_guessed.items():
output = "%s:%s\n" % (key,value)
writer.write(output)
The text file appears like this:
Jack:10
Dave:20
Adam:30
Jack:40
Adam:50
Dave:60
Jack:70
Dave:80
Jack:90
Jack:100
Dave:110
Dave:120
Adam:130
Adam:140
Adam:150
Now, when I run the program code, the dictionary appears like this:
Adam 150 140 130 50
Dave 120 110 80 60
Jack 100 90 70 40
Now, this arranges the dictionary into order of highest scores, and the top 4 scores!
I want the python IDLE to overwrite the GuessScores.txt to this:
Adam:150
Adam:140
Adam:130
Adam:50
Dave:120
Dave:110
Dave:80
Dave:60
Jack:100
Jack:90
Jack:70
Jack:40
BUT when I run the code, this error appears:
Traceback (most recent call last):
File "/Users/Ahmad/Desktop/Test Files SO copy/readFile_prompt.py", line 16, in <module>
writer.write(output)
TypeError: 'str' does not support the buffer interface
The GuessScores.txt file is empty because it cannot write to the file, since there is the error above.
Why is this happening? And what is the fix? I have asked this previously but there were numerous issues. I am running Python 3.3.2 on a Mac 10.8 Mavericks iMac, if that makes any help.
Thanks,
Delbert.

The first issue is that you are trying to write text to a file that you opened in binary mode. In 3.x, this will no longer work. "text" vs. "binary" used to mean very little (only affecting line-ending translation, so no difference at all on some systems). Now it means like what it sounds like: a file opened in text mode is one whose contents are to be treated like text with some specific encoding, and a file opened in binary mode is one whose contents are to be treated as a sequence of bytes.
Thus, you need open('GuessScores.txt', 'w'), not open('GuessScores.txt', 'wb').
That said, you really should be using with blocks to manage the files, and you're going to have to write code that actually formats the dictionary content in the way you want. I assume you intend to output in sorted name order, and you need to iterate over each deque and write a line for each item. Something like:
with open('GuessScores.txt', 'w') as f:
for name, scores in sorted(scores_guessed.items()):
for score in scores:
f.write("{}:{}\n".format(name, score))
(Note also the new-style formatting.)
If necessary, you can explicitly specify the encoding of the file in the open call, with the encoding keyword parameter. (If you don't know what I mean by "encoding", you must learn. I'm serious. Drop everything and look it up.)

The writing problem has to do with the b in your open function. You've opened it in binary mode, so only bytes can be written. You can either remove the b or call bytes on output to give it the right type. You have a logic error anyway though. When I run it on Python 2.7, the output to GuessedScores.txt is this:
Dave:deque([120,110,80,60],maxlen=4)
Jack:deque([100, 90, 70, 40], maxlen=4)
Adam:deque([150, 140, 130, 50], maxlen=4)
So your values are the whole deques, not the individual scores. You'll have to format them, similar to how you did in your print statement.