I'm having an issue with pickle. Things work fine between OSX and Linux, but not Windows and Linux. All pickled strings are stored in memory and sent via an SSL socket. To be 100% clear I have replaced all '\n's with ":::" and all '\r's with "===" (there were none). Scenario:
Client-Win: Small Business Server 2011 running Python 2.7
Client-Lin: Fedora Linux running Python 2.7
Server: Fedora Linux running Python 2.7
Client-Lin sends a pickled object to Server:
ccopy_reg:::_reconstructor:::p0:::(c__main__:::infoCollection:::p1:::c__builtin__:::tuple:::p2:::(VSTRINGA:::p3:::VSTRINGB:::p4:::VSTRINGC:::p5:::tp6:::tp7:::Rp8:::.
Client-Win sends a picked object to Server:
ccopy_reg:::_reconstructor:::p0:::(c__main__:::infoCollection:::p1:::c__builtin__:::tuple:::p2:::(VSTRINGA:::p3:::VSTRINGB:::p4:::VSTRINGC:::p5:::tp6:::tp7:::Rp8:::ccollections:::OrderedDict:::p9:::((lp10:::(lp11:::S'string_a':::p12:::ag3:::aa(lp13:::S'string_b':::p14:::ag4:::aa(lp15:::S'string_c':::p16:::ag5:::aatp17:::Rp18:::b.
For some reason the Windows client sends extra information along with the pickle, and when the Linux client tries to load the pickle string I get:
Unhandled exception in thread started by <function TestThread at 0x107de60>
Traceback (most recent call last):
File "./test.py", line 212, in TestThread
info = pickle.loads(p_string)
File "/usr/lib64/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/usr/lib64/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib64/python2.7/pickle.py", line 1224, in load_build
d = inst.__dict__
AttributeError: 'infoCollection' object has no attribute '__dict__'
Any ideas?
EDIT
Adding additional requested information.
The infoCollection class is defined the same way:
infoCollection = collections.namedtuple('infoCollection', 'string_a, string_b, string_c')
def runtest():
info = infoCollection('STRINGA', 'STRINGB', 'STRINGC')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket(s, ssl_version=ssl.PROTOCOL_TLSv1)
ssl_sock.connect((server, serverport))
ssl_sock.write(pickle.dumps(info))
ssl_sock.close()
And the receiving function is much the same but does a
p_string = ssl_sock.read()
info = pickle.loads(p_string)
Are you using different minor versions of Python? There's a bug in 2.7.3 that makes pickling namedtuples incompatible with older versions. See this:
http://ronrothman.com/public/leftbraned/python-2-7-3-bug-broke-my-namedtuple-unpickling/
A hack, but the cross-platform issue appears to be due to namedtuples and pickle together in a cross-platform environment. I have replaced the namedtuple with my own class and all works well.
class infoClass(object):
pass
def infoCollection(string_a, string_b, string_c):
i = infoClass()
i.string_a = string_a
i.string_b = string_b
i.string_c = string_c
return i
Have you tried saving as a binary pickle file?
with open('pickle.file', 'wb') as po:
pickle.dump(obj, po)
Also - if you're porting between various OS, and if info is just a namedtuple have you looked at JSON (it's generally considered safer than pickle)?
with open('pickle.json', 'w') as po:
json.dump(obj, po)
Edit
From the ssl .read() docs it seems that .read() will only read at most 1024 bytes by default, I'll wager that your info namedtuple is going to be larger than that. It would be difficult to know how big info is a-priori I don't know if just setting nbytes=HUGE NUMBER would do the trick (I think perhaps not).
What happens if you
p_string = ssl_sock.read(nbytes=1.0E6)
info = pickle.loads(p_string)
Just install Pyhton 2.7.8 from https://www.python.org/ftp/python/2.7.8/python-2.7.8.amd64.msi
Related
I am using python 3's module tftpy to attempt to handle tftp style downloading of a file. However, when I run the application I get the following error:
\Python38\site-packages\tftpy\TftpStates.py", line 53, in handleOACK
raise TftpException("No options found in OACK")
tftpy.TftpShared.TftpException: No options found in OACK
How do I get my python project to ignore OACK/ send a new request packet that doesn't include an OACK?
Disclaimer: This is my first time attempting to work with TFTP packets so I am fairly new. If the question I posed isn't the appropriate way to handle it, what should I be doing?
MORE DATA ON THE PROBLEM:
I am using an external chip that is programmed to ignore OACK packet options.
When I used C# and the TFTP.Net package the transfer worked, so I don't believe it is an issue with my TFTP server. However, as our main application is based in python I want to be able to handle this communication via python 3.
I am running python 3.8.5
On my server side it is saying it receives a packet with error code 8.
python Script:
import tftpy
client = tftpy.TftpClient('192.168.0.42', 69)
client.download('triplog.txt', 'faultlog.txt', packethook=None, timeout=5)
full traceback:
Failed to negotiate options: No options found in OACK
Traceback (most recent call last):
File "C:\Users\selena\Documents\PythonScripts\TFTP\TFTPTestScript.py", line 23, in <module>
client.download('triplog.txt', 'faultlog.txt', packethook=None, timeout=5)
File "C:\Users\selena\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\tftpy\TftpClient.py", line 58, in download
self.context.start()
File "C:\Users\selena\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\tftpy\TftpContexts.py", line 402, in start
self.cycle()
File "C:\Users\selena\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\tftpy\TftpContexts.py", line 202, in cycle
self.state = self.state.handle(recvpkt, raddress, rport)
File "C:\Users\selena\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\tftpy\TftpStates.py", line 566, in handle
self.handleOACK(pkt)
File "C:\Users\selena\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\tftpy\TftpStates.py", line 53, in handleOACK
raise TftpException("No options found in OACK")
tftpy.TftpShared.TftpException: No options found in OACK
[Finished in 0.7s with exit code 1]
Credit #ewong for this workaround solution
The code worked after adding in options when initializing the client even though I didn't need them. I'll be filing an issue #https://github.com/msoulier/tftpy to see if this is a bug that needs to be addressed or a deliberate choice.
Solution code:
import tftpy
client = tftpy.TftpClient('192.168.0.42', 69, options={'blksize': 8})
client.download('triplog.txt', 'faultlog.txt', packethook=None, timeout=5)
I'm trying to fetch an instance to point to the already running AutoCAD 2016 application, or to create a new instance if it's not running.
Here's a simple code sample for that:
#Import needed modules
import os
import comtypes.client
from comtypes import COMError
from comtypes.client import CreateObject, GetModule, GetActiveObject
try:
acad = GetActiveObject("AutoCAD.Application.20")
print "AutoCAD is Active"
print "########"
except(OSError, COMError): #If AutoCAD isn't running, run it
acad = CreateObject("AutoCAD.Application.20",dynamic=True)
print "AutoCAD is successfuly Opened"
print "########"
When AutoCAD isn't running, the function CreateObject works successfully & only if dynamic is set to true .. but GetActiveObject throws the Traceback error
File "C:\Python27\lib\site-packages\comtypes\client\__init__.py", line 250, in CreateObject
return _manage(obj, clsid, interface=interface)
File "C:\Python27\lib\site-packages\comtypes\client\__init__.py", line 188, in _manage
obj = GetBestInterface(obj)
File "C:\Python27\lib\site-packages\comtypes\client\__init__.py", line 112, in GetBestInterface
interface = getattr(mod, itf_name)
AttributeError: 'module' object has no attribute 'IAcadApplication'
I've seen this but with no clue.
Also, I've returned to the documentation of comtypes, it says
The GetActiveObject function succeeds when the COM object is already running, and has registered itself in the COM running object table. Not all COM objects do this.
but this also doesn't give me any clue.
P.S GetActiveObject works well with me with other applications such as ETABS 2016 & SAP2000 v18
Any help & insightful responses will be totally appreciated, thanks in advance
It's weird but it does work (at least with me)
I went to the gen folder, its full path is C:\Python27\Lib\site-packages\comtypes\gen, then looked for the file AutoCAD.py, opened it in IDLE, knew its equivalent generated symbol (ID) such as _4E3F492A_FB57_4439_9BF0_1567ED84A3A9_0_1_0, then deleting these 4 files:
AutoCAD.py
AutoCAD.pyc
_4E3F492A_FB57_4439_9BF0_1567ED84A3A9_0_1_0.py
_4E3F492A_FB57_4439_9BF0_1567ED84A3A9_0_1_0.pyc
You can notice that the last 2 files are named by the same equivalent generated symbol which is mentioned in the file AutoCAD.py .
I have just installed ocropus OCR with all dependencies in my windows 7 machine. (I am using 32bit python 2.7) It seems to be working fine except that I cannot load the default OCR model: en-default.pyrnn.gz. , and receiving a Traceback. I am using the following syntax:
python ocropus-rpred -m en-default.pyrnn.gz book\0001\*.png
here is the error
INFO: #inputs47
# loading object /usr/local/share/ocropus/en-default.pyrnn.gz
Traceback (most recent call last):
File "ocropus-rpred" line 109, in <module>
network = ocrolib.load_object(args.model,verbose=1)
File "C:\anaconda32\lib\site-packages\ocrolib\common.py", line 513, in load_object
return unpickler.load()
EOFError
I have checked the file is not empty; also double checked the binary mode flag enabled i.e. "wb" and "rb"; also converted the newlines of common.py using dos2unix. I am being unable to unable to solve this problem. If anyone have expereinced similar issues, kindly share.
import cPickle
import gzip
def save_object(fname,obj,zip=0):
if zip==0 and fname.endswith(".gz"):
zip = 1
if zip>0:
# with gzip.GzipFile(fname,"wb") as stream:
with os.popen("gzip -9 > '%s'"%fname,"wb") as stream:
cPickle.dump(obj,stream,2)
else:
with open(fname,"wb") as stream:
cPickle.dump(obj,stream,2)
def unpickle_find_global(mname,cname):
if mname=="lstm.lstm":
return getattr(lstm,cname)
if not mname in sys.modules.keys():
exec "import "+mname
return getattr(sys.modules[mname],cname)
def load_object(fname,zip=0,nofind=0,verbose=0):
"""Loads an object from disk. By default, this handles zipped files
and searches in the usual places for OCRopus. It also handles some
class names that have changed."""
if not nofind:
fname = ocropus_find_file(fname)
if verbose:
print "# loading object",fname
if zip==0 and fname.endswith(".gz"):
zip = 1
if zip>0:
# with gzip.GzipFile(fname,"rb") as stream:
with os.popen("gunzip < '%s'"%fname,"rb") as stream:
unpickler = cPickle.Unpickler(stream)
unpickler.find_global = unpickle_find_global
return unpickler.load()
else:
with open(fname,"rb") as stream:
unpickler = cPickle.Unpickler(stream)
unpickler.find_global = unpickle_find_global
return unpickler.load()
UPDATE: Hi, please note that I have used Python's native gzip, and it is working fine. Thank you for pointing that out. Here is the correct syntax that is working on Windows: {with gzip.GzipFile(fname,"rb") as stream:}
Your use of gunzip (in the load_object function) is incorrect. Unless passed the -c argument, gunzip writes the decompressed data to a new file, not to its stdout (which is what you seem to be attempting to do).
As a result, it doesn't write anything to its stdout, and your stream variable contains no data, hence the EOFError.
A quick fix is to change your gunzip command line to give it the -c argument.
More info here: http://linux.die.net/man/1/gzip
That said, why are you even shelling out to gunzip to decompress your data? Python's built-in gzip module should handle that without problems.
I have seen simple code in stackoverflow using pyserial in USB ports with Python 3.3 but I can't get this to work on my new installation of pyserial 2.7 [in Windows 7, 64 bit, with 3 USB ports]. Installation of pyserial went smoothly, I can import without error and methods are recognized in the Pyscripter IDE which boosts confidence in a good installation, however:
The code stripped down to its error producing essentials is:
import serial
def main():
ser = serial.Serial(port='COM2')
ser.close()
if __name__ == '__main__':
main
From this I receive a dialog box with the error "SerialException: could not open port 'COM2': FileNotFoundError(2,'The system cannot find the file specified.',None,2)"
The Traceback states:
*** Remote Interpreter Reinitialized ***
>>>
Traceback (most recent call last):
File "<string>", line 420, in run_nodebug
File "C:\Python33\Lib\site-packages\scanport2.py", line 19, in <module>
main()
File "C:\Python33\Lib\site-packages\scanport2.py", line 15, in main
ser = serial.Serial(port='COM2')
File "C:\Python33\Lib\site-packages\serial\serialwin32.py", line 38, in __init__
SerialBase.__init__(self, *args, **kwargs)
File "C:\Python33\Lib\site-packages\serial\serialutil.py", line 282, in __init__
self.open()
File "C:\Python33\Lib\site-packages\serial\serialwin32.py", line 66, in open
raise SerialException("could not open port %r: %r" % (self.portstr, ctypes.WinError()))
serial.serialutil.SerialException: could not open port 'COM2': FileNotFoundError(2, 'The system cannot find the file specified.', None, 2)
And the code segment in the imported module which raises the SerialException is:
# the "\\.\COMx" format is required for devices other than COM1-COM8
# not all versions of windows seem to support this properly
# so that the first few ports are used with the DOS device name
port = self.portstr
try:
if port.upper().startswith('COM') and int(port[3:]) > 8:
port = '\\\\.\\' + port
except ValueError:
# for like COMnotanumber
pass
self.hComPort = win32.CreateFile(port,
win32.GENERIC_READ | win32.GENERIC_WRITE,
0, # exclusive access
None, # no security
win32.OPEN_EXISTING,
win32.FILE_ATTRIBUTE_NORMAL | win32.FILE_FLAG_OVERLAPPED,
0)
if self.hComPort == win32.INVALID_HANDLE_VALUE:
self.hComPort = None # 'cause __del__ is called anyway
raise SerialException("could not open port %r: %r" % (self.portstr, ctypes.WinError()))
I do have an active device connected to COM2 as identified in the Windows device manager. I also have tried scanning all the ports, but the code stops on the first use of serial.Serial
This appears that something may be going on with win32?
I am a newbie for interfacing Python with hardware.
I would try the following:
Unplug and replug the device.
Reboot.
Run WinObj and look in the GLOBAL?? folder; you should see COM2 there as a symbolic link to something more driver-specific.
What type of device do you have connected to COM2? If it uses usbser.sys, you might have better luck substituting \\.\USBSER000 for COM2 in your code, but remember to escape those backslashes properly.
On some machines there are strange problems with low COM port numbers that I can't explain. Try reassigning the device to COM6 in the Device Manager.
It looks like the pyserial download page only contains links for 32 bit python? This unofficial page seems to have links for 64 bit installations, however be cautious installing from unknown sources.
This answer also suggests installing it using pip: https://stackoverflow.com/a/8491164/66349
I'm using the (awesome) mrjob library from Yelp to run my python programs in Amazon's Elastic Map Reduce. It depends on subprocess in the standard python library. From my mac running python2.7.2, everything works as expected
However, when I switched to using the exact same code on Ubuntu LTS 11.04 also with python2.7.2, I encountered something strange:
mrjob loads the job, and then attempts to communicate with its child processes using subprocess and generates this error:
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.3.1-py2.7.egg/mrjob/emr.py", line 1212, in _build_steps
steps = self._get_steps()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.3.1-py2.7.egg/mrjob/runner.py", line 1003, in _get_steps
stdout, stderr = steps_proc.communicate()
File "/usr/lib/python2.7/subprocess.py", line 754, in communicate
return self._communicate(input)
File "/usr/lib/python2.7/subprocess.py", line 1302, in _communicate
stdout, stderr = self._communicate_with_poll(input)
File "/usr/lib/python2.7/subprocess.py", line 1332, in _communicate_with_poll
poller = select.poll()
AttributeError: 'module' object has no attribute 'poll'
This appears to be a problem with subprocess and not mrjob.
I dug into /usr/lib/python2.7/subprocess.py and found that during import it runs:
if mswindows:
... snip ...
else:
import select
_has_poll = hasattr(select, 'poll')
By editing that, I verified that it really does set _has_poll==True. And this is correct; easily verified on the command line.
However, when execution progresses to using Popen._communicate_with_poll somehow the select module has changed! This is generated by printing dir(select) right before it attempts to use select.poll().
['EPOLLERR', 'EPOLLET', 'EPOLLHUP', 'EPOLLIN', 'EPOLLMSG',
'EPOLLONESHOT', 'EPOLLOUT', 'EPOLLPRI', 'EPOLLRDBAND',
'EPOLLRDNORM', 'EPOLLWRBAND', 'EPOLLWRNORM', 'PIPE_BUF',
'POLLERR', 'POLLHUP', 'POLLIN', 'POLLMSG', 'POLLNVAL',
'POLLOUT', 'POLLPRI', 'POLLRDBAND', 'POLLRDNORM',
'POLLWRBAND', 'POLLWRNORM', '__doc__', '__name__',
'__package__', 'error', 'select']
no attribute called 'poll'!?!? How did it go away?
So, I hardcoded _has_poll=False and then mrjob happily continues with its work, runs my job in AWS EMR, with subprocess using communicate_with_select... and I'm stuck with a hand-modified standard library...
Any advice? :-)
I had a similar problem and it turns out that gevent replaces the built-in select module with gevent.select.select which doesn't have a poll method (as it is a blocking method).
However for some reason by default gevent doesn't patch subprocess which uses select.poll.
An easy fix is to replace subprocess with gevent.subprocess:
import gevent.monkey
gevent.monkey.patch_all(subprocess=True)
import sys
import gevent.subprocess
sys.modules['subprocess'] = gevent.subprocess
If you do this before importing the mrjob library, it should work fine.
Sorry for writing a full answer instead of a comment, otherwise I'd lose code indentation.
I cannot help you directly since something seems very strictly tied to your code, but I can help you find out, by relying on the fact that Python modules can be arbitrary object, try something like that:
class FakeModule(dict):
def __init__(self, origmodule):
self._origmodule = origmodule
self.__all__ = dir(origmodule)
def __getattr__(self, attr):
return getattr(self._origmodule, attr)
def __delattr__(self, attr):
if attr == "poll":
raise RuntimeError, "Trying to delete poll!"
self._origmodule.__delattr__(attr)
def replaceSelect():
import sys
import select
fakeselect = FakeModule(select)
sys.modules["select"] = fakeselect
replaceSelect()
import select
del select.poll
and you'll get an output like:
Traceback (most recent call last):
File "domy.py", line 27, in <module>
del select.poll
File "domy.py", line 14, in __delattr__
raise RuntimeError, "Trying to delete poll!"
RuntimeError: Trying to delete poll!
By calling replaceSelect() in your code you should be able to get a traceback of where somebody is deleting poll(), so you can understand why.
I hope my FakeModule implementation is good enough, otherwise you might need to modify it.