so i Worte some code with basic structure like this:
from numpy import *
from dataloader import loadfile
from IPython.parallel import Client
from clustering import *
data = loadfile(0)
N_CLASSES = 10
rowmax = nanmax(data.values, 0)
rowmin = nanmin(data.values, 0)
# Defines the size of block processed at a time
BLOCK_SIZE = 50000
classmean, classcov, classcovinv, classlogdet, classlogprob = init_stats(data, N_CLASSES, rowmax, rowmin)
client = Client()
ids = client.ids
nodes = len(ids)
view = client.load_balanced_view()
dview = client[:]
def get_ml_class(data, args): do sth
dview.scatter('datablock', data)
dview.execute('res1, res2 = get_ml_class(datablock, args)', block=False)
the output of dview.execute part is
<AsyncResult: execute>
which means it is executed, however, when i was trying to pull the result by
dview.pull(['res1','res2'], block=True)
it shows:
NameError: name 'res1' is not defined
Can someone please tell me what is wrong with my code?? Thank you so much!
Let us make a simpler example:
from IPython.parallel import Client
rc = Client()
dview = rc[:]
dview.scatter('a', Range(16))
dview.execute('res1,res2 = a[0], a[1]', block=False)
dview.pull(['res1'], block=True)
This works as expected and gives a the result:
[[0], [4], [8], [12]]
So, we do at least this one right. But let me change the code a bit:
from IPython.parallel import Client
rc = Client()
dview = rc[:]
dview.scatter('a', Range(16))
dview.execute('res1,res2 = a[0], b[1]', block=False)
dview.pull(['res1'], block=True)
Now we have the NameError. Why?
Because there is an error on the execute line (it references to variable b which does not exist). The non-blocking execute does not complain much. In the first (working) case the status is:
<AsyncResult: finished>
and in the second (non-working) case:
<AsyncResult: execute>
Other than that it is very quiet, and the second message does not necessarily mean an error has occured. In order to see the real error message, change blocking to True. Then you'll see what is going wrong.
If you want to know if your non-blocking execute works, you have to capture the AsyncResult object returned by execute. It has several interesting methods, but you would be most interested in ready and successful methods:
ar = dview.execute(...)
ar.ready() # True if the process has finished
ar.successful() # True if there were no exceptions raised
Also, the possible exceptions raised during the execution can be fetched by using the get method of the AsyncResult object. For example my bad example gives in the interactive shell:
>>> ar.get()
[0:execute]:
---------------------------------------------------------------------------
NameError Traceback (most recent call last)<ipython-input-1-608f57d70b2f> in <module>()
----> 1 res1,res2=a[0]**2,b[1]**2
NameError: name 'b' is not defined
[1:execute]:
---------------------------------------------------------------------------
NameError Traceback (most recent call last)<ipython-input-1-608f57d70b2f> in <module>()
----> 1 res1,res2=a[0]**2,b[1]**2
NameError: name 'b' is not defined
...
So, as a summary: Try to find out what goes wrong with the function you try to run remotely. Now it seems to raise some execption. The error might have something to do with args which does not seem to be available for the remote scripts. Maybe a scatter is missing?
Related
I'm working on code that retrieves information from Twilio's Flow system through their API. That part of the code functions fine, but when I try to mock it for unit testing, it's throwing an error from the mocked api response.
Here is the code being tested:
from twilio.rest import Client
class FlowChecker:
def __init__(self, twilio_sid, twilio_auth_token):
self.twilio_SID = twilio_sid
self.twilio_auth_token = twilio_auth_token
self.client = Client(self.twilio_SID, self.twilio_auth_token)
self.calls = self.client.calls.list()
self.flows = self.client.studio.v2.flows
def get_active_executions(self):
active_executions = []
for flow in self.flows.list():
executions = self.client.studio.v2.flows(flow.sid).executions.list()
for execution in executions:
if execution._properties['status'] != 'ended':
active_executions.append({'flow_sid': flow.sid, 'execution': execution})
And here is my unit test code that's throwing the error:
import unittest
from unittest.mock import Mock, patch
from flows.twilio_flows import FlowChecker
class FlowCheckerTest(unittest.TestCase):
#patch('flows.twilio_flows.Client')
def test_get_active_flows(self, mock_client):
flow_checker = FlowChecker('fake_sid', 'fake_auth_token')
mock_call = Mock()
mock_flow = Mock()
mock_flow.sid = 0
mock_execution = Mock()
mock_execution._properties = {'status': 'ended'}
mock_client.calls.list().return_value = [mock_call]
mock_client.studio.v2.flows = [mock_flow]
mock_client.studio.v2.flows(mock_flow.sid).executions.list().return_value = [mock_execution]
self.assertEqual(flow_checker.get_active_executions(), [])
And here is the error traceback:
Ran 2 tests in 0.045s
FAILED (errors=1)
Error
Traceback (most recent call last):
File "C:\Users\Devon\AppData\Local\Programs\Python\Python310\lib\unittest\mock.py", line 1369, in patched
return func(*newargs, **newkeywargs)
File "C:\Users\Devon\PycharmProjects\Day_35\tests\twilio_flows_test'.py", line 19, in test_get_active_flows_when_empty
mock_client.studio.v2.flows(mock_flow.sid).executions.list().return_value = [mock_execution]
TypeError: 'list' object is not callable
Process finished with exit code 1
As you can see, "mock_client.calls.list().return_value = [mock_call]" doesn't throw any errors during init, and the first code block runs fine. It's only the mocked executions.list() that's throwing the error in the test.
Can anyone clear this up?
Thank you!
I've tried researching this specific issue and was unable to find information addressing it. It's a very specific deeply nested function in a vendor supplied client that I need to test, so I don't know what to try.
The problem isn't with .list(), it's with .flows().
mock_client.studio.v2.flows = [mock_flow]
mock_client.studio.v2.flows(mock_flow.sid).executions.list().return_value = [mock_execution]
You assign .flows to be a list, and then you try to call it like a function, which causes the error.
I think maybe you intended to say .flows[mock_flow.sid] instead of .flows(mock_flow.sid)?
Although even that doesn't make sense. .flows is a one-element list, so you would use .flows[0] to access the first (and only) item.
I'm seeing some unexpected behavior in my code related to python multiprocessing, and the Manager class in particular. I wrote out a super simple example to try and better understand what's going on:
import multiprocessing as mp
from collections import defaultdict
def process(d):
print('doing the process')
d['a'] = []
d['a'].append(1)
d['a'].append(2)
def main():
pool = mp.Pool(mp.cpu_count())
with mp.Manager() as manager:
d = manager.dict({'c': 2})
result = pool.apply_async(process, args=(d))
print(result.get())
pool.close()
pool.join()
print(d)
if __name__ == '__main__':
main()
This fails, and the stack trace printed from result.get() is as follows:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "<string>", line 2, in __iter__
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 825, in _callmethod
proxytype = self._manager._registry[token.typeid][-1]
AttributeError: 'NoneType' object has no attribute '_registry'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "mp_test.py", line 34, in <module>
main()
File "mp_test.py", line 25, in main
print(result.get())
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
AttributeError: 'NoneType' object has no attribute '_registry'
I'm still unclear on what's happening here. This seems to me to be a very, very straightforward application of the Manager class. It's nearly a copy of the actual example used in the official python documentation, with the only difference being that i'm using a pool and running the process with apply_async. I'm doing this because that's what i'm using in my actual project.
To clarify, I wouldn't get a stack trace if I didn't have the result = and print(result.get()) in there. I just see {'c': 2} printed when I run the script, which indicated to me that something was going wrong and wasn't being shown.
A couple things to start with: first, this isn't the code you ran. The code you posted has
result = pool.apply_async(process2, args=(d))
but there is no process2() defined. Assuming "process` was intended, the next thing is the
args=(d)
part. That's the same as typing
args=d
but that's not what's needed. You need to pass a sequence of the intended arguments. So you need to change that part to
args=(d,) # build a 1-tuple
or
args=[d] # build a list
Then the output changes, to
{'c': 2, 'a': []}
Why aren't 1 and 2 in the the 'a' list? Because it's only the dict itself that lives on the manager server.
d['a'].append(1)
first gets the mapping for 'a' from the server, which is an empty list. But that empty list is not shared in any way - it's local to process(). You append 1 to it, and then it's thrown away - the server knows nothing about it. Same thing for 2.
To get what you want, you need to "do something" to tell the manager server about what you changed; e.g.,
d['a'] = L = []
L.append(1)
L.append(2)
d['a'] = L
I'm trying to test a large legacy Django application, and I'm getting confused by Python mocking as I have never worked on large Python application.
Specifically, I have method has a long call chain inside that generates an array:
def update(self): # in some class X
# ...
for z in foo.models.Bar.objects.filter(x=1).select('xyz'):
raise Exception("mocked successfully")
I'd like to mock the foo.models.Bar.objects.filter(x=1).select('xyz').
Attempt 1
I've tried several approaches gleaned from various questions, notably using a decorator:
#mock.patch('foo.models.Bar.objects.filter.select')
def test_update(self, mock_select):
mock_select.return_value = [None]
X().update()
I never hit the inside of the mocked call, however- the test should fail due to the exception being raised.
Attempt 2
#mock.patch('foo.models.Bar')
def test_update(self, mock_Bar):
mock_Bar.objects.filter(x=1).select('xyz').return_value = [None]
X().update()
Attempt 3
#mock.patch('foo.models.Bar')
def test_update(self, mock_Bar):
mock_Bar.objects.filter().select().return_value = [None]
X().update()
Attempt 4
I then tried something more basic, to see if I could get an NPE, which didn't work either.
#mock.patch('foo.models.Bar')
def test_update(self, mock_Bar):
mock_Bar.return_value = None
X().update()
All of my attempts pass the test, instead of the exception firing like I expect it to.
It's late so I assume I must be overlooking something basic in the examples I've seen!?
I was able to have it pass by mocking objects. Attempt #3 is close, you just need to change it to filter.return_value.select.return_value to have it pass. Here's my suggestion though as it seems mocking .objects is the preferred way.
#mock.patch('foo.models.Bar.objects')
def test_update(self, mock_bar_objects):
mock_bar_objects.filter.return_value.select.return_value = [None]
X().update()
EDIT: Test run output:
ERROR: test_update (test_x.TestDjango)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/wholevinski/.virtualenvs/p2/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "/home/wholevinski/so_test/django_mock/test/test_x.py", line 10, in test_update
X().update()
File "/home/wholevinski/so_test/django_mock/foo/x_module.py", line 6, in update
raise Exception("mocked successfully")
Exception: mocked successfully
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (errors=1)
I am trying to build a Phasor Measurement Unit using Beaglebone Black rev 3. The following is the code. While running it gives an error as:
prussdrv_open open failed
Traceback (most recent call last):
File "/var/lib/cloud9/pmu.py", line 36, in <module>
pru.open(0) # open connection to PRU 0
SystemError: error return without exception set
The code goes here:
import pypruss as pru
import mmap
import numpy as np
import struct
import time
## MEMORY LOCATIONS ##
PRU_ICSS=0x4A300000
PRU_ICSS_LEN=512*1024
RAM_START=0x00000000
RAM1_START=0x00002000
RAM2_START=0x00012000
TOTAL_BUFFER_LEN=0x00000FA0
BUFFER_LEN=TOTAL_BUFFER_LEN/2
BUFFER1_START=RAM2_START+4
BUFFER2_START=BUFFER1_START+BUFFER_LEN
## FUNCTION DEFINITIONS ##
def processRawADC(value):
value=0x00000FFF&value
value=int(value)
value=(value*1.8)/(2^12)
return value
def channelID(value):
value=0x000F0000&value
value=value>>16
return value
## PRU SETUP ##
pru.modprobe( ) # enable uio_pruss module
pru.init( ) #initialize PRU
pru.open(0) # open connection to PRU 0
pru.pruintc_init( ) # configure interrupt handlers
pru.exec_program(0,"./oneshot.bin") # load assembly file
counter = 0
f=open("/dev/mem","r+b")
output=open("./results.txt","w")
while counter<10 :
start=time.time()
pru.wait_for_event(0)
ddr_mem=mmap.mmap(f.fileno( ),PRU_ICSS_LEN,offset=PRU_ICSS)
shared=struct.unpack('L ',ddr_mem[RAM2_START:RAM2_START+4])
print(shared[0])
if shared[0]==1 :
print ("buffer 1")
for i in range(0,500) :
fifo = struct.unpack ( 'L ' ,ddr_mem[BUFFER2_START+( i*4)
:BUFFER2_START+4+(i*4)])[0]
value=processRawADC(fifo)
channelNum=channelID(fifo)
output.write(str(channelNum)+","+str(value)+"nn")
counter += 1
pru.clear_event(0)
elif shared[0] == 2:
shared=struct.unpack('L ',ddr_mem[RAM2_START:RAM2_START+4])
print("buffer 2")
for i in range(0,500):
fifo=struct.unpack('L',ddr_mem[BUFFER2_START+(i*4) :BUFFER2_START+4+
(i*4)])[0]
value = processRawADC(fifo)
channelNum = channelID(fifo)
output.write(str(channelNum)+","+str(value)+"nn")
counter +=1
pru.clear_event(0)
end=time.time( )
#print end-start
f.close( )
output.close( )
pru.clear_event(0)
pru.pru_disable(0)
pru.exit ( )
I am unable to find, where is the mistake lies. Please Help.
Looks like there is a bug in PyPRUSS code.
Its pypruss_open function does not properly set exception information but returns an error indication (NULL). Python doesn't like when a function does so.
Looking at the pypruss_open source, it will fail in such way if prussdrv_open fails and returns -1 as an error indication. It in turn might fail either itself (if that device is already opened) or if __prussdrv_memmap_init fails.
Unfortunately, looks like there is no way to get information about the exact reason of the error.
What can you do to debug this issue? If you won't be able to find anything obvious (like missing /dev/uid0 after calling pru.modprobe()) then you can run your script with strace to see which system calls precede an error. Then you look at the source code under links I gave you above and see when exactly does the failure happen.
I am thinking to make a progress bar with python in terminal. First, I have to get the width(columns) of terminal window. In python 2.7, there is no standard library can do this on Windows. I know maybe I have to call Windows Console API manually.
According to MSDN and Python Documentation, I wrote the following code:
import ctypes
import ctypes.wintypes
class CONSOLE_SCREEN_BUFFER_INFO(ctypes.Structure):
_fields_ = [
('dwSize', ctypes.wintypes._COORD),
('dwCursorPosition', ctypes.wintypes._COORD),
('wAttributes', ctypes.c_ushort),
('srWindow', ctypes.wintypes._SMALL_RECT),
('dwMaximumWindowSize', ctypes.wintypes._COORD)
]
hstd = ctypes.windll.kernel32.GetStdHandle(ctypes.c_ulong(-11)) # STD_OUTPUT_HANDLE = -11
print hstd
csbi = CONSOLE_SCREEN_BUFFER_INFO()
print ctypes.sizeof(csbi) # <---------------
ret = ctypes.windll.kernel32.GetConsoleScreenBufferInfo(ctypes.c_ulong(hstd), csbi)
print ret
print csbi.dwSize.X
It works fine. I set about deleting some print in code. But after that, it doesn't work! GetLastError return 6 (Invalid Handle). After times of trying, I found that there must be SOMETHING at the pointed position of the code such as print 'hello', import sys or sys.stdout.flush(). At first, I guess that maybe it need time to do something. So I tried to put time.sleep(2) at that position, but it still doesn't work.
But, if I do use struct instead of ctypes.Structure, there's no such problem.
import ctypes
import struct
hstd = ctypes.windll.kernel32.GetStdHandle(-11) # STD_OUTPUT_HANDLE = -11
csbi = ctypes.create_string_buffer(22)
res = ctypes.windll.kernel32.GetConsoleScreenBufferInfo(hstd, csbi)
width, height, curx, cury, wattr, left, top, right, bottom, maxx, maxy = struct.unpack("hhhhHhhhhhh", csbi.raw)
print bufx
Is there any one can tell me why the irrelevant code made such a difference?
You need to pass the struct by reference:
ret = ctypes.windll.kernel32.GetConsoleScreenBufferInfo(
ctypes.c_ulong(hstd),
ctypes.byref(csbi)
)
I would also recommend that you declare the restype for GetStdHandle. That will mean that your code is ready to run under a 64 bit process. I'd write it like this:
ctypes.windll.kernel32.GetStdHandle.restype = ctypes.wintypes.HANDLE
hstd = ctypes.windll.kernel32.GetStdHandle(-11) # STD_OUTPUT_HANDLE = -11
csbi = CONSOLE_SCREEN_BUFFER_INFO()
ret = ctypes.windll.kernel32.GetConsoleScreenBufferInfo(
hstd,
ctypes.byref(csbi)
)
Actually, in my version of Python, your code reports a much more useful error. I see this:
Traceback (most recent call last):
File "test.py", line 16, in
ret = ctypes.windll.kernel32.GetConsoleScreenBufferInfo(ctypes.c_ulong(hstd), csbi)
ValueError: Procedure probably called with too many arguments (20 bytes in
excess)
This is enough to make it clear that there is an binary mismatch at the interface between the Python code and the native code.
I suspect that if you get a more recent version of Python, you'd also benefit from this stack imbalance checking.