Edit: Really appreciate help in finding bug - but since it might prove hard to find/reproduce, any general debug help would be greatly appreciated too! Help me help myself! =)
Edit 2: Narrowing it down, commenting out code.
Edit 3: Seems lxml might not be the culprit, thanks! The full script is here. I need to go over it looking for references. What do they look like?
Edit 4: Actually, the scripts stops (goes 100%) in this, the parse_og part of it. So edit 3 is false - it must be lxml somehow.
Edit 5 MAJOR EDIT: As suggested by David Robinson and TankorSmash below, I've found a type of data content that will send lxml.etree.HTML( data ) in a wild loop. (I carelessly disregarded it, but find my sins redeemed as I've paid a price to the tune of an extra two days of debug! ;) A working crashing script is here. (Also opened a new question.)
Edit 6: Turns out this is a bug with lxml version 2.7.8 and below (at
least). Updated to lxml 2.9.0, and bug is gone. Thanks also to the fine folks over at this follow-up question.
I don't know how to debug this weird problem I'm having.
The below code runs fine for about five minutes, when the RAM is suddenly completely filled up (from 200MB to 1700MB during the 100% period - then when memory is full, it goes into blue wait state).
It's due to the code below, specifically the first two lines. That's for sure. But what is going on? What could possibly explain this behaviour?
def parse_og(self, data):
""" lxml parsing to the bone! """
try:
tree = etree.HTML( data ) # << break occurs on this line >>
m = tree.xpath("//meta[#property]")
#for i in m:
# y = i.attrib['property']
# x = i.attrib['content']
# # self.rj[y] = x # commented out in this example because code fails anyway
tree = ''
m = ''
x = ''
y = ''
i = ''
del tree
del m
del x
del y
del i
except Exception:
print 'lxml error: ', sys.exc_info()[1:3]
print len(data)
pass
You can try Low-level Python debugging with GDB. Probably there is a bug in Python interpreter or in lxml library and it is hard to find it without extra tools.
You can interrupt your script running under gdb when CPU usage goes to 100% and look at stack trace. It will probably help to understand what's going on inside script.
it must be due to some references which keep the documents alive. one must always be careful with string results from xpath evaluation. I see you have assigned None to tree and m but not to y,x and i .
Can you also assign None to y,x and i .
Tools are also helpful when trying to track down memory problems. I've found guppy to be a very useful Python memory profiling and exploration tool.
It is not the easiest to get started with due to a lack of good tutorials / documentation, but once you get to grips with it you will find it very useful. Features I make use of:
Remote memory profiling (via sockets)
Basic GUI for charting usage, optionally showing live data
Powerful, and consistent, interfaces for exploring data usage in a Python shell
Related
This is an old problem as is demonstrated as in https://community.intel.com/t5/Analyzers/Unable-to-view-source-code-when-analyzing-results/td-p/1153210. I have tried all the listed methods, none of them works, and I cannot find any more solutions on the internet. Basically vtune cannot find the custom python source file no matter what is tried. I am using the most recently version as of speaking. Please let me whether there is a solution.
For example, if you run the following program.
def myfunc(*args):
# Do a lot of things.
if __name__ = '__main__':
# Do something and call myfunc
Call this script main.py. Now use the newest vtune version (I have using Ubuntu 18.04), run the vtune-gui and basic hotspot analysis. You will not found any information on this file. However, a huge pile of information on Python and its other codes are found (related to your python environment). In theory, you should be able to find the source of main.py as well as cost on each line in that script. However, that is simply not happening.
Desired behavior: I would really like to find the source file and function in the top-down manual (or any really). Any advice is welcome.
VTune offer full support for profiling python code and the tool should be able to display the source code in your python file as you expected. Could you please check if the function you are expecting to see in the VTune results, ran long enough?
Just to confirm that everything is working fine, I wrote a matrix multiplication code as shown below (don't worry about the accuracy of the code itself):
def matrix_mul(X, Y):
result_matrix = [ [ 1 for i in range(len(X)) ] for j in range(len(Y[0])) ]
# iterate through rows of X
for i in range(len(X)):
# iterate through columns of Y
for j in range(len(Y[0])):
# iterate through rows of Y
for k in range(len(Y)):
result_matrix[i][j] += X[i][k] * Y[k][j]
return result_matrix
Then I called this function (matrix_mul) on my Ubuntu machine with large enough matrices so that the overall execution time was in the order of few seconds.
I used the below command to start profiling (you can also see the VTune version I used):
/opt/intel/oneapi/vtune/2021.1.1/bin64/vtune -collect hotspots -knob enable-stack-collection=true -data-limit=500 -ring-buffer=10 -app-working-dir /usr/bin -- python3 /home/johnypau/MyIntel/temp/Python_matrix_mul/mat_mul_method.py
Now open the VTune results in the GUI and under the bottom-up tab, order by "Module / Function / Call-stack" (or whatever preferred grouping is).
You should be able to see the the module (mat_mul_method.py in my case) and the function "matrix_mul". If you double click, VTune should be able to load the sources too.
Basing my code on this allegedly Python-specific documentation example, I have:
def on_output(spin):
adj = spin.get_adjustment()
val = int(adj.get_value())
s = "%02d" % val
print "on_output: %s" % s
spin.set_text(s)
which I connect to my SpinButton's "output" signal. It seems to work when the control is first displayed (shows "00"), but when I click SpinButton's increment button, the formatted value from on_output is overwritten, so e.g. my "01" is shown as plain "1". Looks like another signal or event is causing the control to reformat itself after on_output, but I'm struggling a bit to diagnose. Any experts on GTK3 with Python, please help with debugging suggestions.
Platform is Xubuntu 18.10, Python 2.7, GTK3 3.22.
Wow! If ever there was a case for responding with 'RTFM!' this was it. As very politely pointed out by Alexander Dmitriev, the 'return' statement is missing. In my first attempt, I returned True, but it failed (for some presumably unrelated reason) so I tried False which made no difference. Somehow, the return value got lost after that, and when I re-read the doco 'carefully' I missed seeing it -- we see what we expect to see! I must be getting too old for this hacking game, maybe time to hang up my keyboard. :)
Morning folks,
I'm trying to get a few unit tests going in Python to confirm my code is working, but I'm having a real hard time getting a Mock anything to fit into my test cases. I'm new to Python unit testing, so this has been a trying week thus far.
The summary of the program is I'm attempting to do serial control of a commercial monitor I got my hands on and I thought I'd use it as a chance to finally use Python for something rather than just falling back on one of the other languages I know. I've got pyserial going, but before I start shoving a ton of commands out to the TV I'd like to learn the unittest part so I can write for my expected outputs and inputs.
I've tried using a library called dummyserial, but it didn't seem to be recognising the output I was sending. I thought I'd give mock_open a try as I've seen it works like a standard IO as well, but it just isn't picking up on the calls either. Samples of the code involved:
def testSendCmd(self):
powerCheck = '{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK']).encode('utf-8')
read_text = 'Stuff\r'
mo = mock_open(read_data=read_text)
mo.in_waiting = len(read_text)
with patch('__main__.open', mo):
with open('./serial', 'a+b') as com:
tv = SharpTV(com=com, TVID=999, tvInput = 'DVI')
tv.sendCmd(SharpCodes['POWER'], SharpCodes['CHECK'])
com.write(b'some junk')
print(mo.mock_calls)
mo().write.assert_called_with('{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK']).encode('utf-8'))
And in the SharpTV class, the function in question:
def sendCmd(self, type, msg):
sent = self.com.write('{0}{1:>4}\r'.format(type,msg).encode('utf-8'))
print('{0}{1:>4}\r'.format(type,msg).encode('utf-8'))
Obviously, I'm attempting to control a Sharp TV. I know the commands are correct, that isn't the issue. The issue is just the testing. According to documentation on the mock_open page, calling mo.mock_calls should return some data that a call was made, but I'm getting just an empty set of []'s even in spite of the blatantly wrong com.write(b'some junk'), and mo().write.assert_called_with(...) is returning with an assert error because it isn't detecting the write from within sendCmd. What's really bothering me is I can do the examples from the mock_open section in interactive mode and it works as expected.
I'm missing something, I just don't know what. I'd like help getting either dummyserial working, or mock_open.
To answer one part of my question, I figured out the functionality of dummyserial. The following works now:
def testSendCmd(self):
powerCheck = '{0}{1:>4}\r'.format(SharpCodes['POWER'], SharpCodes['CHECK'])
com = dummyserial.Serial(
port='COM1',
baudrate=9600,
ds_responses={powerCheck : powerCheck}
)
tv = SharpTV(com=com, TVID=999, tvInput = 'DVI')
tv.sendCmd(SharpCodes['POWER'], SharpCodes['CHECK'])
self.assertEqual(tv.recv(), powerCheck)
Previously I was encoding the dictionary values as utf-8. The dummyserial library decodes whatever you write(...) to it so it's a straight string vs. string comparison. It also encodes whatever you're read()ing as latin1 on the way back out.
I have the following code in a function:
phi = fourier_matrix(y, fs)
N = np.size(phi,axis=1)
x = np.ones(N)
for i in range(136):
W_pk = np.diag(x)
temp = pinv(np.dot(phi, W_pk))
q_k = np.dot(temp, y)
x = np.dot(W_pk, q_k)
where phi is (96,750), W_pk is (750,750) and q_k is (750,).
This throws the following error:
Python(12001,0x7fff99d8b380) malloc: * error for object 0x7fc71aa37200: incorrect checksum for freed object - object was probably modified after being freed.
* set a breakpoint in malloc_error_break to debug
If I comment the last dot product the error does not appear.
I think I need to free memory in some way or maybe do the dot product in a different way?
Also, this only happens when I run it from a mac. On windows or linux it does not throw the error.
Python is 3.6 (tried with 3.7), and numpy is 1.14.5, also tried with 1.15
Any help would be greatly appreciated, since I really need to make this work!
Thanks in advance.
EDIT I:
I tried this portion of the code on a jupyter notebook, and it didn't fail. This confused me even more! It fails when I run it in Visual Studio Code on a mac. The rest of my code, an algorithm to remove artifacts from a signal, works as it should until I add that last piece of code x = np.dot(W_pk, q_k). Maybe it works on jupyter because I don't run the rest of the algorithm there? but as I said, it only crashes on that last dot product.
EDIT II: I added the piece of code above the for loop to this question, because I found that the problem is somehow related to how x is being used. You see it's declared above as a float64 ndarray. When it reaches the last line of the for loop, the dot product returns a complex128 (should be complex64, don't know what's happening there) and overwrites the x array. The first time works, but the second time it crashes when trying to overwrite. If I use a new variable for the dot product, say z, then it does not crash! not sure why... but I need to overwrite x in each iteration.
Furthermore, if I do something like this:
z = np.dot(W_pk, q_k)
x = abs(z) #I don't need complex numbers at this point
Then it crashes with the same error on the first dot product (presumably):
temp = pinv(np.dot(phi, W_pk))
Also, the memory consumption is not that bad, around 110M according some measurements, and the same algorithm does not crash on iPython with twice the memory usage. This is what I find the most obscure, why doesn't it crash on iPython??
This is a real project written in VBA before.
I want to move it to Python and use 'ActiveX Automation scripts for AutoCAD with Python' method. This is my code:
# -*- coding: utf-8 -*-
from pyautocad import Autocad, APoint, aDouble
acad = Autocad(False, True)
acad.prompt("Hello, Autocad from Python\n")
print acad.doc.Name
xx = acad.model.AddCircle(APoint(0, 0), 10)
print(xx)
yy = acad.model.Add3Dpoly(aDouble([0, 0, 0, 10, 10, 10, 30, 20, 30, 0, 0, 0]))
print(yy.ObjectName)
print(yy.PlotStyleName)
# How to contruct an objectlist for AddRegion?
#regions = acad.model.AddRegion([yy])
#acad.model.AddExtrudedSolid(regions[0], 20, 0)
My question is, how to construct an object list for AddRegion? Maybe comtypes have some topic about VARINT. I really have no experience about COM and so on...
Getting it all to work right can be more work than it seems like it should be. Reading data using python; not so bad. Writing data, bit trickier. Casual user / Beginners; be warned of what your getting into.
What you will probably need
It helps tremendously if you are familiar with autolisp, because it just works better (in this case), is documented better, and integrates better,... and you will likely need it to milk 'unknown/hidden/undocumented' information that python isn't telling you. (see the vlax- and vla- series of lisp functions).
Next you need win32com make_py and gen_py scripts from the command line, or you can use win32com.client.gencode while staying mostly in python.
Be prepared to visually parse very ugly text (and I wasn't even talking about lisp =] ). Be prepared to fail, and get excited to find out why.
Most of it has to do with COM-Variants. And then you get wierd stuff like Variant-Variant-Arrays. If you check out win32com.client.pythoncom, you will notice all of the datatypes are mapped to integers. (VT_BOOL is 11 for example).
The Nitty Gritty
Next time you attempt a ModelSpace.AddCircle, pay attention to the debug output you get;
All of the parameters passed to InvokeTypes are what you need to watch for... (this is taken from my make-py output for the Autocad Registered Interfaces
def AddLine(self, StartPoint=defaultNamedNotOptArg, EndPoint=defaultNamedNotOptArg):
ret = self._oleobj_.InvokeTypes(
1581, LCID, 1, (9, 0), ((12, 1), (12, 1)),StartPoint, EndPoint)
if ret is not None:
ret = Dispatch(ret, u'AddLine', '{DF524ECB-D59E-464B-89B6-D32822282778}'
This tells you exactly which COM types win32com THINKS it wants, so make sure you are at least matching that.
I have found that many input functions are actually documented and invoked wrong (I learned this after much back and forth with AutoLisp). What we see above has a value of 1581 on the outside (which is something like a class name, not really a datatype), and then tuple that basically says (DISPATCH, EMPTY):(9,0), and then an array of VT_VARIANTS:((12,1),(12,1)).
There is usually a missing outer wrapper that COM was expecting, and for some reason make-py does not realize this. if you go through extensive AutoLisp vlax- nonsense, you will notice their is an additional wrapper around that one. I believe it is either a VARIANT_ARRAY, or quite literally, a VARIANT-VARIANT-ARRAY (quadruple pointer or something). The codes for this are (vt_array=8192, vt_variant=12).
Im sorry I don't remember specifics, but I believe the portion reading ((12,1),(12,1)), should become (8192, 12, ((12,1),(12,1))), or something like that. Even once you do figure out what it should be, Im not sure if their is a quick fix. As of AutoCAD 2010, for me, this meant going through the ungodly large gen_py output, finding the functions I really wanted, and manually changing the InvokeTypes() call to match what COM was expecting.
Everything worked simply as expected after that.
Possible Workarounds
COM is ugly. If you are new to python, but semi-experienced in AutoCAD (meaning you want to do some fairly hefty automation), stay away from the python->win32com->AutoCAD pipeline. Use LISP. As much as that pains me to say, your gonna end up writing so many LISP test cases and debuggers to accompany your python pains, you might as well just commit.
Ironpython and .NET
I believe this interface is much more supported than COM in general
Visual Studio Professional (2008+)
I never used official VS-Pro tools (I used PythonWIN and MINGW), Im not sure if there is any extra magic provided that would change how win32com handles AutoCAD. I know the official AutoCAD ARX extensions provide their source in a Studio project. Worst case you would have actual documentation close at hand, which is where this entire topic of python-AutoCAD becomes tainted.