I have a program- main.py that generates values for a few parameters depending on the user provided inputs. I need to share these variables with other modules- aux.py. To do that I did the following (simplified code below) -
This the main.py file:
# these three variables are dynamically calculated depending on the user provided input
a = 12
b = 13
c = 14
def declare_global():
global a, b, c
declare_global()
import aux
aux.print_all()
This is the aux.py file
def print_all():
a = globals()['a']
b = globals()['b']
c = globals()['c']
print(a)
print(b)
print(b)
Running the main.py file results in the following error
Traceback (most recent call last): File
"/Users/new/Library/Preferences/PyCharmCE2018.3/scratches/global_experiment/main.py",
line 13, in
aux.print_all() File "/Users/new/Library/Preferences/PyCharmCE2018.3/scratches/global_experiment/aux.py",
line 2, in print_all
a = globals()['a'] KeyError: 'a'
There is a very similar post addressing the same issue here but the posts there suggest adding all global variables to a new file and then importing the created file. But I cannot do that as I'll have no prior information as to what values these variables will take.
So, how do I share variables across modules?
EDIT
I lost some complexity of the actual program in order to make a minimalistic example. The reason I cannot pass the variables a,b,c to the print_all function is because in the actual program, the variables a,b,c are not serializable. They are spark dataframe objects. And one of the modules I use there (which here I am representing by print_all) serializes all its inputs I end up with an error (spark dataframes aren't serializable). Using a few workarounds I have arrived here where I use global variables and no inputs to the functions.
a = 12
b = 13
c = 14
import aux
aux.print_all(a, b, c)
Then your aux file will become:
def print_all(a, b, c):
a = globals()['a']
b = globals()['b']
c = globals()['c']
print(a)
print(b)
print(b)
Related
I have some predefined variables and some preexisted formulas(all multiplications). Formulas are calculating as expected besides when any of the variables are not defined.
Knowing that my formulas only handle multiplications I was thinking to give the value =1 to any undefined variable, in that way the calculation does break.
Is there any way to loop through the formulas, isolate the undefined variables and give them a value =1?
In the example below, ‘k’ is not defined and this calculation(kbd=kbd) is generating an error
#defined variables
a=10
b=5
c=3
d=6
e=7
f=2
g=9
#preexisted formulas
abc=a*b*c
fed=d*e*f
efg=e*f*g
kbd=k*b*d
NameError Traceback (most recent call last)
<ipython-input-10-f9e3059809fc> in <module>
----> 1 kbd=k*b*d
NameError: name 'k' is not defined
You can use something like this:
a=10
b=5
c=3
d=6
e=7
f=2
g=9
formulas = ['a*b*c','d*e*f','e*f*g','k*b*d']
for form in formulas:
variables = form.split('*')
for v in variables:
if v not in locals():
locals()[v]=1
eval(form)
Sympy is made for addressing your title 'How to manage undefined variables in a calculation'. The variables can be defined as Sympy symbols:
from sympy import *
from sympy.abc import a, b, c, d, e, f, g, k
# alternative of above line `a, b, c, d, e, f, g, k=symbols('a b c d e f g k')`
a=10
b=5
c=3
d=6
e=7
f=2
g=9
abc = a*b*c
fed = sympify(d*e*f) #you can convert an arbitrary expression to a type that can be used inside SymPy, based on https://docs.sympy.org/latest/modules/evalf.html; not really necessary for these multiplication examples
efg=e*f*g
kbd = k*b*d
print(N(abc)) #based on https://docs.sympy.org/latest/modules/evalf.html
print(abc)
print(fed) #based on https://docs.sympy.org/latest/modules/evalf.html
print(fed.evalf()) #based on https://docs.sympy.org/latest/modules/evalf.html
print(kbd)
print(simplify(kbd)) # based on https://www.tutorialspoint.com/sympy/sympy_simplification.html
kbd # - or `simplify(kbd)` - as last line in a Jupyter cell, you'll see fancy `30k` with `k` in italics in out cell.
That last line won't do anything in a script though, yet the output will look fancy in a Jupyter cell using the Python kernel where you've run %pip install sympy in the cell above. (You can try that in the cell of a Jupyter notebook here.)
Output from the print commands:
150.000000000000
84
84.0000000000000
30*k
30*k
Seen in 'Out' cell if run in Jupyter:
30𝑘
I'm trying to use MATLAB engine to call a MATLAB function in Python, but I'm having some problems. After manage to deal with NumPy arrays as input in the function, now I have some error from MATLAB:
MatlabExecutionError: Undefined function 'simple_test' for input
arguments of type 'int64'.
My Python code is:
import numpy as np
import matlab
import matlab.engine
eng = matlab.engine.start_matlab()
eng.cd()
Nn = 30
x= 250*np.ones((1,Nn))
y= 100*np.ones((1,Nn))
z = 32
xx = matlab.double(x.tolist())
yy = matlab.double(y.tolist())
Output = eng.simple_test(xx,yy,z,nargout=4)
A = np.array(Output[0]).astype(float)
B = np.array(Output[1]).astype(float)
C = np.array(Output[2]).astype(float)
D = np.array(Output[3]).astype(float)
and the Matlab function is:
function [A,B,C,D] = simple_test(x,y,z)
A = 3*x+2*y;
B = x*ones(length(x),length(x));
C = ones(z);
D = x*y';
end
Is a very simple example but I'm not able to run it!
I know the problem is in the z variable, because when I define z=32 the error is the one I mentioned, and when I change for z=32. the error changes to
MatlabExecutionError: Undefined function 'simple_test' for input
arguments of type 'double'.
but I don't know how to define z.
I am calling pdb on some function func i.e.,
def fun():
a = 10
c = fun2(a)
d = 40
return c+d
def fun2(a):
xyz ='str'
return a+10
Now I am running pdb using pdb.runcall(func,a,b) now It will open a pdb console for debugging, now suppose I press 2 time s(step) and q to quit in pdb console
but problem is I don't want to do this manually, I want to make some script which do something like this (automatic tell pdb first two command is s then third is q) , I am asking because there are many functions which needs atleast two time c (continue) to overall excecution of function so that it can yield/return some valid output (like say generators)
Any help will be a serious help for me.
Update after better understanding the question:
In general, I don't think this is the ideal way to test code; designing code for testability (e.g. using TDD) will often result in functions that are easier to test (e.g. using mocks/fake objects, dependency injection etc), and I would encourage you to consider refactoring the code if possible. The other issue with this approach is that the tests may become very tightly coupled to the code. However, I'll assume here that you know what you are doing, and that the above is not an option for whatever reason.
Scripting pdb
If you want to script pdb from code, this is actually possible by instantiating your own pdb.Pdb class and passing in the stdin and, at the time of writing, stdout argument (I'm not sure both should be required - I've filed https://bugs.python.org/issue33749).
Example (I just added the extra input argument to fun):
def fun(i):
a = 10 + i
c = fun2(a)
d = 40
return c+d
def fun2(a):
xyz ='str'
return a+10
import pdb
import io
output = io.StringIO()
# this contains the pdb commands we want to execute:
pdb_script = io.StringIO("p i;; i = 100;; n;; p a;; c;;")
mypdb = pdb.Pdb(stdin=pdb_script, stdout=output)
Normal result (no scripting):
In [40]: pdb.runcall(fun, 1)
...:
> <ipython-input-1-28966c4f6e38>(2)fun()
-> a = 10 + i
(Pdb)
(Pdb) c
Out[40]: 61
Scripted pdb:
In [44]: mypdb = pdb.Pdb(stdin=pdb_script, stdout=output)
In [45]: mypdb.runcall(fun, 1)
Out[45]: 160
In [50]: print(output.getvalue())
> <ipython-input-1-28966c4f6e38>(2)fun()
-> a = 10 + i
(Pdb) 1
> <ipython-input-1-28966c4f6e38>(3)fun()
-> c = fun2(a)
110
You may find using pdb_script.seek(0) helpful to reset the script.
Original answer - using conditional breakpoints
It sounds like what you really want is to only get into the debugger when your code is in a certain state. This can be done with conditional breakpoints (see pdb docs for details).
For example, let's say you want to break in fun2 if a > 10:
...:
In [2]: import pdb
In [3]: pdb.runcall(fun, 1)
> <ipython-input-1-28966c4f6e38>(2)fun()
-> a = 10 + i
(Pdb) break fun2, a > 10
Breakpoint 1 at <ipython-input-1-28966c4f6e38>:6
(Pdb) c
> <ipython-input-1-28966c4f6e38>(7)fun2()
-> xyz ='str'
(Pdb) c
Out[3]: 61
In [4]: pdb.runcall(fun, -1)
> <ipython-input-1-28966c4f6e38>(2)fun()
-> a = 10 + i
(Pdb) c
Out[4]: 59
Notice in the first case you hit the breakpoint, in the second you didn't.
Original answer - using breakpoints and executing commands when hit
You could also try setting a breakpoint and using the commands facility.
Lets say I have the following config file:
config.py:
x = 2
y = x * 2
I would like to import this in the file main.py, preferably using load_source command, but I also want to be able to change the value of x at the time of import such that the change in x propagates to the other variables in the config.py. For example, I want the following code, prints 6 and not 4.
main.py:
import imp
config = imp.load_source('', 'config.py')
config.x = 3
print config.y
What is the best way to do that? I know I can write functions in config.py to do this for me, but I prefer the config to be simple variable definitions only.
Put the code into a class:
class Config(object):
def __init__(self, x=2):
self.x = x
self.y = x * 2
Then, in your main program:
c = Config(3)
print c.y
i came across this as an interview question. This question seemed interesting. So, i am posting it here.
Consider the operation which gives semantic error like division by zero. By default, python compiler gives output like "Invalid Operation" or something. Can we control the output that is given out by Python compiler, like print some other error message, skip that division by zero operation, and carry on with rest of the instructions?
And also, how can i evaluate the cost of run-time semantic checks?
There are many python experts here. I am hoping someone will throw some light on this. Thanks in advance.
Can we control the output that is given out by Python compiler, like print some other error message, skip that division by zero operation, and carry on with rest of the instructions?
No, you cannot. You can manually wrap every dangerous command with a try...except block, but I'm assuming you're talking about an automatic recovery to specific lines within a try...except block, or even completely automatically.
By the time the error has fallen through such that sys.excepthook is called, or whatever outer scope if you catch it early, the inner scopes are gone. You can change line numbers with sys.settrace in CPython although that is only an implementation detail, but since the outer scopes are gone there is no reliable recorvery mechanism.
If you try to use the humorous goto April fools module (that uses the method I just described) to jump blocks even within a file:
from goto import goto, label
try:
1 / 0
label .foo
print("recovered")
except:
goto .foo
you get an error:
Traceback (most recent call last):
File "rcv.py", line 9, in <module>
goto .foo
File "rcv.py", line 9, in <module>
goto .foo
File "/home/joshua/src/goto-1.0/goto.py", line 272, in _trace
frame.f_lineno = targetLine
ValueError: can't jump into the middle of a block
so I'm pretty certain it's impossible.
And also, how can i evaluate the cost of run-time semantic checks?
I don't know what that is, but you're probably looking for a line_profiler:
import random
from line_profiler import LineProfiler
profiler = LineProfiler()
def profile(function):
profiler.add_function(function)
return function
#profile
def foo(a, b, c):
if not isinstance(a, int):
raise TypeError("Is this what you mean by a 'run-time semantic check'?")
d = b * c
d /= a
return d**a
profiler.enable()
for _ in range(10000):
try:
foo(random.choice([2, 4, 2, 5, 2, 3, "dsd"]), 4, 2)
except TypeError:
pass
profiler.print_stats()
output:
Timer unit: 1e-06 s
File: rcv.py
Function: foo at line 11
Total time: 0.095197 s
Line # Hits Time Per Hit % Time Line Contents
==============================================================
11 #profile
12 def foo(a, b, c):
13 10000 29767 3.0 31.3 if not isinstance(a, int):
14 1361 4891 3.6 5.1 raise TypeError("Is this what you mean by a 'run-time semantic check'?")
15
16 8639 20192 2.3 21.2 d = b * c
17 8639 20351 2.4 21.4 d /= a
18
19 8639 19996 2.3 21.0 return d**a
So the "run-time semantic check", in this case would be taking 36.4% of the time of running foo.
If you want to time specific blocks manually that are larger than you'd use timeit on but smaller than you'd want for a profiler, instead of using two time.time() calls (which is quite an inaccurate method) I suggest Steven D'Aprano's Stopwatch context manager.
I would just use an exception, this example is using python 3. For Python 2, simple remove the annotations after the function parameters. So you function signature would look like this -> f(a,b):
def f(a: int, b: int):
"""
#param a:
#param b:
"""
try:
c = a / b
print(c)
except ZeroDivisionError:
print("You idiot, you can't do that ! :P")
if __name__ == '__main__':
f(1, 0)
>>> from cheese import f
>>> f(0, 0)
You idiot, you can't do that ! :P
>>> f(0, 1)
0.0
>>> f(1, 0)
You idiot, you can't do that ! :P
>>> f(1, 1)
1.0
This is an example of how you could catch Zero Division, by making an exception case using ZeroDivisionError.
I won't go into any specific tools for making loggers, but you can indeed understand the costs associated with this kind of checking. You can put a start = time.time() at the start of the function and end = time.time() at the end. If you take the difference, you will get the execution time in seconds.
I hope that helps.