We Keep Coding

Is there an easy way to delete most variables but keep the loaded data for ongoing code?

The situation is, there is a for-loop which runs 100 times, sharing a same loaded data. As each single loop goes, it will create a growing number of variables which occupies a lot of memory resource and then further slows down the running speed. The code can be abstractly written as
data = sio.loadmat('very_big_data_set.mat')
for t in range(100):
config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.Session(graph=tf.get_default_graph(), config=config)
var_1 = ...
var_2 = ...
var_10000 = ...
model = ...
sess.close()
# The line below is a solution but it's very inconvinient
del config, sess, var_1, ..., var_10000, model
My question is if there is a python command I can use to delete all of created variables during the for-loop but keep the loaded data for next loop? I know the method del config, sess, var_1, ..., var_10000, model but it requires me to list all of variables I want to delte. So I am looking for a more easy way, kind of like del all except data.

Generally, if you want to use variables that need to be cleaned up after you're done with it, especially if you're using them repeatedly, writing a function is a good idea:
def do_something(arg):
# arg gets the same value as x (or points to the same thing)
y = 10
# now y exists, we can do do something with y and arg
r = y + arg
# now r, y and arg exist
return r
# once the function returns, arg no longer exists or references x
# y and r are no longer referenced either and will no longer exist
for x in range(10)
print(do_something(x))
# here, y and r don't exist anymore, because the function has ended
# here, y and r don't exist, but x still does, since a loop variable sticks around
# capture all this in a function and call it and x would be cleaned up as well

What you need is reduced scope for your variable. Actually, your variables is global, which made the data present in memory until the program ends.
Here is an example of your needs:
def data_valorisation(config, sess):
# All theses variable will be deleted and recreated each time you call the function
var_1 = ...
var_2 = ...
[...]
var_10000 = ...
model = ...
data = sio.loadmat('very_big_data_set.mat')
for t in range(100):
config = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)
sess = tf.Session(graph=tf.get_default_graph(), config=config)
data_valorisation(config, sess)
sess.close()
You should always put computation into function and avoid doing it in a global context. For this, you can use function or classes.
Variables and scope

how to make a clause in pyswip

can any one here help me to make prolog clauses inside python using pyswip like this
database:
man(peter).
woman(adam).
man(jonathan).
man(paul).
woman(cloe).
father(jonathan, peter).
father(pierre, adam).
brother(pierre, paul).
father(pierre, cloe).
these are functions
child(X, Y) :- father(Y,X).
son(X, Y) :- man(X) , father(Y, X).
daughter(X, Y) :- woman(X), father(Y, X).
brother(X, Y) :- man(X), father(Z, Y), father(Z, X).
sister(X, Y) :- woman(X), father(Z, Y), father(Z, X).
how can i define these prolog functions inside python through pyswip

I don't have time to make a detailed answer right now, I'll update it latter but here's a simple example of an interface in python I had done to a prolog program that played reversi.
#!/usr/bin/python
import sys
from pyswip import Prolog, Functor, Variable, Query
prolog = Prolog()
prolog.consult('./reversi_game.pl')
prolog.consult('./alphabeta.pl')
start_board = Functor("startBoard", 1)
b = Variable()
start_board_query = Query(start_board(b))
start_board_query.nextSolution()
print()
print_board(list(b.get_value())) # an 8*8 grid filled with 0 except at the 4 center squares that have x's and o's
start_board_query.closeQuery()
set_to_x = Functor("setToX", 1)
xp = Variable()
set_player_query = Query(set_to_x(xp))
set_player_query.nextSolution()
x_player = xp.get_value()
print()
print(x_player) # 'x'
set_player_query.closeQuery()
So what it is going on here? To define the predicate interface you create a Functor, giving it a string which is the name of the predicate in Prolog and its arity, you create as many variables as you need and pass them to your Functor creating a Query from it.
You can then proceed calling nextSolution() on the Query object for as long as you want depending on how many solutions you need, if I remember correctly the result will be None when it fails and it stops giving any solutions. Then use the get_value() function to extract the values of the variables of your predicate.
You can also check this out:
https://github.com/yuce/pyswip/tree/master/examples
Hope it helps.
EDITED:
I know this "promised a bit more detailed answer" is a bit late now, but anyways.

test getting skipped in pytest

I am trying to use parametrize for which I want to give testcases which I get from a different function using pytest.
I have tried this
test_input = []
rarp_input1 = ""
rarp_output1 = ""
count =1
def test_first_rarp():
global test_input
config = ConfigParser.ConfigParser()
config.read(sys.argv[2])
global rarp_input1
global rarp_output1
rarp_input1 = config.get('rarp', 'rarp_input1')
rarp_input1 =dpkt.ethernet.Ethernet(rarp_input1)
rarp_input2 = config.get('rarp','rarp_input2')
rarp_output1 = config.getint('rarp','rarp_output1')
rarp_output2 = config.get('rarp','rarp_output2')
dict_input = []
dict_input.append(rarp_input1)
dict_output = []
dict_output.append(rarp_output1)
global count
test_input.append((dict_input[0],count,dict_output[0]))
#assert test_input == [something something,someInt]
#pytest.mark.parametrize("test_input1,test_input2,expected1",test_input)
def test_mod_rarp(test_input1,test_input2,expected1):
global test_input
assert mod_rarp(test_input1,test_input2) == expected1
But the second test case is getting skipped. It says
test_mod_rarp1.py::test_mod_rarp[test_input10-test_input20-expected10]
Why is the test case getting skipped? I have checked that neither the function nor the input is wrong. Because the following code is working fine
#pytest.mark.parametrize("test_input1,test_input2,expected1,[something something,someInt,someInt])
def test_mod_rarp(test_input1,test_input2,expected1):
assert mod_rarp(test_input1,test_input2) == expected1
I have not put actual inputs here. Its correct anyway. Also I have config file from which I am taking inputs using configParser. test_mod_rarp1.py is the python file name where I am doing this. I basically want to know if we can access variables(test_input in my example) from other functions to use in parametrize if that is causing problem here. If we can't how do I change the scope of the variable?

Parametrization happens at compile time so that is the reason if you want to parametrized on data generated at run time it skips that.
The ideal way to acheive what you are trying to do is by using fixture parametrization.
Below example should clear things for you and then you could apply the same logic in your case
import pytest
input = []
def generate_input():
global input
input = [10,20,30]
#pytest.mark.parametrize("a", input)
def test_1(a):
assert a < 25
def generate_input2():
return [10, 20, 30]
#pytest.fixture(params=generate_input2())
def a(request):
return request.param
def test_2(a):
assert a < 25
OP
<SKIPPED:>pytest_suites/test_sample.py::test_1[a0]
********** test_2[10] **********
<EXECUTING:>pytest_suites/test_sample.py::test_2[10]
Collected Tests
TEST::pytest_suites/test_sample.py::test_1[a0]
TEST::pytest_suites/test_sample.py::test_2[10]
TEST::pytest_suites/test_sample.py::test_2[20]
TEST::pytest_suites/test_sample.py::test_2[30]
See test_1 was skipped because parameterization happened before execution of generate_input() but test_2 gets parameterized as required

How use the newton function for root finding of the Scipy's optimize package

I want to use the newton function loaded as
from scipy.optimize import newton
in order to find the zeros of a function enetered by the user. I write a script that first ask to the user to specify a function together with its first derivative, and also the starting point of the algorithm. First of all typing help(newton) I saw which parameters takes the function and the relative explanation:
newton(func, x0, fprime=None, args=(), tol=1.48e-08, maxiter=50)
func : function
The function whose zero is wanted. It must be a function of a
single variable of the form f(x,a,b,c...), where a,b,c... are extra
arguments that can be passed in the `args` parameter.
In which way I have to pass my function? If I use for func e.g. x**3 (and its first derivative) the response is NameError: name 'x' is not defined. On the internet I find that first I have to define my function and its first derivative and pass the names as parameters. So I made the following
fie = raw_input('Enter function in terms of x (e.g. x**2 - 2*x). F= ')
dfie = raw_input('Enter first derivative of function above DF = ')
x0 = input('Enter starting point x0 = ')
def F(x,fie):
y = eval(fie)
return y
def DF(x, dfie):
dy = eval(dfie)
return dy
print newton(F,x0,DF)
But I get the output
102 for iter in range(maxiter):
103 myargs = (p0,) + args
--> 104 fder = fprime(*myargs)
105 if fder == 0:
106 msg = "derivative was zero."
TypeError: DF() takes exactly 2 arguments (1 given)
and the same issue for F if I omit DF. Looking at the code in /usr/local/share/src/scipy/scipy/optimize/zeros.py I see that it evaluates the first derivative with fder=fprime(*myargs) so maybe I have to put in args something that make it working. I was thinking about it but no solution comes to me.

First, be aware that using eval makes your program vulnerable to malicious users. If that concern does not apply, you can create F and DF like this:
F = eval('lambda x :'+fie)
DF = eval('lambda x :'+dfie)
Then both functions expect only a single argument, and you can leave the args argument empty.
EDIT. If you really want to stick to your code as closely as possible, this should also work, but it does not look very nice to me. newton will send the same args to both functions.
def F(x,fie,dfie):
y = eval(fie)
return y
def DF(x,fie,dfie):
dy = eval(dfie)
return dy
print newton(F,x0,DF,(fie,dfie))

homogenization the functions can be compiled into a calculate networks?

Inside of a network, information (package) can be passed to different node(hosts), by modify it's content it can carry different meaning. The final package depends on hosts input via it's given route of network.
Now I want to implement a calculating network model can do small jobs by give different calculate path.
Prototype:
def a(p): return p + 1
def b(p): return p + 2
def c(p): return p + 3
def d(p): return p + 4
def e(p): return p + 5
def link(p, r):
p1 = p
for x in r:
p1 = x(p1)
return p1
p = 100
route = [a,c,d]
result = link(p,result)
#========
target_result = 108
if result = target_result:
# route is OK
I think finally I need something like this:
p with [init_payload, expected_target, passed_path, actual_calculated_result]
|
\/
[CHAOS of possible of functions networks]
|
\/
px [a,a,b,c,e] # ok this path is ok and match the target
Here is my questions hope may get your help:
can p carry(determin) the route(s) by inspect the function and estmated result？
(1.1 ) for example, if on the route there's a node x()
def x(p): return x / 0 # I suppose it can pass the compile
can p know in somehow this path is not good then avoid select this path?
(1.2) Another confuse is if p is a self-defined class type, the payload inside of this class essentially is a string, when it carry with a path [a,c,d], can p know a() must with a int type then avoid to select this node?'
same as 1.2 when generating the path, can I avoid such oops
def a(p): return p + 1
def b(p): return p + 2
def x(p): return p.append(1)
def y(p): return p.append(2)
full_node_list = [a,b,x,y]
path = random(2,full_node_list) # oops x,y will be trouble for the inttype P and a,b will be trouble to list type.
pls consider if the path is lambda list of functions
PS: as the whole model is not very clear in my mind the any leading and directing will be appreciated.
THANKS!

You could test each function first with a set of sample data; any function which returns consistently unusable values might then be discarded.
def isGoodFn(f):
testData = [1,2,3,8,38,73,159] # random test input
goodEnough = 0.8 * len(testData) # need 80% pass rate
try:
good = 0
for i in testData:
if type(f(i)) is int:
good += 1
return good >= goodEnough
except:
return False
If you know nothing about what the functions do, you will have to essentially do a full breadth-first tree search with error-checking at each node to discard bad results. If you have more than a few functions this will get very large very quickly. If you can guarantee some of the functions' behavior, you might be able to greatly reduce the search space - but this would be domain-specific, requiring more exact knowledge of the problem.
If you had a heuristic measure for how far each result is from your desired result, you could do a directed search to find good answers much more quickly - but such a heuristic would depend on knowing the overall form of the functions (a distance heuristic for multiplicative functions would be very different than one for additive functions, etc).

Your functions can raise TypeError if they are not satisfied with the data types they receive. You can then catch this exception and see whether you are passing an appropriate type. You can also catch any other exception type. But trying to call the functions and catching the exceptions can be quite slow.
You could also organize your functions into different sets depending on the argument type.
functions = { list : [some functions taking a list], int : [some functions taking an int]}
...
x = choose_function(functions[type(p)])
p = x(p)

I'm somewhat confused as to what you're trying to do, but: p cannot "know about" the functions until it is run through them. By design, Python functions don't specify what type of data they operate on: e.g. a*5 is valid whether a is a string, a list, an integer or a float.
If there are some functions that might not be able to operate on p, then you could catch exceptions, for example in your link function:
def link(p, r):
try:
for x in r:
p = x(p)
except ZeroDivisionError, AttributeError: # List whatever errors you want to catch
return None
return p