We Keep Coding

PySpark Foreach method TypeError: cannot pickle '_thread.RLock' object

I am trying to use foreach method on my dataframe but I keep getting this error:
PicklingError: Could not serialize object: TypeError: cannot pickle '_thread.RLock' object
My code looks similar to this:
def test_func(x, df1):
print(x)
return df1
def myfunc(df,df1):
res = df.foreach(partial(test_func, df1))
I am literally doing nothing in the test_func function, but I still keep getting this error.
Does anyone know what the cause might be?
thank you!
UPDATE: In my case, df and df1 are two dataframes. I realized that if I pass a variable into the partial function, for example partial(test_func, 2) I do not get this error. However, if I pass normally a dataframe, so partial(test_func, df1), then I get this error. How can I use foreach to iterate over a dataframe and at the same time edit another dataframe?

can someone explain why i'm getting an error on a sympy.Set()

works:
import sympy as sp
import pandas as pd
test = pd.DataFrame( {'greeting':['hey','hi','hello','howdy']} )
test1['greeting'].map(lambda x: sp.Set(*x).is_proper_subset( sp.Set(*x) ))
doesn't work:
import sympy as sp
import pandas as pd
test = pd.DataFrame( {'greeting':['hey','hi','hello','howdy']} )
test1['greeting'].map(lambda x: sp.Set('hi',).is_proper_subset( sp.Set(*x) ))```
i get:
AttributeError: 'str' object has no attribute 'args'
i've also tried with numbers on another dataframe:
test['triplets'].map(lambda x: sp.Set(*(813,)).is_proper_subset( sp.Set(*x) ))
and i get the same result.
https://docs.sympy.org/latest/modules/sets.html
class sympy.sets.sets.Set(*args)[source]¶
The base class for any kind of set.
ok...why is it working for each Series value when i pass it through lambda, but not when i write it manually
note: my end goal is to have this inside a for loop where the the iteration is passed into where i have 'hi'

From sympy's Set documentation:
class sympy.sets.sets.Set(*args)[source]¶
This is not meant to be used directly as a container of items. It does not behave like the builtin set; see FiniteSet for that.
The main problem is that is_proper_subset is meant for Interval type of "sets". It doesn't seem to handle simpler sets well, but gives a rather cryptic error message. When debugging, it often helps to reduce the problem as much as possible:
import sympy as sp
sp.Set(1, 2).is_proper_subset(sp.Set(1, 2, 3))
This raises the error AttributeError: 'int' object has no attribute 'args'.
Similarly:
sp.Set('a', 'b').is_proper_subset(sp.Set('a', 'b', 'c'))
Leads to AttributeError: 'str' object has no attribute 'args'.
The best solution to the original problem, is to use standard python functions.

i don't know what the deal is, i'm hoping there's an answer - but until then, i've resorted to this:
test1['greeting'].map(lambda x: set('hi').issubset(set(x)) and set('hi')!=set(x))

'dict' object is not callable error - when I'm not using any 'dict's in networkx

I'm using Python to work with networkx and draw some graphs.
I ran into a problem raising:
TypeError: 'dict' object is not callable
on this line of code:
set_node_color(num, list(Graph.node()))
I searched to find that this error is raised when I'm using a variable name dict.
The problem is, I'm not using any variables with the name dict, nor am I using any dictionary types anywhere in the code.
In case it's necessary, printing the type of Graph gives <class 'networkx.classes.digraph.Digraph'>.
I also tried printing the type for Graph.node() only to receive the same error, telling me 'dict' object is not callable.
So I suspect Graph.node() to be a dict type variable, but using (Graph.node()).items() raises the same TypeError.
Any help or advices would be nice. Thanks.

Maybe Graph.node is a dict object, so Graph.node() is not callable.

Why I encounter error: AttributeError: 'int' object has no attribute 'type'

I am using llvmlite for my project in combination with Pyvex. I have defined some functions in llvmlite like the following:
def int32(val):
return ir.Constant(ir.IntType(32), val)
def put64(putoffset, val):
llvmtmp = builder.gep(regtag, (int32(0), int32(putoffset)), True)
return builder.store(val, llvmtmp)
However, when I want to call this function using the following code:
for stmt in irsb.statements:
if isinstance(stmt, pyvex.IRStmt.Put):
putoffset = stmt.offset
put64("t3", putoffset)
I encounter the error: AttributeError: 'int' object has no attribute 'type'
does anyone know how can I resolve this problem?

I did a little digging. The problem is happening on this line
return ir.Constant(ir.IntType(32), val) in your int32() function.
Constant is defined as such in llvmlite:
Constant(typ, constant)
typ is the type of the represented value
(a :class:~llvmlite.ir.Type instance). constant is the Python
value to be represented. Which Python types are allowed for constant
Which Python types are allowed for constant depends on typ.
Instead of passing in Python types you just pass in int and string as val. try put type() around it.
return ir.Constant(ir.IntType(32), type(val))
Hope this helps

I understand the problem. Actually the problem is that we are passing a string to the function but it needs to be an llvmlite object. Hence python complains that strings don't have an attribute type. Passing llvmlite object, the problem would be solved.

AttributeError: 'tuple' object has no attribute 'shape'

So I have been writing a code to standardize the elements of a matrix and the function I used is as follows:
def preprocess(Data):
if stdn ==True:
st=np.empty((Data.shape[0],Data.shape[1]))
for i in xrange(0,Data.shape[0]):
st[i,0]=Data[i,0]
for i in xrange(1,Data.shape[1]):
st[:,i]=((Data[:,i]-np.min(Data[:,i]))/(np.ptp(Data[:,i])))
np.random.shuffle(st)
return st
else:
return Data
It works very well outside the class but when used inside of it it gives me this error:
AttributeError: 'tuple' object has no attribute 'shape'
Any idea on how I can fix it??
P.S. This is a KNN classification code

According to the error you posted, Data is of type tuple and there is no attribute shape defined for data. You could try casting Data when you call your preprocess function, e.g.:
preprocess(numpy.array(Data))