We Keep Coding

Trying to replicate a sql statement in pyspark, getting column not iterable

Using Pyspark to transform data a DataFrame. The old extract used this SQL line :
case when location_type = 'SUPPLIER' then SUBSTRING(location_id,1,length(location_id)-3)
I brought in the data and loaded it into a DF, then was trying to do the transform using:
df = df.withColumn("location_id", F.when(df.location_type == "SUPPLIER",
F.substring(df.location_id, 1, length(df.location_id) - 3))
.otherwise(df.location_id))`
The substring method takes a int as the third argument but the length() method is giving a Column object. I had no luck trying to cast it and haven't found a method that would accept the Column. Also tried using the expr() wrapper but again could not make it work.
the supplier IDs look like 12345-01. The transform needs to strip the -01.

As you mention it, you can use expr to be able to use substring with indices that come from other columns like this:
df = df.withColumn("location_id",
F.when(df.location_type == "SUPPLIER",
F.expr("substring(location_id, 1, length(location_id) - 3)")
).otherwise(df.location_id)
)

python pandas is giving a keyerror for a column I group by, even though a boolean expression shows that the column is part of the dataframe

I cannot seem to print the following line: summarydata["Name"].groupby(["Tag"]).size()
without getting the error:
File "C:\Users\rspatel\untitled0.py", line 76, in <module>
print(summarydata["Name"].groupby(["Tag"]).size())
File "C:\Users\rspatel\Anaconda3\lib\site-packages\pandas\core\series.py", line 1720, in groupby
return SeriesGroupBy(
File "C:\Users\rspatel\Anaconda3\lib\site-packages\pandas\core\groupby\groupby.py", line 560, in __init__
grouper, exclusions, obj = get_grouper(
File "C:\Users\rspatel\Anaconda3\lib\site-packages\pandas\core\groupby\grouper.py", line 811, in get_grouper
raise KeyError(gpr)
KeyError: 'Tag'
I have checked that Tag is included as a column in the summarydata dataframe by the following:
if 'Tag' in summarydata.columns:
print("true")
else :
print("false")
which prints out as true. Therefore I am not sure why a key error is being thrown when the column is in the dataframe.

You are trying to group by a key on the column itself. Instead you want:
summarydata["name"].groupby(summarydata["Tag"])
from the docs:
by: (mapping, function, label, or list of labels)
Used to determine the groups for the groupby. If by is a function, it’s called on each value of the object’s index. If a dict or Series is passed, the Series or dict VALUES will be used to determine the groups (the Series’ values are first aligned; see .align() method). If an ndarray is passed, the values are used as-is to determine the groups. A label or list of labels may be passed to group by the columns in self. Notice that a tuple is interpreted as a (single) key.
In other words you can pass it anything ;) (This is why I don't like pandas...)
You can pass it:
a fn, which is called on every value
a dict(!) or series, which will be used to group (what you want)
a numpy array (ditto)
a label or list of labels, in which case it groups by the column in the object in question
But in your case, you've already selected the name column, so the Tag column no longer exists! (Think about what summarydata["name"] returns.)
So if you want to group like that, you need to group first:
summarydata.groupby("Tag")["name"]

'Series' object is not callable on a dataframe column with lists

I have a dataframe with many rows and columns.
One column - 'diagnostic superclass' has labels for every patient stored in rows as lists.
It looks like this:
['MI', 'HYP', 'STTC']
['MI', 'CD', 'STTC']
I need to obtain a first label from every row
The desired output is a column which stores every first list element of every row
so I wrote a function:
def labels(column_with_lists):
label = column_with_lists
for a in column_with_lists() :
list_label = column_with_lists[0]
label = list_label[0]
return label
So when I run the code I face the following problem:
Traceback (most recent call last):
File "C:/Users/nikit/PycharmProjects/ECG/ECG.py", line 77, in <module>
print(labels(y_true['diagnostic_superclass']))
File "C:/Users/nikit/PycharmProjects/ECG/ECG.py", line 63, in labels
for a in column_with_lists() :
TypeError: 'Series' object is not callable

The error is because of the () after the variable name, which means to to call it as a function. It's a pandas series, not a function.
One way to get a new series from the series of lists is with pandas.Series.apply()
def labels(column_with_lists):
return column_with_lists.apply(lambda x: x[0])

As #Vivek Kalyanarangan said, remove the parenthesis and it will work but I think that you are confuse, why you are iterate in this part if you dont use "a" for anything?
for a in column_with_lists :
list_label = column_with_lists[0]
label = list_label[0]
I think that you must storage the first item of each row in a list. In fact, you don't need to use a function:
first_element_of_each_row = [i[0] for i in y_true['diagnostic_superclass'].to_numpy()]
This should be work.

How to get the maximum number of digits after the decimal point in a Pandas series

I read a list of float values of varying precision from a csv file into a Pandas Series and need the number of digits after the decimal point. So, for 123.4567 I want to get 4.
I managed to get the number of digits for randomly generated numbers like this:
df = pd.Series(np.random.rand(100)*1000)
precision_digits = (df - df.astype(int)).astype(str).str.split(".", expand=True)[1].str.len().max()
However, if I read data from disk using pd.read_csv where some of the rows are empty (and thus filled with nan), I get the following error:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/home/tgamauf/workspace/mostly-sydan/venv/lib/python3.6/site-packages/pandas/core/generic.py", line 4376, in __getattr__
return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'str'
What is going wrong here?
Is there a better way to do what I need?

pd.read_csv() typically returns a DataFrame object. The StringMethods object returned by using .str is only defined for a Series object. Try using pd.read_csv('your_data.csv' , squeeze=True) to have it return a Series object; then you will be able to use .str

For example you have following data with NaN in it .
df=pd.Series([1.111,2.2,3.33333,np.nan])
idx=df.index# record the original index
df=df.dropna()# remove the NaN row
(df - df.astype(int)).astype(str).str.split(".", expand=True)[1].str.len().reindex(idx)

The version with df - df.astype(int) does not work correctly for me, simply applying the same str.split without it does:
def get_max_decimal_length(df):
"""Get the maximum length of the fractional part of the values or None if no values present."""
values = df.dropna()
return None if values.empty else values.astype(str).str.split(".", expand=True)[1].str.len().max()

Checking for the first value of data frame and assigning a value in Python

r is a dataframe with five columsn i,j,k,l,m
Below is my code,
for i in pd.unique(r.id):
sub=r[(r.id==i)] //subsetting the dataframe for each ID
sub=sub.drop_duplicates(["i","j","k","l","m"]) // dropping the duplicates
sub['k']=pd.to_datetime(sub['k'],unit='s',utc=False)
g=int(sub.iloc[0]['m']) // want to get the first value of the column
if g>64:
r=(g/64)-1
else:
r=0
if(len(sub)>1):
sub.m=r*64 + m
This works well for one ID. When there are multiple IDs, I am getting,
Traceback (most recent call last):
File "C:/project1/Final.py", line 90, in <module>
sub=r[(r.id==i)]
AttributeError: 'int' object has no attribute 'id'
Can anybody help me in solving this problem? I want to loop for all the IDs in r dataframe so that I can make some calculations.

Inside the for loop you overwrite the dataframe r with an int:
if g>64:
r=(g/64)-1
else:
r=0
This is fine with one ID because on the first iteration r is still a dataframe.
It fails with multiple IDs because on the second iteration it has been overwritten by the int, and throws the error.