We Keep Coding

Add character to column based on text condition using pandas

I'm trying to do some data cleaning using pandas. Imagine I have a data frame which has a column call "Number" and contains data like: "1203.10", "4221","3452.11", etc. I want to add an "M" before the numbers, which have a point and a zero at the end. For this example, it would be turning the "1203.10" into "M1203.10".
I know how to obtain a data frame containing the numbers with a point and ending with zero.
Suppose the data frame is call "df".
pointzero = '[0-9]+[.][0-9]+[0]$'
pz = df[df.Number.str.match(pointzero)]
But I'm not sure on how to add the "M" at the beginning after having "pz". The only way I know is using a for loop, but I think there is a better way. Any suggestions would be great!

You can use boolean indexing:
pointzero = '[0-9]+[.][0-9]+[0]$'
m = df.Number.str.match(pointzero)
df.loc[m, 'Number'] = 'M' + df.loc[m, 'Number']
Alternatively, using str.replace and a slightly different regex:
pointzero = '([0-9]+[.][0-9]+[0]$)'
df['Number'] = df['Number'].str.replace(pointzero, r'M\1', regex=True))
Example:
Number
0 M1203.10
1 4221
2 3452.11

you should make dataframe or seires example for answer
example:
s1 = pd.Series(["1203.10", "4221","3452.11"])
s1
0 M1203.10
1 4221
2 3452.11
dtype: object
str.contains + boolean masking
cond1 = s1.str.contains('[0-9]+[.][0-9]+[0]$')
s1.mask(cond1, 'M'+s1)
output:
0 M1203.10
1 4221
2 3452.11
dtype: object

Return nth value in pandas column with str.find() string

I have a column and I need to return the nth character. I used string.find() to get the indexes but I cannot find an answer how to return the values in a pd dataframe column.
Values
str.find() index
Outcome should be
asdfa 5-23
7
-
kj 1-13 adlkadg
5
-
.....
.....
...
Column "Values" is scraped from the internet.
My code to find the second column is :
df["str.find() index"] = df["Values"].str.find("-")

Return nth value in pandas column with str.find() string
There are many ways to do this, but since you are asking for the method which uses str.find(), here are some ways you can do this.
You can do it without explicitly using the str.find() index column you created by using apply
df['Outcome1'] = df['Values'].apply(lambda x: x[x.find("-")])
But, if you want to use the column you have created with index, then you can use df.apply with lambda over each row -
df['Outcome2'] = df.apply(lambda row: row['Values'][row['str.find() index']], axis=1)
Values str.find() index Outcome1 Outcome2
0 asdfa 5-23 7 - -
1 kj 1-13 adlkadg 4 - -

Transform a dataframe without looping?

I would like to analyse and transform the following DataFrame
import random
import string
import numpy as np
import pandas as pd
# generate example dataframe
df=pd.DataFrame()
df['Name']=[str(x) for x in np.random.choice(['a','b','c'],10)]
df['Cat1']=[str(x) for x in np.random.choice(['x',''],10)]
df['Cat2']=[str(x) for x in np.random.choice(['x',''],10)]
df['Cat3']=[str(x) for x in np.random.choice(['x',''],10)]
df.head(10)
This produces a DataFrame like this:
Sample DataFrame
The task is to count the 'x' in columns Cat1, Cat2, Cat3 for each unique entry in column 'Name'. This can be achieved wth help ofthe groupby() function:
grouped=df.groupby(['Name'])
dfg=grouped['Cat1','Cat2','Cat3'].sum()
dfg
Result of analysis
And the result is this almost what I wanted. Now, I needed to replace the 'x' by a number, e.g., 'xxxx' by 4, 'x' by 1, and so forth. The solution uses a loop over all columns:
for col in range(0,len(dfg.columns)):
dfg[dfg.columns[col]]=list(map(lambda x: len(x), dfg[dfg.columns[col]]))
dfg
Final result.
Now, I wonder how I can avoid that loop and achieve the same final result?
Thanks a lot for sharing your ideas and guidance.

Try:
df.set_index('Name').eq('x')\
.groupby('Name')['Cat1','Cat2','Cat3'].sum()\
.astype(int).reset_index()
Output:
Name Cat1 Cat2 Cat3
0 a 5 3 4
1 b 1 1 0
2 c 1 1 1

Depending on your source of data, this could be easily solved by replacing the "x" with a 1 and setting the empty cells to 0. So you also had to change the datatype of the column to integer.
Calling sum() then on your group will already give you the numeric answer.

string manipulations in Pandas

I am trying to manipulate a large list of strings, so cannot do this manually. I am new to python so am having trouble figuring this out.
I have a dataframe with columns:
df = pd.read_csv('filename.csv')
df
A B
0 big_apples
1 big_oranges
2 small_pears
3 medium_grapes
and I need it to look more like:
A B
0 apples(big)
1 oranges(big)
2 pears(small)
3 grapes(medium)
I was thinking of using a startswith() function and .replace()/concatenate everything. But then I would have to create columns for each of these and i need it to recognize the unique prefixes. Is there a more efficient method?

You can do some string formatting and apply it to the Series:
df.B.apply(lambda x: '{}({})'.format(*x.split('_')[::-1]))
0 apples(big)
1 oranges(big)
2 pears(small)
3 grapes(medium)
Here apply is applying the formatting to each item of the series. Then apply the string formatting you desire (I'm using [::-1] to reverse the order of the string) and * to "unpack" the return values that are in a list

how do you pass multiple variables to pandas dataframe to use them with .map to create a new column

To pass multiple variables to a normal python function you can just write something like:
def a_function(date,string,float):
do something....
convert string to int,
date = date + (float * int) days
return date
When using Pandas DataFrames I know you can create a new column based on the contents of one like so:
df['new_col']) = df['column_A'].map(a_function)
# This might return the year from a date column
# return date.year
What I'm wondering is in the same way you can pass multiple pieces of data to a single function (as seen in the first example above), can you use multiple columns in the creation of a new pandas DataFrame column?
For example combining three separate parts of a date Y - M - D into one field.
df['whole_date']) = df['Year','Month','Day'].map(a_function)
I get a key error with the following test.
def combine(one,two,three):
return one + two + three
df = pd.DataFrame({'a': [1,2,3], 'b': [2,3,4],'c': [4,5,6]})
df['d'] = df['a','b','b'].map(combine)
Is there a way of creating a new column in a pandas DataFrame using .map or something else which takes as input three columns and returns a single column?
-> Example input: 1, 2, 3
-> Example output: 1*2*3
Likewise is there also a way of having a function take in one argument, a date and return three new pandas DataFrame columns; one for the year, month and day?

Is there a way of creating a new column in a pandas dataframe using .MAP or something else which takes as input three columns and returns a single column. For example input would be 1, 2, 3 and output would be 1*2*3
To do that, you can use apply with axis=1. However, instead of being called with three separate arguments (one for each column) your specified function will then be called with a single argument for each row, and that argument will be a Series containing the data for that row. You can either account for this in your function:
def combine(row):
return row['a'] + row['b'] + row['c']
>>> df.apply(combine, axis=1)
0 7
1 10
2 13
Or you can pass a lambda which unpacks the Series into separate arguments:
def combine(one,two,three):
return one + two + three
>>> df.apply(lambda x: combine(*x), axis=1)
0 7
1 10
2 13
If you want to pass only specific rows, you need to select them by indexing on the DataFrame with a list:
>>> df[['a', 'b', 'c']].apply(lambda x: combine(*x), axis=1)
0 7
1 10
2 13
Note the double brackets. (This doesn't really have anything to do with apply; indexing with a list is the normal way to access multiple columns from a DataFrame.)
However, it's important to note that in many cases you don't need to use apply, because you can just use vectorized operations on the columns themselves. The combine function above can simply be called with the DataFrame columns themselves as the arguments:
>>> combine(df.a, df.b, df.c)
0 7
1 10
2 13
This is typically much more efficient when the "combining" operation is vectorizable.
Likewise is there also a way of having a function take in one argument, a date and return three new pandas dataframe columns; one for the year, month and day?
As above, there are two basic ways to do this: a general but non-vectorized way using apply, and a faster vectorized way. Suppose you have a DataFrame like this:
>>> df = pandas.DataFrame({'date': pandas.date_range('2015/05/01', '2015/05/03')})
>>> df
date
0 2015-05-01
1 2015-05-02
2 2015-05-03
You can define a function that returns a Series for each value, and then apply it to the column:
def dateComponents(date):
return pandas.Series([date.year, date.month, date.day], index=["Year", "Month", "Day"])
>>> df.date.apply(dateComponents)
11: Year Month Day
0 2015 5 1
1 2015 5 2
2 2015 5 3
In this situation, this is the only option, since there is no vectorized way to access the individual date components. However, in some cases you can use vectorized operations:
>>> df = pandas.DataFrame({'a': ["Hello", "There", "Pal"]})
>>> df
a
0 Hello
1 There
2 Pal
>>> pandas.DataFrame({'FirstChar': df.a.str[0], 'Length': df.a.str.len()})
FirstChar Length
0 H 5
1 T 5
2 P 3
Here again the operation is vectorized by operating directly on the values instead of applying a function elementwise. In this case, we have two vectorized operations (getting first character and getting the string length), and then we wrap the results in another call to DataFrame to create separate columns for each of the two kinds of results.

I normally use apply for this kind of thing; it's basically the DataFrame version of map (the axis parameter lets you decide whether to apply your function to rows or columns):
df.apply(lambda row: row.a*row.b*row.c, axis =1)
or
df.apply(np.prod, axis=1)
0 8
1 30
2 72