We Keep Coding

Python how to filter a csv based on a column value and get the row count

I want to do data insepction and print count of rows that matches a certain value in one of the columns. So below is my code
import numpy as np
import pandas as pd
data = pd.read_csv("census.csv")
The census.csv has a column "income" which has 3 values '<=50K', '=50K' and '>50K'
and i want to print number of rows that has income value '<=50K'
i was trying like below
count = data['income']='<=50K'
That does not work though.

Sum Boolean selection
(data['income'].eq('<50K')).sum()

The key is to learn how to filter pandas rows.
Quick answer:
import pandas as pd
data = pd.read_csv("census.csv")
df2 = data[data['income']=='<=50K']
print(df2)
print(len(df2))
Slightly longer answer:
import pandas as pd
data = pd.read_csv("census.csv")
filter = data['income']=='<=50K'
print(filter) # notice the boolean list based on filter criteria
df2 = data[filter] # next we use that boolean list to filter data
print(df2)
print(len(df2))

How to update values in a CSV column using a list?

list=[8,5,3,4,5,7,3,5]
Screenshot of CSV file in Excel
I would like to replace all the values in the csv file using python. So for example all the values under items to be replaced with values from the list. So in row 2 under Items header the value 4 to be replaced with 8 from the list and so on. How can I do that?

Use pandas:
import pandas as pd
df = pd.read_csv('filename.csv')
df['items'] = [8,5,3,4,5,7,3,5]
df.to_csv('filename.csv', index=False)

Since you are updating all the values in a column, you can start from a dataframe.
import pandas as pd
import numpy as np
df = pd.read_csv('input.csv')
# now update the column
newlist=[8,5,3,4,5,7,3,5]
df['items'] = np.array(newlist)
#write back to csv
df.to_csv("output.csv", index=False)

getting the values of a pandas multiindex after querying the multindex Dataframe

Provided I have a multiindex data Frame as follows:
import pandas as pd
import pandas as pd
import numpy as np
input_id = np.array(['input_id'])
docType = np.array(['pre','pub','app','dw'])
docId = np.array(['34455667'])
sec_type = np.array(['bib','abs','cl','de'])
sec_ids = np.array(['x-y','z-k'])
index = pd.MultiIndex.from_product([input_id,docType,docId,sec_type,sec_ids])
content= [str(np.random.randint(1,10))+ '##' + str(np.random.randint(1,10)) for i in range(len(index))]
df = pd.DataFrame(content, index=index, columns=['content'])
df.rename_axis(index=['input_id','docType','docId','secType','sec_ids'], inplace=True)
I would like to query the multiindex DF
# query a multiindex DF
idx = pd.IndexSlice
df.loc[idx[:,'pub',:,'de',:]]
Resulting in:
I would like to get directly the values of the multiindex column sec_ids as a list. How do I have to modify to get the follwoing result:
['x-y','z-k']
Thanks

You can use the MultiIndex.get_level_values() method to get the values of a specific level of a MultiIndex. So in this case call it after your slice.
df.loc[idx[:,'pub',:,'de',:]].index.get_level_values('sec_ids').tolist()
#['x-y', 'z-k']

Is there any function to assign values in a Pandas Dataframe

I am trying to assign values to some rows using pandas dataframe. Is there any function to do this?

For a whole column:
df = df.assign(column=value)
... where column is the name of the column.
For a specific column of a specific row:
df.at[row, column] = value
... where row is the index of the row, and column is the name of the column.
The later changes the dataframe "in place".

There is a good tutorial here.
Basically, try this:
import pandas as pd
import numpy as np
# Creating a dataframe
# Setting the seed value to re-generate the result.
np.random.seed(25)
df = pd.DataFrame(np.random.rand(10, 3), columns =['A', 'B', 'C'])
# np.random.rand(10, 3) has generated a
# random 2-Dimensional array of shape 10 * 3
# which is then converted to a dataframe
df
You will get something like this:

Data Frame Indexing

Using python3 I wrote a code for calculating data. Code is as follows:
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
def data(symbols):
dates = pd.date_range('2016/01/01','2016/12/23')
df=pd.DataFrame(index=dates)
for symbol in symbols:
df_temp=pd.read_csv("/home/furqan/Desktop/Data/{}.csv".format(symbol),
index_col='Date',parse_dates=True,usecols=['Date',"Close"],
na_values = ['nan'])
df_temp=df_temp.rename(columns={'Close':symbol})
df=df.join(df_temp)
df=df.fillna(method='ffill')
df=df.fillna(method='bfill')
df=(df/df.ix[0,: ])
return df
symbols = ['FABL','HINOON']
df=data(symbols)
print(df)
p_value=(np.zeros((2,2),dtype="float"))
p_value[0,0]=0.5
p_value[1,1]=0.5
print(df.shape[1])
print(p_value.shape[0])
df=np.dot(df,p_value)
print(df.shape[1])
print(df.shape[0])
print(df)
When I print df for second time the index has vanished. I think the issue is due to matrix multiplication. How can I get the indexing and column headings back into df?

To resolve your issue, because you are using numpy methods, these typically return a numpy array which is why any existing columns and index labels will have been lost.
So instead of
df=np.dot(df,p_value)
you can do
df=df.dot(p_value)
Additionally because p_value is a pure numpy array, there is no column names here so you can either create a df using existing column names:
p_value=pd.DataFrame(np.zeros((2,2),dtype="float"), columns = df.columns)
or just overwrite the column names directly after calculating the dot product like so:
df.columns = ['FABL', 'HINOON']