This question already has answers here:
Import multiple CSV files into pandas and concatenate into one DataFrame
(20 answers)
How do I combine two dataframes?
(8 answers)
Closed 8 months ago.
I am trying to join a lot of CSV files into a single dataframe after doing some conversions and filters, when I use the append method for the sn2 dataframe, the exported CSV contains all the data I want, however when I use the append method for the sn3 dataframe, only the data from the last CSV is exported, what am I missing?
sn2=pd.DataFrame()
sn3=pd.DataFrame()
files=os.listdir(load_path)
for file in files:
df_temp=pd.read_csv(load_path+file)
df_temp['Date']=file.split('.')[0]
df_temp['Date']=pd.to_datetime(df_temp['Date'],format='%Y%m%d%H%M')
filter1=df_temp['Name']=='Atribute1'
temp1=df_temp[filter1]
sn2=sn2.append(temp1)
filter2=df_temp['Name']=='Atribute2'
temp2=df_temp[filter2]
sn3=pd.concat([temp2])
You have to pass all the dataframes that you want to concatenate to concat:
sn3 = pd.concat([sn3, temp2])
Related
This question already has answers here:
How to reversibly store and load a Pandas dataframe to/from disk
(13 answers)
Saving and Loading of dataframe to csv results in Unnamed columns
(4 answers)
Closed 6 months ago.
Which file format can be used to save a Pandas DataFrame object and then loading it back with the proper index? I.e. if column blah was an index before saving it to the file, I want that after loading it back again blah to be an index without me having to tell this to Pandas.
df.to_pickle('file.pickle')
df = pd.read_pickle('file.pickle')
This question already has answers here:
Pandas groupby with delimiter join
(2 answers)
Closed 8 months ago.
I have a CSV file called Project.csv
I am reading this file using pandas df = pd.read_csv('Project.csv',low_memory=False)
Inside this CSV file, there are rows with duplicate Project ID and Name but the other Column data are unique. I was looking for a way to find duplicate rows based on Project ID and merge records with ',' if they are unique.
Project.csv
I am looking to store this record in a data frame and filter it to make it look like this:
A simply groupby will do the job
after_df = your_df.groupby(['Project Id'])[['Project Name','Project Types','Owner']].agg(set)
This will give you a similar result to what you want. If you want to take out the key symbool of the strings parameters so you have a nice looking string do this.
after_df.astype(str).replace(r'{|}|\'','',regex=True)
This question already has answers here:
Joining pandas DataFrames by Column names
(3 answers)
Pandas Merging 101
(8 answers)
Closed last year.
I am following this article, but I was only able to get it to work by making sure there were matching titles, the two still had computer names, but they were called differently in the title, how could I modify my command so that it still references the same column, is that possible?
lj_df2 = pd.merge(d2, d3, on="PrimaryUser", how="left")
For example, I have this, but on my other csv, I have Employee # not primary user
This question already has answers here:
What's the most efficient way to export multiple pandas dataframes to csv files?
(3 answers)
Closed 1 year ago.
I am new to python. I am sure there is simple way to do this but I am struggling a bit.
I have 100 dataframes with names Lens1,Lens2, ..., Lens100.
I want to write each dataframe to a csv file.
Lens1.to_csv(path+"lens 1.csv", index=False) This command for Lens2 and Lens2.csv... and so on till Lens100 save as Lens100.csv. so a 100 times...
I have tried the following:
for key,j in range(101):
x='Lens%s'%(j)
x.to_csv(path+x+".csv")
It does not seem to work and the error is
'str' object has no attribute 'to_csv'.
Any help will be much appreciated.
You are getting this error because the x in your for loop are not dataframes, but strings. You cannot call a dataframe by adding a str to its name; Python will consider it as a string.
You can store the dataframes in a list dataframes first, then store the names of the dataframes in a list name. Then, proceed using the following codes.
for j in range(101):
x = dataframes[j]
x.to_csv(path + name[j] + ".csv", index = False)
This question already has answers here:
Using a loop in Python to name variables [duplicate]
(5 answers)
Append multiple pandas data frames at once
(5 answers)
Closed 5 years ago.
I have pandas data frame numbered from x1,x2....x100 with same columns.
I want to append them all using a for loop. How can i do that?
I know how to append two dataframe but how to do it for 100 of them. The main problem here is how can i have a dynamic variable name.
I want to append the data frames not concat.
x=x1.append(x2)
x=x.append(x3)
and so on.
I want to this in a loop.