I am trying to convert 200 text files into csv files. I am using below code I am able to run it but it does not produce csv files. Could anyone tell any easy and fast way to do? Many Thanks
dirpath = 'C:\Files\Code\Analysis\Input\qobs_RR1\\'
output = 'C:\Files\Code\Analysis\output\qobs_CSV.csv'
csvout = pd.DataFrame()
files = os.listdir(dirpath)
for filename in files:
data = pd.read_csv(filename, sep=':', index_col=0, header=None)
csvout = csvout.append(data)
csvout.to_csv(output)
The problem is that your os.listdir gives you the list of filenames inside dirpath, not the full path to these files. You can get the full path by prepending the dirpath to filenames with os.path.join function.
import os
import pandas as pd
dirpath = 'C:\Files\Code\Analysis\Input\qobs_RR1\\'
output = 'C:\Files\Code\Analysis\output\qobs_CSV.csv'
csvout_lst = []
files = [os.path.join(dirpath, fname) for fname in os.listdir(dirpath)]
for filename in sorted(files):
data = pd.read_csv(filename, sep=':', index_col=0, header=None)
csvout_lst.append(data)
pd.concat(csvout_lst).to_csv(output)
Edit: this can be done with a one-liner:
pd.concat(
pd.read_csv(os.path.join(dirpath, fname), sep=':', index_col=0, header=None)
for fname in sorted(os.listdir(dirpath))
).to_csv(output)
Edit 2: updated the answer, so the list of files is sorted alphabetically.
Related
I want to concat all csv file that have this specific word 'tables' on the filename.
Below code is upload all csv file without filter the specific word that i want.
# importing the required modules
import glob
import pandas as pd
# specifying the path to csv files
#path = "csvfoldergfg"
path = "folder_directory"
# csv files in the path
files = glob.glob(path + "/*.csv")
# defining an empty list to store
# content
data_frame = pd.DataFrame()
content = []
# checking all the csv files in the
# specified path
for filename in files:
# reading content of csv file
# content.append(filename)
df = pd.read_csv(filename, index_col=None)
content.append(df)
# converting content to data frame
data_frame = pd.concat(content)
print(data_frame)
example filename are:
abcd-tables.csv
abcd-text.csv
abcd-forms.csv
defg-tables.csv
defg-text.csv
defg-forms.csv
From the example filenames. The expected output is concat filenames
abcd-tables.csv
defg-tables.csv
into single dataframe. Assuming the header are same.
*Really appreciate you guys can solve this
You can use:
import pandas as pd
import pathlib
path = 'folder_directory'
content = []
for filename in pathlib.Path(path).glob('*-tables.csv'):
df = pd.read_csv(filename, index_col=None)
content.append(df)
df = pd.concat(content, ignore_index=True)
I am trying to open multiple files with pandas into a dataframe.
Only files with a prefix ~$ show an error of
XLRDError: Unsupported format, or corrupt file: Expected BOF record; found b'\x15Microso'
Here is two of my list of filepaths:
bulk_uploads /~$0730-0731.xlsx',
bulk_uploads /0701-0702.xlsx'
The one without the prefix opens perfectly fine, and I am not sure why the other one throws an error.
Here is the code I am trying:
import pandas as pd
import glob
path = 'bulk_uploads ' # use your path
all_files = glob.glob(path + "/*.xlsx")
li = []
for filename in all_files:
df = pd.read_excel(filename, sheet_name = 1)
df['Date'] = str(filename)[:-4]
li.append(df)
# frame = pd.concat(li, axis=0, ignore_index=True)
Is there either a way to chance any files that have this prefix to lose it, or another way around it?
It looks like they are files which I have previously opened (I have no files currently open)
import pandas as pd
import glob
import re
path = 'bulk_uploads ' # use your path
all_files = glob.glob(path + "/*.xlsx")
li = []
special=re.compile('$~') #####add more special characters if any
for filename in all_files:
if special.search(filename):
os.remove(filename)
else:
df = pd.read_excel(filename, sheet_name = 1)
df['Date'] = str(filename)[:-4]
li.append(df)
Can you give this a try and see if it works fine?
It seems that your folder is having temporary files..
Hey People I would like to merge 2000 Csv files into one of 2000 sub-folders. Each sub-folder contains three Csv files with different names. so I need to select only one Csv from each folder.
I know the code for how to merge bunch of Csv files if they are in the same - folder.
import pandas as pd
import glob
path = r'Total_csvs'
all_files = glob.glob(path + "/*.csv")
li = []
for filename in all_files:
df = pd.read_csv(filename, index_col=None, header=0)
li.append(df)
frame = pd.concat(li, axis=0, ignore_index=True)
frame.to_csv('Total.csv',index=False)
But my problems with 2000 Csv files look totally different.
Folder structure is:
Main folder (with in this 2000 subfolders, within subfolders I had multiple Csv Files and I need to select only one Csv file from this. Finally concating all 2000 Csv files.)
Coming to Naming Conventions (all the subfolders had different names, but the subfolder name and the Csv name within the subfolder is same)
Any suggestions or a sample code (how to read 2000 Csv from sub-folders) would be helpful.
Thanks in advance
You can loop through all the subfolders using os.listdir.
Since the CSV filename is the same as the subfolder name, simply use the subfolder name to construct the full path name.
import os
import pandas
folders = os.listdir("Total_csvs")
li = []
for folder in folders:
# Since they are the same name
selected_csv = folder
filename = os.path.join(folder, selected_csv + ".csv")
df = pd.read_csv(filename, index_col=None, header=0)
li.append(df)
frame = pd.concat(li, axis=0, ignore_index=True)
frame.to_csv('Total.csv',index=False)
We can iterate on every subfolder, determine expected_csv_path, check if it exists. If it exists, we add them to our all_files list.
Try following:
import pandas as pd
import os
path = r'Total_csvs'
li = []
for f in os.listdir(path):
expected_csv_path = os.path.join(path, f, f + '.csv')
csv_exists = os.path.isfile(expected_csv_path)
if csv_exists:
df = pd.read_csv(expected_csv_path, index_col=None, header=0)
li.append(df)
frame = pd.concat(li, axis=0, ignore_index=True, sort=False)
frame.to_csv('Total.csv',index=False)
If you are using Python 3.5 or newer you could use glob.glob in recursive manner following way:
import glob
path = r'Total_csvs'
all_csv = glob.glob(path+"/**/*.csv",recursive=True)
Now all_csv is list of relative paths to all *.csv inside Total_csv and subdirectories of Total_csv and subdirectories of subdirectories of Total_csv and so on.
For example purpose lets assume that all_csv is now:
all_csv = ['Total_csvs/abc/abc.csv','Total_csv/abc/another.csv']
So we need to get files with names correnponding to directory of their residence, this could be done following way:
import os
def check(x):
directory,filename = x.split(os.path.sep)[-2:]
return directory+'.csv'==filename
all_csv = [i for i in all_csv if check(i)]
print(all_csv) #prints ['Total_csvs/abc/abc.csv']
Now all_csv is list of paths to all .csv you are seeking and you can use it same way as you did with all_csv in "flat" (non-recursive) case.
You can do it without joining paths:
import pathlib,pandas
lastparent=None
for ff in pathlib.Path("Total_csvs").rglob("*.csv"): # recursive glob
print(ff)
if(ff.parent!=lastparent): # process the 1st file in the dir
lastparent= ff.parent
df = pd.read_csv(str(ff),... )
...etc.
I am trying to write a code which takes all the .csv files in a directory, which are semi colon delimited, and formats the .csv file into columns. This is my code:
import pandas as pd
import glob
path = r'C:...'
csv_file = path + "\*.csv"
allFiles = glob.glob(path + "\*.csv")
for file in allFiles:
dataframe = pd.read_csv(file, delimiter=';')
dataframe.to_csv(file, encoding='utf-8', index=False)
I have tested the dataframe = part of this code, it works as desired for one .csv file, but I cannot get this to repeat for all files in the folder. Any ideas? Thanks.
If all you want to do is change ; to , in the files, something like this would work:
for root, dirs, files in os.walk("/dirname/"):
csv_files = [ff for ff in files if ff.endswith('.csv')]
for f in csv_files:
with open(f) as tf:
s = f.read()
with open(f, "w") as tf:
f.write(s.replace(";", ","))
You can use pandas and do something like this:
import pandas as pd
df1 = pd.read_csv("csv_semicolon.csv", delimiter=";")
df1.to_csv("csv_tab.csv", sep="\t", index=False)
I have multiple .txt files in a directory and I want to merge them into one by importing in python. The catch here is that after the merge I want to convert it into one csv file on which the whole program is based.
So far I only had to input one .txt file and converted it into csv file by this code:
import io
bytes = open('XYZ.txt', 'rb').read()
df=pd.read_csv(io.StringIO(bytes.decode('utf-8')), sep='\t', parse_dates=['Time'] )
df.head()
Now I need to input multiple .txt files, merge them and then convert them into csv files. Any workaround?
If the headers are same then it should be as easy as this
import os
import io
merged_df = pd.DataFrame()
for file in os.listdir("PATH_OF_DIRECTORY"):
if file.endswith(".txt"):
bytes = open(file, 'rb').read()
merged_df = merged_df.append(pd.read_csv(io.StringIO(
bytes.decode('utf-8')), sep='\t', parse_dates=['Time']))
print(len(merged_df))
import glob
path="location/of/folder"
allFiles = glob.glob(path + "\\*.txt")
list_ = []
for file in allFiles:
print(file)
df = pd.read_csv(io.StringIO(file.decode('utf-8')), sep='\t', parse_dates=['Time'])
list_.append(df)
combined_files = pd.concat(list_)