I'm attempting to write a quick python script to iterate through all csv files in the current folder and remove the header row from them then store them in a separate folder.
The current working directory has four sample csv files and the python script. Once executed the script creates the HeaderRemoved directory.
It appears that once the folder is created the code that is attempting to read the files is trying to access the folder but looking at the code I'm not sure why it would be.
I'm on a windows machine at the moment.
import csv, os, argparse, string
from ctypes import *
os.makedirs('HeaderRemoved', exist_ok=True)
# Loop through files in the current working directory
for csvFile in os.listdir('.'):
if not csvFile.endswith('.csv'):
continue # Skips non-csv files
print ('Removing header from ' + csvFile + '...')
# Read in CSV skipping the first row
csvRows = []
csvFileObj = open(csvFile)
csvReader = csv.reader(csvFileObj)
for row in csvReader:
if csvReader.line_num == 1:
continue # Skips the first row
csvRows.append(row)
csvFileObj.close()
# Write out the CSV file
csvFileObj = open (os.path.join('HeaderRemoved', csvFile), 'w', newline='')
for row in csvRows:
csvWriter.writerow(row)
csvFileObj.close()
Sample output:
Removing header from examplefile_1.csv...
Removing header from examplefile_2.csv...
Removing header from examplefile_3.csv...
Removing header from examplefile_4.csv...
Traceback (most recent call last): File "atbs_csv_parse.py", line 14, in <module>
csvFileObj = open(csvFile) PermissionError: [Errno 13] Permission denied: 'HeaderRemoved'
In my case, I had opened the csv file via Excel and ran the script. Then this Permission denied exception occurred.
Just closed the opened file and run the script again :)
In my case, the same error was because I was passing a directory name instead the file name.
Maybe could be the same problem of others.
The file in the script is opened somewhere in the system. That is the reason for getting "PermissionError : [Error 13]".
Solution:
Just close the file and run the script. You won't get the error.
As Charles Duffy commented under my original question, the issue was in fact that the lines of code for reading and writing the files had not been indented to fall within the for loop. Correcting the indentation fixed the issue and it now works as desired.
A good reminder to always check the simple things.... I got so wrapped up in why it wasn't working that I didn't even notice the lack of indentation.
If you are facing the problem where you can use the csv file with hard coded path but can't use with the windows directory or file path as the pandas or other library do not have the permission to use that object diectly, to so you have to convert it to stirng and use. I faced this issue and this solution worked for me.
from pathlib import Path
cur_dir=Path.cwd()
#cwd = os.getcwd()
csv_path=str(cur_dir)+"\\..\\Dataset\\FuelConsumptionCo2.csv"
df = pd.read_csv(csv_path)
Related
I am trying to loop through 22 .txt files in a directory, read each lines from the file, and apply some conditions, before writing the modified lines in a newly created file in a different folder.
The issue is that the first 12 files are successfully created, but then I get a permission denied error for all subsequent files. The permissions for the files that fail are the exact same ones as the one that succeed.
Here is a simplified version of my code, excluding a bunch of if conditions for simplicity:
files = os.listdir('./folder/')
for file in files:
with open('./folder/' + file, "r") as input:
with open('./folder/mod/' + file[:-4] + '_mod.txt', "w") as output:
lines = input.readlines()
for i in range(len(lines)):
if lines[i][-7:-1] != ' },':
output.write('\n' + lines[i][:-4].replace('"',''))
output.close()
input.close()
I have tried to exclude the first 12 files completely from the source folder, and then not a single file gets written. As if the issue really lies with the .txt files themselves. These files were all generated at the same time, and are fairly simple, so I can't pinpoint any difference between them.
Here is the Stack Trace returned:
Traceback (most recent call last):
File "C:\Users\myname\Documents\project1\main.py", line 8, in <module>
with open('./folder/' + file, "r") as input:
IOError: [Errno 13] Permission denied: './folder/mod'
***EDIT
If I leave only 1 file that was successfully read and written in the source folder, the operation is successful but I still get the same permission denied error.
The stack trace says the problem is that:
`with open('./folder/' + file, "r") as input:`
is trying to open and read "folder/mod/"
os.listdir() will create a list of everything in a folder, including directories.
So since you can't "read" a directory, python is giving you a permission error.
Adding a test to see if the "file" in the for loop is actually a file should solve the problem:
I'm using Python 3.5 and I'm having some problems opening a CSV file. I've tried entering the entire path but it still doesn't work, but the file is clearly in the folder. (My code is called 'simplecsvtest.py')
Here's the code snippet:
import csv
import sys
file = open(r"C:\python35\files\results.csv", 'rt')
try:
reader = csv.reader(file, delimiter='\t')
... some code here ...
finally:
file.close()
And here's what PowerShell says:
PS C:\python35\files> python simplecsvtest.py
Traceback (most recent call last):
File "simplecsvtest.py", line 20, in
file = open(r"C:\python35\files\results.csv", 'rt')
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\python35\\files\\results.csv'
Well, I'm very certain that 'results.csv' is in that folder: here's the filepath in Windows Explorer:
C:\Python35\files
(Note: The folder has capital 'P' for Python35, and I've tried having both capitalized and uncapitalized 'P' in the code, neither works)
The CSV file is a "Microsoft Excel Comma Separated Values Files", if that matters, but the extension is still csv. Can anyone tell me what's wrong?
I would suggest creating a folder inside your project folder and then use a relative path :
file = open(r".\files\results.csv", 'rt')
. implies that the path is relative to your current directory
I figured out a work-around solution myself:
Somehow, if I copy all the data from the csv and paste it in a new excel spreadsheet and save it as a csv, it works. I don't know why.
Although the file path is absolute it wouldn't work this way.
You have to use "forward" slashes in the path name instead of "backward" slashes. As
file = open(r"C:/python35/files/results.csv", 'rt')
My current python script is like this:
import csv
with open ('2017_IL_sales.csv','r') as infile:
reader=csv.reader(infile)
with open('2017_IL_sales_report.csv') as outfile:
writer=csv.writer(outfile)
dict_report={rows[1]:rows[2] for rows in reader}
print dict_report
In brief, I want to open a csv file called 2017_IL_sales then create a dictionary for two columns inside. But with whatever reason, everytime I run the code via IDLE it told me this IOError: [Errno 2] No such file or directory: '2017_IL_sales.csv'. Anyone knows the reason?
Make sure that the script you are running is in the same folder as the file you are trying to read. If that is not possible, make sure to specify the correct file path.
Is IDLE's working directory that folder?
Run this in IDLE and see what you get:
import os
os.getcwd()
I am trying to make a minor modification to a python script made by my predecessor and I have bumped into a problem. I have studied programming, but coding is not my profession.
The python script processes SQL queries and writes them to an excel file, there is a folder where all the queries are kept in .txt format. The script creates a list of the queries found in the folder and goes through them one by one in a for cycle.
My problem is if I want to rename or add a query in the folder, I get a "[Errno 2] No such file or directory" error. The script uses relative path so I am puzzled why does it keep making errors for non-existing files.
queries_pathNIC = "./queriesNIC/"
def queriesDirer():
global filelist
l = 0
filelist = []
for file in os.listdir(queries_pathNIC):
if file.endswith(".txt"):
l+=1
filelist.append(file)
return(l)
Where the problem arises in the main function:
for round in range(0,queriesDirer()):
print ("\nQuery :",filelist[round])
file_query = open(queries_pathNIC+filelist[round],'r'); # problem on this line
file_query = str(file_query.read())
Contents of queriesNIC folder
00_1_Hardware_WelcomeNew.txt
00_2_Software_WelcomeNew.txt
00_3_Software_WelcomeNew_AUTORENEW.txt
The scripts runs without a problem, but if I change the first query name to
"00_1_Hardware_WelcomeNew_sth.txt" or anything different, I get the following error message:
FileNotFoundError: [Errno 2] No such file or directory: './queriesNIC/00_1_Hardware_WelcomeNew.txt'
I have also tried adding new text files to the folder (example: "00_1_Hardware_Other.txt") and the script simply skips processing the ones I added altogether and only goes with the original files.
I am using Python 3.4.
Does anyone have any suggestions what might be the problem?
Thank you
The following approach would be an improvement. The glob module can produce a list of files ending with .txt quite easily without needing to create a list.
import glob, os
queries_pathNIC = "./queriesNIC/"
def queriesDirer(directory):
return glob.glob(os.path.join(directory, "*.txt"))
for file_name in queriesDirer(queries_pathNIC):
print ("Query :", file_name)
with open(file_name, 'r') as f_query:
file_query = f_query.read()
From the sample you have given, it is not clear if you need further access to the round variable or the file list.
Currently I am trying to read in a csv file using the csv module in python. When I run the piece of code below I get an error that states the file does not exist. My first guess is that maybe, I have the file saved in the wrong place or I need to provide pyton with a file path. currently I have the file saved in C:\Documents and Settings\eag29278\My Documents\python test code\test_satdata.csv.
one side note im note sure wether having set the mode to 'rb' (read binary) was the right move.
import csv
with open('test_satdata.csv', 'rb') as csvfile:
satreader = csv.reader(csvfile, delimiter=' ', lineterminator=" ")
for row in satreader:
print ', '.join(row)
Here is the errror code.
Traceback (most recent call last):
File "C:/Python27/test code/test csv parse.py", line 2, in <module>
with open('test_satdata.csv', 'rb') as csvfile:
IOError: [Errno 2] No such file or directory: 'test_satdata.csv'
Your code is using a relative path; python is looking in the current directory (whatever that may be) to load your file. What the current directory is depends on how you started your Python script and if you executed any code that may have changed the current working directory.
Use a full absolute path instead:
path = r'C:\Documents and Settings\eag29278\My Documents\python test code\test_satdata.csv'
with open(path, 'rb') as csvfile:
Using 'rb' is entirely correct, the csv module recommends you do so:
If csvfile is a file object, it must be opened with the ‘b’ flag on platforms where that makes a difference.
Windows is such a platform.
You can hit this error when you're running a Python script from a Directory where the file is not contained.
Sounds simple to fix, put the CSV file in the same folder as the .PY file. However when you're running under an IDE like VSCode the command output might cd to another directory when it executes your python file.
PS C:\git\awesome> cd 'c:\git\awesome'; ${env:PYTHONIOENCODING}='UTF-8'; ${env:PYTHONUNBUFFERED}='1';
& 'C:\Program Files (x86)\Python37-32\python.exe' 'c:\Users\jeremy\.vscode\extensions\ms-python.python-2019.9.34911\pythonFiles\ptvsd_launcher.py'
'--default' '--client' '--host' 'localhost' '--port' '1089'
'c:\git\Project\ReadCSV.py'
See how I'm in my awesome repo and the first command is: cd 'c:\git\awesome';
Then it executes the python file: 'c:\git\Project\ReadCSV.py'
So its expecting the CSV file in 'c:\git\awesome'.
To fix it, either use the full file names or CD to the directory containing the CSV file you wish to read.
Your current guess is right: either put the file in your test code directory or point python to the right path.
Make a fresh rename of your folder. That worked for me.