I'm trying to input data from a csv and run that through a simple function and return it to the same CSV but appended onto the end of the row.
I'm having a hard time understanding the difference between the various modes(r, r+, a+, etc).
I've read the documentation but as a beginner am not understanding quite what they mean and which one is the right for me to be using in this case.
def appendCurrentTime():
with open("file.csv", "??") as myfile: #Opens File
reader = csv.reader(myfile)
for row in reader: #Runs through each row
myfile.write(current_time(row[1])) #appends the time, pulling in row[1] into current_time
**Also posting code for the first time, sorry if it doesn't come through clearly.
Looking at your code, the main problem is that the CSV file is not a "database", so you can't modify a row in the middle without corrupting the next row.
So, open the source file in "rb" mode and destination file in "wb" mode, and AFTER you processed the whole file - delete original and rename new file to the original name.
And, description for the cases you specified:
"r" - open text file for reading, starting from the beginning.
"r+" - open text file for reading and writing, starting from the beginning.
"a+" - create file if not exists; then open as a text file for reading and writing, starting from the end of file, plus you can only write to the end of the file.
In short, "R" means Read, "W" write, "+" - "and other way around", "B" - binary, absense of "B" means text. "A" differs from "W" in that "W" clears the file contents when you call open.
IMHO you should probably open files in binary mode most of the time, even if they are logically "text", to prevent unintended conversions of special characters (especially important with Unicode). So, I would recommend "wb" to "write new file", "rb" to "read existing" and "ab" to "append to existing file" e.g. a log file.
For even more information, read the documentation for the POSIX fopen function - Python tries to adhere to its semantics as much as possible, even on Windows systems.
Since it is a CSV file and implied text, the "rU" flag would be appropriate. The r flag means read and U is universal line terminator mode. This way it won't matter if the file is using Windows, Mac or Unix line terminators.
Also take a look into the Sniffer provided by the CSV library. It will automatically detect most dialects for you and can be passed into the reader function.
http://docs.python.org/2/library/csv.html#csv.Sniffer
Related
Is there a way to stop Python rewriting files?
For example, when I run code once, it writes number "1" in text file. If I run same code tommorow, it won't erase that 1 from yesterday, it will add one more "1", so it should look like this:
1
1
My would look like this:
file = open("filename","w")
file.write(str(1) + "\n)
file.close()
If someone can tell me where is my mistake, please help me.
Thank you in advance !
If you read the docs for open, that second parameter that you're passing a "w" to is the mode:
mode is an optional string that specifies the mode in which the file is opened. It defaults to 'r' which means open for reading in text mode. Other common values are 'w' for writing (truncating the file if it already exists), 'x' for exclusive creation and 'a' for appending…
And then there's a handy chart of all of the mode characters, which reiterates the same information:
'w': open for writing, truncating the file first
...
'a': open for writing, appending to the end of the file if it exists
So, just use "a" instead of "w", and Python will add your new line to the end of the file, instead of truncating the file to nothing and then adding your new line to that now-empty file.
It's probably worth knowing that this is very closely based on the behavior of the fopen function from C and POSIX—which many other languages have also copied. So, in almost any language you run into, if there's a w mode for writing that overwrites the file, there's probably also an a mode for appending.
Started Python a week ago and I have some questions to ask about reading and writing to the same files. I've gone through some tutorials online but I am still confused about it. I can understand simple read and write files.
openFile = open("filepath", "r")
readFile = openFile.read()
print readFile
openFile = open("filepath", "a")
appendFile = openFile.write("\nTest 123")
openFile.close()
But, if I try the following I get a bunch of unknown text in the text file I am writing to. Can anyone explain why I am getting such errors and why I cannot use the same openFile object the way shown below.
# I get an error when I use the codes below:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
readFile = openFile.read()
print readFile
openFile.close()
I will try to clarify my problems. In the example above, openFile is the object used to open file. I have no problems if I want write to it the first time. If I want to use the same openFile to read files or append something to it. It doesn't happen or an error is given. I have to declare the same/different open file object before I can perform another read/write action to the same file.
#I have no problems if I do this:
openFile = open("filepath", "r+")
writeFile = openFile.write("Test abc")
openFile2 = open("filepath", "r+")
readFile = openFile2.read()
print readFile
openFile.close()
I will be grateful if anyone can tell me what I did wrong here or is it just a Pythong thing. I am using Python 2.7. Thanks!
Updated Response:
This seems like a bug specific to Windows - http://bugs.python.org/issue1521491.
Quoting from the workaround explained at http://mail.python.org/pipermail/python-bugs-list/2005-August/029886.html
the effect of mixing reads with writes on a file open for update is
entirely undefined unless a file-positioning operation occurs between
them (for example, a seek()). I can't guess what
you expect to happen, but seems most likely that what you
intend could be obtained reliably by inserting
fp.seek(fp.tell())
between read() and your write().
My original response demonstrates how reading/writing on the same file opened for appending works. It is apparently not true if you are using Windows.
Original Response:
In 'r+' mode, using write method will write the string object to the file based on where the pointer is. In your case, it will append the string "Test abc" to the start of the file. See an example below:
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\n'
>>> f.write("foooooooooooooo")
>>> f.close()
>>> f=open("a","r+")
>>> f.read()
'Test abc\nfasdfafasdfa\nsdfgsd\nfoooooooooooooo'
The string "foooooooooooooo" got appended at the end of the file since the pointer was already at the end of the file.
Are you on a system that differentiates between binary and text files? You might want to use 'rb+' as a mode in that case.
Append 'b' to the mode to open the file in binary mode, on systems
that differentiate between binary and text files; on systems that
don’t have this distinction, adding the 'b' has no effect.
http://docs.python.org/2/library/functions.html#open
Every open file has an implicit pointer which indicates where data will be read and written. Normally this defaults to the start of the file, but if you use a mode of a (append) then it defaults to the end of the file. It's also worth noting that the w mode will truncate your file (i.e. delete all the contents) even if you add + to the mode.
Whenever you read or write N characters, the read/write pointer will move forward that amount within the file. I find it helps to think of this like an old cassette tape, if you remember those. So, if you executed the following code:
fd = open("testfile.txt", "w+")
fd.write("This is a test file.\n")
fd.close()
fd = open("testfile.txt", "r+")
print fd.read(4)
fd.write(" IS")
fd.close()
... It should end up printing This and then leaving the file content as This IS a test file.. This is because the initial read(4) returns the first 4 characters of the file, because the pointer is at the start of the file. It leaves the pointer at the space character just after This, so the following write(" IS") overwrites the next three characters with a space (the same as is already there) followed by IS, replacing the existing is.
You can use the seek() method of the file to jump to a specific point. After the example above, if you executed the following:
fd = open("testfile.txt", "r+")
fd.seek(10)
fd.write("TEST")
fd.close()
... Then you'll find that the file now contains This IS a TEST file..
All this applies on Unix systems, and you can test those examples to make sure. However, I've had problems mixing read() and write() on Windows systems. For example, when I execute that first example on my Windows machine then it correctly prints This, but when I check the file afterwards the write() has been completely ignored. However, the second example (using seek()) seems to work fine on Windows.
In summary, if you want to read/write from the middle of a file in Windows I'd suggest always using an explicit seek() instead of relying on the position of the read/write pointer. If you're doing only reads or only writes then it's pretty safe.
One final point - if you're specifying paths on Windows as literal strings, remember to escape your backslashes:
fd = open("C:\\Users\\johndoe\\Desktop\\testfile.txt", "r+")
Or you can use raw strings by putting an r at the start:
fd = open(r"C:\Users\johndoe\Desktop\testfile.txt", "r+")
Or the most portable option is to use os.path.join():
fd = open(os.path.join("C:\\", "Users", "johndoe", "Desktop", "testfile.txt"), "r+")
You can find more information about file IO in the official Python docs.
Reading and Writing happens where the current file pointer is and it advances with each read/write.
In your particular case, writing to the openFile, causes the file-pointer to point to the end of file. Trying to read from the end would result EOF.
You need to reset the file pointer, to point to the beginning of the file before through seek(0) before reading from it
You can read, modify and save to the same file in python but you have actually to replace the whole content in file, and to call before updating file content:
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
I needed a function to go through all subdirectories of folder and edit content of the files based on some criteria, if it helps:
new_file_content = ""
for directories, subdirectories, files in os.walk(folder_path):
for file_name in files:
file_path = os.path.join(directories, file_name)
# open file for reading and writing
with io.open(file_path, "r+", encoding="utf-8") as edit_file:
for current_line in edit_file:
if condition in current_line:
# update current line
current_line = current_line.replace('john', 'jack')
new_file_content += current_line
# set the pointer to the beginning of the file in order to rewrite the content
edit_file.seek(0)
# delete actual file content
edit_file.truncate()
# rewrite updated file content
edit_file.write(new_file_content)
# empties new content in order to set for next iteration
new_file_content = ""
edit_file.close()
I'm actually working on a project to send file using UDP, and since this protocol is not reliable I added some information on each packet which is the index of the data. So I can write the received data in the correct order.
I have problems to write bytes in a specific position in a file
this is the part of my code that handle writing new data :
while i < packet_num:
buf,address = recieve_packet(s,data_size+10)
i += 1
if buf:
print(buf)
index = int(buf[0:10].decode())
data = buf[10:]
f.seek(seek_pointer + index*data_size,0)
f.write(data)
list_index.append(index)
in this case the seek function has no effect and the data is just appended to the file. I'm using "a+b" mode to open the file.
Quoting from tutorialspoint.com,
Note that if the file is opened for appending using either 'a' or 'a+', any seek() operations will be undone at the next write.
"a" mode write operations append to the end of the file. What seek does is it sets the write/read pointer to a specific location in the file.
Therefore, when a write is called, it will write to the end of file, regardless of the read/write pointer.
However, because you've opened the file in a+b, you would be able to seek to a specific location and read it.
If you open using 'append' mode, all writes go to the end of the file. If ypu are already keeping track of where received data, then opening in w+b mode is all you need to do.
wb creates (or empties) the file, and allows writing (in binary, rather than text mode). w+b Does the same, but allows reading as well. If you want to open an existing file without truncating it, mode r+b will allow both reading and writing, while preserving the existing data (again, the b is for binary mode, which I expect is correct for your uses).
I am getting the following error when trying to output to a text file:
io.UnsupportedOperation: not writable.
My code:
def PostCodeStore(self):
#Opens the Postcode file in append mode
file = open("PostCode_File.txt", "r")
PostCodeValue= PostCodeVar.get()
#Writes the Postcode value to the file and adds a new line
file.write(PostCodeValue + "\n")
#Closes the file so it saves the value
file.close()
The reason is that you can't write to a file opened with 'r'. You must open with 'a'. From the documentation:
The first argument is a string containing the filename. The second argument is another string containing a few characters describing the way in which the file will be used. mode can be 'r' when the file will only be read, 'w' for only writing (an existing file with the same name will be erased), and 'a' opens the file for appending; any data written to the file is automatically added to the end. 'r+' opens the file for both reading and writing. The mode argument is optional; 'r' will be assumed if it’s omitted.
On Windows, 'b' appended to the mode opens the file in binary mode, so there are also modes like 'rb', 'wb', and 'r+b'. Python on Windows makes a distinction between text and binary files; the end-of-line characters in text files are automatically altered slightly when data is read or written. This behind-the-scenes modification to file data is fine for ASCII text files, but it’ll corrupt binary data like that in JPEG or EXE files. Be very careful to use binary mode when reading and writing such files. On Unix, it doesn’t hurt to append a 'b' to the mode, so you can use it platform-independently for all binary files.
This is a common issue, and (as seen in the docs), one that can cause some problems when writing cross-compatible programs, as Windows has a different set of permissions for binary files. You are likely not going to be working with binary files (maybe zipfiles), so I wouldn't worry about that too much. Just remember:
'w' # Writing
'r' # Reading (default)
'a' # Appending
'r+' # Read/Write
I believe I understand your problem correctly. That is, that your program wants to write many lines, one at a time (append). If you don't want to do that and just want a single entry in the file with a newline, you will want to use 'w'.
Also (and as a side note), Python class names generally start with a capital letter. That's why PostCodeValue shows up with different formatting. Conventionally, a variable like this should be named with small letters and underscores, such as post_code_value.
You're wrong
file = open("PostCode_File.txt", "r")#Opens the Postcode file in append mode
With r as mode, you open the file read-only, not append. That'd be a.
I know this is a simple question, but I am extremely stuck.
file=open("record.txt","w+")
record = file.read()
print("The record is "+str(record)+"!!")
main code...
file.write(str(reaction))
file.close()
I have got his code and I've got a number of 0.433534145355 in the file, but when I do the command of print the +str(record)+, it only comes up with The record is !! and the number is not there. What is wrong with this code. Is there a special code with decimal places, and I do not want to use int().
As it says here:
'w+' Open for reading and writing. The file is created if it does not
exist, otherwise it is truncated. The stream is positioned at
the beginning of the file.
so yes, your file is also opened for reading, but it is truncated (i.e. it is now zero bytes long, it's empty), leaving nothing left to read of what was there already.
Essentially, the w in 'w+' means the mode is orientated to writing, giving you the option to read as well (useful in those cases when you need to seek back and read what you have written. There will be nothing to read unless you write)
Instead you can use:
'r+' Open for reading and writing. The stream is positioned at the
beginning of the file.
In this case, the r in 'r+' signifies the mode is orientated to reading, giving you the option to seek and write where necessary (useful when data is present already, but might need to be changed)
If you want to read from a file, you have to open it for reading too (r).