We Keep Coding

How to write to a text file using Python such a way that I can read it simultaneously in the terminal/gnuplot

I am running a long Python program which prints values to a .txt file in an iterative way. I am trying to read the values using terminal "gedit/tail/less" commands and trying to plot them in Gnuplot. But I am not able to read the .txt file till the whole execution is over. What is the correct argument for such file handling ?

The files are written when they are closed or when the size of the buffer is too large to store.
That is even when you use file.write("something"), something isn't written in the file till you close the file, or with block is over.
with open("temp.txt","w") as w:
w.write("hey")
x=input("touch")
w.write("\nhello")
w.write(x)
run this code and try to read the file before touch, it'll be empty, but after the with block is over you can see the contents.
If you are going to access the file from many sources, then you have to be careful of this, and also not to modify it from multiple sources.
EDIT: I forgot to say, you have to continuously close the file and open it in append mode if you want some other program to read it while you are writing to the file.

Reading from a CSV file while it is being written to

So before I start I know this is not the proper way to go about doing this but this is the only method that I have for accessing the data I need on the fly.
I have a system which is writing telemetry data to a .csv file while it is running. I need to see some of this data while it is being written but it is not being broadcast in a manner which allows me to do this.
Question: How do I read from a CSV file which is being written to safely.
Typically I would open the file and look at the values but I am hoping to be able to write a python script which is able to examine the csv for me and report the most recent values written without compromising the systems ability to write to the file.
I have absolutely NO access to the system or the manner in which it is writing to the CSV I am only able to see that the CSV file is being updated as the system runs.
Again I know this is NOT the right way to do this but any help you could provide would be extremely helpful.
This is mostly being run in a Windows environment

You can do something like:
tailf csv_file | python your_script.py
and read from sys.stdin

How do I exit without saving a file in Python

I am new to python and have until now written only a few programs to help with my job (I'm a sysadmin). I am writing this script now which will write the output of a MySQL query to a file. While in-between looping, I want to check for an extra condition and if the condition does not match, I want to close the file that I am writing to without saving what it has already written to the file. Like 'exit without saving'. I wrote this simple code to see if not closing the file with a close() will exit without saving, but it is creating the file with the content after I run and exit this code. So, is there a legal way in Python to exit a file without saving?
#/usr/bin/python
fo=open('tempfile.txt','a')
fo.write('This content\n')
P.S:- Python version is 2.4.3 (sorry, cannot upgrade)

There is no such concept in programming.
For the vast majority of the programming languages out there, the write command will attempt to put data directly in the file. This may or may not occur instantly for various reasons so many languages also introduce the concept of flush which will guarantee that your data is written to the file
What you want to do instead is to write all your data to a huge buffer (a string) then conditionally write or skip writing to the file.

Use the tempfile module to create your temporary, then if you need to save it you can do so explicitly using shutil.copyfileobj.
See Quick way to save a python TempFile?
Note that this is only if you absolutely need a temporary file (large amounts of data, etc.); if your contents are small then just using a stringbuffer and only writing it if you need to is a better approach.

Check for the condition before opening the file:
#/usr/bin/python
if condition():
fo=open('tempfile.txt','a')
fo.write('This content\n')

The safest way to do this is not to write to, or even open, the file until you know you want to save it. You could, for example, save what you might eventually write to a string, or list of same, so you can write them once you've decided to.

Consistent reading and writing a file in python

I'm a Python beginner and facing the following : I have a script periodically reading a settings file and doing something according to those settings . I have another script triggered by some UI that writes the settings file with user input values. I use the ConfigParser module to both read and write the file.
I am wondering if this scenario is capable of leading into an inconsistent state (like in middle of reading the settings file, the other script begins writing). I am unaware if there are any mechanism behind the scene to automatically protect against this situations.
If such inconsistencies are possible, what could I use to synchronize both scripts and mantain the integrity of the operations ?

I'm a Python beginner and facing the following : I have a script periodically reading a settings file and doing something according to those settings . I have another script triggered by some UI that writes the settings file with user input values.
There may be a race condition when the reader reads while the writer writes to the file, so that the reader may read the file while it is incomplete.
You can protect from this race by locking the file while reading and writing (see Linux flock() or Python lockfile module), so that the reader never observes the file incomplete.
Or, better, you can first write into a temporary file and when done rename it to the final name atomically. This way the reader and writer never block:
def write_config(config, filename):
tmp_filename = filename + "~"
with open(tmp_filename, 'wb') as file:
config.write(file)
os.rename(tmp_filename, filename)
When the writer uses the above method no changes are required to the reader.

When you write the config file write it to a temporary file first. When it's done, rename it to the correct name. The rename operation (os.rename) is normally implemented as an atomic operation on Unix systems, Linux and Windows, too, I think, so there will be no risk of the other process trying to read the config while the writing has not been finished yet.

There are al least two ways to address this issue (assuming you are on a unix-ish system):
If you want to write, write to a temporary file first, then do something unix can do atomically, especially rename the temporary file into place.
Lock the file during any operation, e.g. with the help of this filelock module.
Personally, I like the first option because it utilizes the OS, although some systems have had problems with the atomicity: On how rename is broken in Mac OS X - another limitation: the rename system call can not rename files across devices.

Python, subprocesses and text file creation

Apologies if this kind of thing has been answered elsewhere. I am using Python to run a Windows executable file using subprocess.Popen(). The executable file produces a .txt file and some other output files as part of its operation. I then need to run another executable file using subprocess.Popen() that uses the output from the original .exe file.
The problem is, it is the .exe file and not Python that is controlling the creation of the output files, and so I have no control over knowing how long it takes the first text file to write to disk before I can use it as an input to the second .exe file.
Obviously I cannot run the second executable file before the first text file finishes writing to disk.
subprocess.wait() does not appear to be helpful because the first executable terminates before the text file has finished writing to disk. I also don't want to use some kind of function that waits an arbitrary period of time (say a few seconds) then proceeds with the execution of the second .exe file. This would be inefficient in that it may wait longer than necessary, and thus waste time. On the other hand it may not wait long enough if the output text file is very large.
So I guess I need some kind of listener that waits for the text file to finish being written before it moves on to execute the second subprocess.Popen() call. Is this possible?
Any help would be appreciated.
UPDATE (see Neil's suggestions, below)
The problem with os.path.getmtime() is that the modification time is updated more than once during the write, so very large text files (say ~500 Mb) require a relatively large wait time in between os.path.getmtime() calls. I use time.sleep() to do this. I guess this solution is workable but is not the most efficient use of time.
On the other hand, I am having bigger problems with trying to open the file for write access. I use the following loop:
while True:
try:
f = open(file, 'w')
except:
# For lack of something else to put in here
# (I don't want to print anything)
os.path.getmtime(file)
else:
break
This approach seems to work in that Python essentially pauses while the Windows executable is writing the file, but afterwards I go to use the text file in the next part of the code and find that the contents that were just written have been wiped.
I know they were written because I can see the file size increasing in Windows Explorer while the executable is doing its stuff, so I can only assume that the final call to open(file, 'w') (once the executable has done its job) causes the file to be wiped, somehow.
Obviously I am doing something wrong. Any ideas?

There's probably many ways to do what you want. One that springs to mind is that you could poll the modification time with os.path.getmtime(), and see when it changes. If the modification date is after you called the executable, but still a couple seconds ago, you could assume it's done.
Alternatively, you could try opening the file for write access (just without actually writing anything). If that fails, it means someone else is writing it.

This all sounds so fragile, but I assume your hands are somewhat tied, too.
One suggestion that comes to mind is if the text file that is written might have a recognizable end-of-file marker to it. I created a text file that looks like this:
BEGIN
DATA
DATA
DATA
END
Given this file, I could then tell if "END" had been written to the end of the file by using os.seek like this:
>>> import os
>>> fp = open('test.txt', 'r')
>>> fp.seek(-4, os.SEEK_END)
>>> fp.read()
'END\n'