Maybe this is just how Python works but I think I'm missing something, I've looked around and I can't find anything addressing this.
Are text files only created after the process is terminated, or have I made a mistake?
Here's the code I'm using to generate the text file:
user_name_token = open("user_name_token.txt", "w")
If that is just how Python works and .txt files are only generated after the process is terminated, then there's no need to look at the code below. However, if I've made a mistake and that's why the .txt file is only generated after the process is terminated, please do let me know.
I've put what I feel to be the relevant code sequence below.
The following code, if the file exists, converts the text inside to a str value and leads to a greeting where the str value is used but it isn't relevant here. Instead, we'll carry on as if the file does not exist yet; and go to initial_greeting_and_initial_user_name_set() after initialization():
initialization()
def initialization():
global user_name
global initial_login_token
if os.path.isfile("user_name_token.txt") == True:
file = open("user_name_token.txt")
user_name = file.read().replace("\n", " ")
file.close
print(user_name)
time_based_greeting()
else:
initial_greeting_and_initial_user_name_set()
The following code allows the user to create the user name and leads to initial_greeting_and_inital_user_name_set_token_generation() when the button is pressed:
def initial_greeting_and_initial_user_name_set():
global initial_greeting_label
global initial_greeting_space0
global user_name
global initial_greeting_progression_button
initial_greeting_label = Label(root, text="Hello! It's nice to meet you, my name is Eve, what's yours?")
initial_greeting_space0 = Label(root, text=" ")
user_name = Entry(root)
initial_greeting_progression_button = Button(root,\
text="Enter",\
command=initial_greeting_and_initial_user_name_set_token_generation)
initial_greeting_label.pack()
initial_greeting_space0.pack()
user_name.pack()
initial_greeting_progression_button.pack()
The following code uses the user Entry to create a .txt file with the user name inside and loops back to initialization():
def initial_greeting_and_initial_user_name_set_token_generation():
global user_name_token
user_name_token = open("user_name_token.txt", "w")
user_name_token.write(user_name.get())
initialization()
Thank you for any help, especially given it's a long read if I have in-fact made a mistake and that's why the .txt file is only created after the window is closed.
Again, if it's just normal for .txt files to only be generated after the process is terminated and not when the code to create it has been run; then there's no need to take the above code into account.
I'm not sure that's the problem, but in that code:
initialization()
def initialization():
global user_name
global initial_login_token
if os.path.isfile("user_name_token.txt") == True:
file = open("user_name_token.txt")
user_name = file.read().replace("\n", " ")
file.close
print(user_name)
time_based_greeting()
else:
initial_greeting_and_initial_user_name_set()
It seems to me like you forgot to call the file.close method.
Add brackets at the end, so it will look as follows:
file.close().
It is unclear just what your code is doing as you're opening a file for writing and then opening it again for reading before you've closed it for writing. You should close your file for writing before you go on to open it for reading:
def initial_greeting_and_initial_user_name_set_token_generation():
global user_name_token
user_name_token = open("user_name_token.txt", "w")
user_name_token.write(user_name.get())
user_name_token.close()
initialization()
or better yet:
def initial_greeting_and_initial_user_name_set_token_generation():
global user_name_token
with open("user_name_token.txt", "w") as user_name_token:
user_name_token.write(user_name.get())()
initialization()
This latter construct is the preferred way to open a file such that you're sure it gets closed. It will remain open only inside the with block and will then be closed automatically.
Python does not hold back any I/O operations. Like other languages, it performs the operations right away. It may buffer up write operations and write the results in larger blocks, but all writing is completed when you call close() on the file (or when the associated with block is exited, closing the file automatically).
Related
Is there some way to "capture" all attempted writes to a particular file /my/special/file, and instead write that to a BytesIO or StringIO object instead, or some other way to get that output without actually writing to disk?
The use case is: there's a 'handler' function, whose contract is that it should write its output to /my/special/file. I don't have any control over this handler function -- I don't write it, I don't know its contents and I can't change its contents, and the contract cannot change. I'd like to be able to do something like this:
# 'output' has whatever 'handler' has written to `/my/special/file`
output = handler.run(data)
Even if this is an odd request, I'd like to be able to do this even with a 'hackier' answer.
EDIT: my code (and handler) will be invoked many times on a lot of chunks of data, so performance (both latency and throughput) are important.
Thanks.
If you're talking about code in your own Python program, you could monkey-patch the built in open function before that code gets called. Here's a really stupid example, but it shows that you can do this. This causes code that thinks it's writing to a file to instead write into an in-memory buffer. The calling code then prints what the foreign code wrote to the file:
import io
# The function you don't have access to that writes to a file
def foo():
f = open("/tmp/foo", "w")
f.write("blahblahblah\n")
f.close()
# The buffer to contain the captured text
capture_buffer = ""
# My silly file-like object that only handles write(str) and close()
class MyFileClass:
def write(self, str):
global capture_buffer
capture_buffer += str
def close(self):
pass
# patch open to return a MyFileClass instance
def my_open2(*args, **kwargs):
return MyFileClass()
open = my_open2
# Call the target function
foo()
# Print what the function wrote to "the file"
print(capture_buffer)
Result:
blahblahblah
Sorry for not spending more time with this. Just showing you it's possible. As others say, a mocking module might be the way to go to not have to grow your own thing here. I don't know if they allow access to what is written. I guess they must. Such a module is just going to do a better job of what I've shown here.
If your program does other file IO with open, or whichever method the mystery code uses to open the file, you'd check the incoming path and only return your special object if it was the one path you're interested in. Otherwise, you could just call the original open, which you could stash away under another name.
I have a function that watches a csv file. It iterates through all the lines in a csv file and returns each line, waits for the file to be updated, and if it is, returns that new value. This is how it looks like:
def follow():
file_path = filedialog.askopenfilename()
with open(file_path) as csvDataFile:
csvReader = csv.reader(csvDataFile)
csvDataFile.seek(0,0)
while True:
line = csvDataFile.readline()
if not line:
time.sleep(0.5)
continue
yield line
I am doing this for a gui. I have a button that, when pressed, loads the csv file and calls this function. It saves each line to a list that is a member variable.
def browseForFile(self):
line = pull_csv_data.follow()
for item in line:
self.list.append(item)
When another button is pressed, it iterates through that list and displays the information on the gui:
def listItems(self):
for i in self.list:
time.sleep(0.1)
item = QListWidgetItem(i)
self.prev_scenes.addItem(item)
The problem is, whenever I click the button to display the data in the list, it stops responding until I kill the python script that has the follow() function. For clarification, the follow function is in a separate file that I include in my "main" file.
Essentially.. the follow function does not allow for other processes to run at the same time, I think. Is that true? is there a workaround? Or is there a better way to do this?
The follow function is in the same process as your main function. Because of this, when your follow function sleeps, so does your entire program. If you want this to be run asynchronously, you'll have to make it do so explicitly.
You should probably use something like https://pythonhosted.org/watchdog/index.html or https://github.com/seb-m/pyinotify/wiki instead of trying to write your own file handling function.
I'm running a program the takes in data from other clients, and have been having an enormous amount of problems writing, and changing information in a file, and I feel like I have tried everything. I want to save the information in case the program stops for some reason, and so the data would have saved. I feel like i have tried everything, using file.flush, using os.fsync() with it, I have tried using with open(file) as file: statements to close the file when the program stops, and currently, I's trying atexit to have a function write to the file when it closes, which hasn't worked out, plus doesn't call on errors, so is kinda irrelevant. I'm looking for a way to write to a file, repeatedly, and, well, work. I may not understand something, so please explain it to me. I have been having trouble without end, and need help.
EDIT
AccData = {}
client = discord.Client()
User = discord.User
def SaveData():
pickle.dump(AccData,data)
data.close()
print("data saved")
atexit.register(SaveData)
f = open('DisCoin.json','rb')
AccData = pickle.load(open('DisCoin.json','rb'))
f.seek(0)
f.close()
data = open('DisCoin.json','wb')
Python catches its own exceptions, most signals and exit() then runs atexit routines for cleanup. So, you can deal with normal badness there.
But other bad things happen. A segmenation fault or other internal error. An unknown signal. Code that calls os._exit(). These will cause an early termination and data not yet flushed will be lost. Bad things can happen to any program and if they need extra resiliency, they need some method to handle that.
You can write things to temporary files and rename them to the "live" file only when they are complete. If a program terminates, at least its last saved data is still there.
You can write a log or journal of changes and rebuild the data you want by scanning that log. That's how many file systems work, and "Big Data" map/reduce systems to basically the same thing.
You can move to a database and use its transaction processing or any OLPT system to make sure you do all-or-none updates to your data store.
Your example code is especially fragile because
data = open('DisCoin.json','wb')
trashes existing data on disk. There is no going back with this code! Step one, then, is don't do that. Keep old data until the new stuff is ready.
Here is an example class that manages temporary files for you. Use it instead of open and it will create a temporary file for you to update and will only go live with the data of the with clause exits without an exception. There is no need for an atexit handler if you use this in a with clause.
import shutil
import os
class SidelineFile:
def __init__(self, *args, **kw):
self.args = list(args)
self.kw = kw
def __enter__(self):
self.closed = False
self.orig_filename = self.args[0]
self.args[0] += '.tmp'
try:
mode = self.args[1]
except IndexError:
try:
mode = self.kw['mode']
except KeyError:
mode = 'r'
if 'a' in mode:
shutil.copy2(self.orig_filename, self.args[0])
self.file_obj = open(*self.args, **self.kw)
return self.file_obj
def __exit__(self, exc_type, exc_value, traceback):
if not self.closed:
self.file_obj.close()
self.closed = True
if not exc_type:
os.rename(self.args[0], self.orig_filename)
else:
os.remove(self.args[0])
fn = 'test.txt'
with SidelineFile(fn, 'w') as fp:
fp.write("foo")
print(1, repr(open(fn).read()))
with SidelineFile(fn, mode='a') as fp:
fp.write("bar")
print(2, repr(open(fn).read()))
with SidelineFile(fn, 'w') as fp:
fp.write("foo")
print(3, repr(open(fn).read()))
try:
with SidelineFile(fn, 'a') as fp:
fp.write("bar")
raise IndexError()
except IndexError:
pass
print(4, repr(open(fn).read()))
Personally, I like to achieve this by defining a print function for it.
import os
def fprint(text,**kwargs):
os.chdir('C:\\mypath')
myfile=open('output.txt','a')
if kwargs:
print(text,end=kwargs['end'],file=myfile)
else:
print(text,file=myfile)
myfile.close()
fprint('Hello')
input()
fprint('This is here too',end='!!\n')
The above code will write 'Hello' into the file 'output.txt' at C:\mypath, save it, then after you enter some input will write 'This is here too!!' into the file. If you check the file while the script is waiting for input, it should already contain 'Hello'.
I'd like to ask if there's any way we could watch a directory in python and parse the latest text file being generated on the directory.
Tho i have this start up code which parse a certain text file.
import time
def follow(thefile):
thefile.seek(0,2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
if __name__ == '__main__':
logfile = open(r'\\some directory\files.txt',"r")
loglines = follow(logfile)
for line in loglines:
print line,
See the bold files.txt i need that to be dynamic by watching the directory for newly generated text files and switch to the latest text file and parse it.
It will run on Windows XP service Pack 3
I'm using Python 2.7
Directory i'm watching is also using windows XP
Thank you.
To check for new files, repeatedly get a list of files currently in the directory with os.listdir('directory'). Save the entries in a set and calculate the difference of the set with the previous set.
# Initialize before an event loop:
old_entries = set()
# You need a loop that calls two handlers, each handler returning soon.
# Inside your loop, check for a "new file" event this way:
now_entries = os.listdir(r'\\some directory')
now_entries.symmetric_difference_update(old_entries)
for new_entry in now_entries:
handle_new_file(new_entry)
Your program needs to listen for two events:
New file in the directory.
New line in the old file.
You call follow(), which is like an event handler that never returns. I think you want that handler to return to one main event loop that checks for each kind of event. Your follow() function never returns because it continues within the while True infinite loop unless a new line is added to the file for it to yield. It will never yield if no more lines are getting added to that file.
Take a look into the FindFirstChangeNotification API
http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
The approach here is to use the MS FindFirstChangeNotification API, exposed via the pywin32 win32file module. It needs a little explanation: you get a change handle for a directory (optionally with its subdirectories) for certain kinds of change. You then use the ubiquitous WaitForSingleObject call from win32event, which fires when something's changed in one of your directories.
Essentially because the Windows OS is responsible for managing creation/modification of files, you can ask it to let you know immediately when a file is changed/created.
I am trying to make a program(in python) that as I write it writes to a file and opens to a certain window that I have already created.I have looked allarund for a vaible soution bt it would seem that multi-threading may be the only option.
I was hoping that when option autorun is "activated" it will:
while 1:
wbuffer = textview.get_buffer()
text = wbuffer.get_text(wbuffer.get_start_iter(), wbuffer.get_end_iter())
openfile = open(filename,"w")
openfile.write(text)
openfile.close()
I am using pygtk and have a textview window, but when I get the buffer it sits forever.
I am thinking that I need to multi-thread it and queue it so one thread will be writing the buffer while it is being queued.
my source is here. (I think the statement is at line 177.)
any help is much appreciated. :)
and here is the function:
def autorun(save):
filename = None
chooser = gtk.FileChooserDialog("Save File...", None,
gtk.FILE_CHOOSER_ACTION_SAVE,
(gtk.STOCK_CANCEL, gtk.RESPONSE_CANCEL,
gtk.STOCK_SAVE, gtk.RESPONSE_OK))
response = chooser.run()
if response == gtk.RESPONSE_OK: filename = chooser.get_filename()
filen = filename
addr = (filename)
addressbar.set_text("file://" + filename)
web.open(addr)
chooser.destroy()
wbuffer = textview.get_buffer()
while 1:
text = wbuffer.get_text(wbuffer.get_start_iter(), wbuffer.get_end_iter())
time.sleep(1)
openfile = open(filename,"w")
openfile.write(text)
openfile.close()
Though not too easy to see exactly what your GTK-stuff not included here is doing, the main problem is that the control needs to be returned to the gtk main-loop. Else the program will hang.
So if you have a long process (like this eternal one here), then you need to thread it. The problem is that you need the thread to exit nicely when the main program quits, so you'll have to redesign a bit around that. Also, threading with gtk needs to be initialized correctly (look here).
However, I don't think you need threading, instead you could connect the changed signal of your TextBuffer to a function that writes the buffer to the target-file (if the user has put the program in autorun-mode). A problem with this is if the buffer gets large or program slow, in which case, you should consider threading the callback of the changed signal. So this solution requires to make sure you don't get into the situation where save-requests get stacked on top of each other because the user is faster at typing than the computer is saving. Takes some design thought.
So, finally, the easier solution: you may not want the buffer to save for every button-press. In which case, you could have the save-function (which could look like your first code-block without the loop) on a timeout instead. Just don't make the time-out too short.