We Keep Coding

How to prevent reload from disk when using open() in Python?

This is my first post on StackOverflow. I'm also a beginner in Python. So I was just tinkering with the open() function, I was making a simple program to replace text in other .txt files. Here is my code:
f = open("file.txt", "r+")
f.truncate(0)
f.write("This text has been replaced.")
f.close()
print("Text replaced")
So, after running this program, the text in "file.txt" is getting changed. However, when I do ctrl + z, it's showing Undo Reload from Disk?, and when you click OK, the text gets back to normal.
How to prevent this? I am using Python 3.9, Pycharm code editor.
Thank you

I think your question might be more about the PyCharm UI editor.
If I understand correctly:
you have "file.txt" open in the PyCharm editor
you run the code above
At this point, the filesystem has the "file.txt" with the updated contents. The PyCharm editor (buffer) still has the old contents.
When you ctrl+z, PyCharm notices the filesystem has been updated and prompts to see how you want to proceed. When you click "OK", PyCharm writes its buffer to the file, which is the original contents.

Strange behavior when trying to create and write to a text file on macOS [duplicate]

This question already has answers here:
Convert UTF-8 with BOM to UTF-8 with no BOM in Python
(7 answers)
Closed last year.
I'm opening a plain text file, parsing it, and adding different lines to existing, empty string variables. I add these variables into a new variable that is a multi-line fstring. Trying to write the data to a new text file is not behaving as expected.
Reading the original file works fine. Text is properly parsed, variables populated.
The multi-line fstring variable seems fine. Prints normally. Even tried formatting it different ways which I show below.
When writing to a new file, that's where the strangeness starts. I've tried 2 ways:
Straight coding the open function with w or w+
Adding the above to a function and using that inside main()
The file is saved to disk with the correct name. Trying to double-click open in Finder produces nothing. Right-click to open produces nothing. Trying to move to trash with command+delete gives an error:
It sounds like the file goes to trash, but as the file disappears from the folder a new one is created with the same name in its place.
If I try to open in TextMate via File > Open, it opens as a blank file with no errors.
Since I can't get rid of the file, I have to delete the directory and create the directory again with the same name, or force delete in Terminal using rm. Restarting the system does not help. Relaunching Finder does nothing. Saving text files from other apps works fine. Directory is chmod 755.
If I copy an existing text file into the output directory, rename it to what the file is expected to be named, and let python overwrite the contents, it doesn't work either. The file modification date changes (and I see the file "blink" in Finder) but the contents remain the same. However, the file is not corrupted and opens normally.
If I do the same but delete the text inside of the copied file first, then run the script, python writes no data to the file, I can't open it by double-clicking on it, and I get error -43 again with the odd non-trashing behavior.
The strangest thing is this: if I add another with open() at the end of the script, and open the file that was just created and supposedly written to, and print its contents, the contents print. It's like when the script ends the file contents are being removed or its being corrupted somehow. Tried to close the file inside the script even though it's not needed, but same behavior persists.
Code:
Here's the code for writing:
FORMAT='utf-8'
OUTPUT_DIR = '/Path/To/SaveFolder'
# as a function
def write_to_file(content, fpath, name):
the_file = os.path.join(fpath, name)
with open(the_file, 'w+', encoding=FORMAT) as t:
t.write(content)
def main():
print(f" Writing File...\n")
filename = f"{pcode}_{author}_{title}_text.txt"
write_to_file(multiline_var, OUTPUT_DIR, filename)
# or hard coded in main()
def main():
print(f" Writing File...\n")
filename = f"{pcode}_{author}_{title}_text.txt"
the_file = os.path.join(OUTPUT_DIR, filename)
with open(the_file, 'w+', encoding=FORMAT) as t:
t.write(multiline_var)
I have tried using w w+ wt and wt+ and with and without encoding='utf-8'
Here is an example of multi-line fstring variable:
# using triple quotes
multiline_var = f"""
[PROJ-{pcode}] {full_title} by {author}
{description}
{URL}
{DIVIDER_1}
{TEXT_BLURB}
Some text here and then {SOME_MORE_TEXT}"
{DIVIDER_1}
{SOME_LINK}
"""
# or inside parens
multiline_var = (
f"[PROJ-{pcode}] {full_title} by {author}\n"
f"{description}\n\n"
f"{URL}\n"
f"{DIVIDER_1}\n"
f"{TEXT_BLURB}\n\n"
f"Some text here and then {SOME_MORE_TEXT}\n"
f"{DIVIDER_1}\n\n"
f"{SOME_LINK}"
)
Using exiftool on the text file shows the following, so it looks the data is there but must be corrupted:
File Size : 1797 bytes
File Modification Date/Time : 2021:12:31 15:55:39-05:00
File Access Date/Time : 2021:12:31 15:58:13-05:00
File Inode Change Date/Time : 2021:12:31 15:55:39-05:00
File Permissions : -rw-r--r--
File Type : TXT
File Type Extension : txt
MIME Type : text/plain
MIME Encoding : utf-8
Byte Order Mark : No
Newlines : Unix LF
Line Count : 55
Word Count : 181
Not sure what I'm doing wrong. VScode shows no syntax errors in the script. There are no errors in Terminal when running the script. Have I made some simple mistake in the above code? Maybe the fstring variable is causing a problem?

Thanks to #bnaecker for leading me to the solution to this problem.
It appeared that when creating/writing to a text file with a long name, Python can corrupt it. Not sure why, as I save long names for images with Python image libraries all the time. Using a short name like "MyFile.txt" it worked just fine, but that was a red herring.
I have updated this post with my journey to the final solution for using the long names that are needed for my project, though I'm not sure why the problem exists.
First Attempts:
So far creating using a short name and then renaming to a long one.... attempts have failed. I did notice that python is locking the file it creates and never unlocks it. Not sure if this is the problem. Setting chflags with os.system('chflags nouchg') command does not work, not even with sudo, and not even in the Terminal doing it manually.
Using os.rename() in Python corrupts the file
Using os.system('mv oldFile.txt newFile.txt') corrupts the file
Manually using mv command in Terminal corrupts the file
Manually changing the filename in the Finder does not (wtf?)
I kept looking for workarounds but nothing did the job.
Round 2:
Progress!
After much tinkering, I discovered a hidden character inside the file. I ran cat /path/longfilename.txt in Terminal, selected and copied the output and pasted into VScode. Here is what I saw:
Somehow a hidden character is getting into the project code number.
Pasting it into a Unicode search engine it came up as a ZERO WIDTH NO-BREAK SPACE also known in Unicode as EF BB BF. However, when pasting this symbol into TextMate it shows up as <U+FEFF> which is?...
The Byte Order Mark!
Opening a normal utf-8 text file in a hex editor also shows the files starting with EFBBBF for the BOM.
Now, the text file being read and parsed at first has no blank lines to start the file, so I added a line break, and also tried adding some spaces. This time when writing the file I could open it, however, after sending it to the trash, the same behavior occurred and the file was broken again. It seems that because other corrupted versions were in the trash, it added the symbol back to the file name for some reason.
So what appears to be happening, for whatever reason, when Python opens the text file I'm parsing that has no line break at the top, it seems to be grabbing the BOM from the file and adding that to the first variable which is grabbing the first line of the text file. Since that text is a number code that starts the file name, the BOM symbol is being added to the file name as well as the code inside the text file.
Just... wow
The Current Solution:
I have to leave a blank line at the start of the text file that I'm opening and parsing and a simple line break won't do it. I have no idea why this is. I added some spaces for good measure because randomly the BOM would be added to the variable and filename again. So far (knock on wood) as long as the first line of that initial file has some spaces and then a line break, and previous corrupted files have been deleted from the trash, a long file name can be used for all the files I'm creating and writing to without any problems.
This corruption even persists if I remove the encoding flag from both of the open functions I'm using (one to read and parse, the other to create and write).
If anyone knows why this is happening, please share. I've never seen it mentioned before. I'm not sure if it's a python 3.8 bug, a mac OS bug, the way TextMate wrote the original file, or a combination of these.
Correct Solution:
Thanks to #tripleee for the proper way to handle this, as I don't remember seeing this before, though I haven't been using python for very long.
In order to ignore the BOM, reading in the text file to be parsed with an encoding='utf-8-sig' does the job. Seems to be why it exists. :)
Problem solved.

Editing temporary file with Vim in Python subprocess not working as expected on Mac OS

My initial goal was to get user input via a command-line text-editor from within a Python script. More specifically, my plan was to create a temporary file and populate it with some pre-written text, open the file with a text-editor and allow the user to modify the file content, read the data from the file after the user exits the editor, then finally delete the file after it's all over.
I seem to have found a way to do this that is working for me, but along the way I tried a couple of approaches that did not work and I'd like to understand exactly why.
Consider the following Python script (a slightly modified version of the script taken from this post):
#!/usr/bin/env python2
# -*- encoding: ascii -*-
"""callvim.py
Demonstrates calling a text-editor (e.g. Vim) from within a Python script,
including passing input to the editor and reading output from the editor.
"""
import tempfile
import os
from subprocess import call
# Get the text editor from the shell, otherwise default to Vim
EDITOR = os.environ.get('EDITOR','vim')
# Set initial input with which to populate the buffer
initial_message = "Hello world!"
# Open a temporary file to communicate through
with tempfile.NamedTemporaryFile(suffix=".tmp") as tf:
# Write the initial content to the file I/O buffer
tf.write(initial_message)
# Flush the I/O buffer to make sure the data is written to the file
tf.flush()
# Open the file with the text editor
call([EDITOR, tf.name])
# Rewind the file offset to the beginning of the file
tf.seek(0)
# Read the file data into a variable
edited_message = tf.read()
# Output the data
print(edited_message)
I've tried running this script in two different environments so far: on a macOS computer (running macOS 10.12) and on a Debian computer (running Debian 8.8). Both computers have the same (minor) version of Vim installed (Vim 7.4).
When I run this script with EDITOR=vim on my Debian 8 (Jessie) machine it works as expected. I'm prompted with Vim and a buffer containing the string "Hello world!". After editing the buffer to contain the string "Goodbye world!", saving the file, and exiting Vim, I see the string "Goodbye world!" printed to the console.
When I run the same script on my macOS 10.12 (Sierra) machine it does not seem to work. The same procedure results in "Hello world!" being displayed on-screen - as if the file is being read before it is edited.
However if run the script on my Mac with EDITOR=nano then once again everything seems to work as expected.
I tried a few variations on this script using different methods from the tempfile module (e.g. using tempfile.TemporaryFile() and tempfile.mkstemp()) with the same results.
Now consider the following alternative script:
#!/usr/bin/env python2
# -*- encoding: ascii -*-
"""callvim.py
Demonstrates calling a text-editor (e.g. Vim) from within a Python script,
including passing input to the editor and reading output from the editor.
"""
import subprocess
import os
# Create a temporary file and write some default text
file_path = "tempfile"
file_handle = open(file_path, "w")
file_handle.write("Hello world!")
file_handle.close()
# Open the file with Vim
subprocess.call(["vim", file_path])
# Rewind to the beginning of the file
file_handle = open(file_path, 'r')
# Read the data from the file
data = file_handle.read()
# Close the temporary file
file_handle.close()
# Delete the temporary file
os.remove(file_path)
# Print the data
print(data)
This script, which avoids using the tempfile module, appears to be working consistently across both platforms.
So it seems that this script may be failing for some reason having to do with how Vim and the tempfile Python module interact on macOS. What's going on here?

This is happening because your second script closes the file handle before invoking vim, then opens a new one afterwards, whereas the first script doesn't. It has nothing to do with the tempfile module per se. This code works as expected:
import tempfile, os
from subprocess import call
initial_message = "Hello world!"
tf = tempfile.NamedTemporaryFile(suffix=".tmp", delete=False)
tf.write(initial_message)
tf.close()
call(['vim', tf.name])
tf = open(tf.name, 'r')
edited_message = tf.read()
tf.close()
os.unlink(tf.name)
print(edited_message)
Note the delete=False in the call to NamedTemporaryFile, which ensures that the file isn't deleted when we close it the first time (we have to delete it manually with os.unlink later).
I think what's going on here is that vim isn't editing your file in-place, it's writing to a swap file. When you save and quit, vim replaces the original file with the edited one, leaving your file handle pointing to the old, unedited one. There are ways to prevent vim from using swap files (see, e.g. here), but I wouldn't recommend them. Just remember to update your file handles after vim has done its business.

How do I allow opening of files that have Unicode characters in their filenames?

I have this Python script here that opens a random video file in a directory when run:
import glob,random,os
files = glob.glob("*.mkv")
files.extend(glob.glob("*.mp4"))
files.extend(glob.glob("*.tp"))
files.extend(glob.glob("*.avi"))
files.extend(glob.glob("*.ts"))
files.extend(glob.glob("*.flv"))
files.extend(glob.glob("*.mov"))
file = random.choice(files)
print "Opening file %s..." % file
cmd = "rundll32 url.dll,FileProtocolHandler \"" + file + "\""
os.system(cmd)
Source: An answer in my Super User post, 'How do I open a random file in a folder, and set that only files with the specified filename extension(s) should be opened?'
This is called by a BAT file, with this as its script:
C:\Python27\python.exe "C:\Programs\Scripts\open-random-video.py" cd
I put this BAT file in the directory I want to open random videos of.
In most cases it works fine. However, I can't make it open files with Unicode characters (like Japanese or Korean characters in my case) in their filenames.
This is the error message when the BAT file and Python script is run on a directory and opens a file with Unicode characters in its filename:
C:\TestDir>openrandomvideo.BAT
C:\TestDir>C:\Python27\python.exe "C:\Programs\Scripts\open-random-video.py" cd
The filename, directory name, or volume label syntax is incorrect.
Note that the filename of the .FLV video file in that log is changed from its original filename (소시.flv) to '∩╗┐' in the command line log.
EDIT: I learned that the above command line error message is due to saving the BAT file as 'UTF-8 with BOM'. Saving it as 'ANSI or UTF-16' shows the following message instead, but still does not open the file:
C:\TestDir>openrandomvideo.BAT
C:\TestDir>C:\Python27\python.exe "C:\Programs\Scripts\open-random-video.py" cd
Opening file ??.flv...
Now, the filename of the .FLV video file in that log is changed from its original filename (소시.flv) to '??.flv.' in the command line log.
I'm using Python 2.7 on Windows 7, 64-bit.
How do I allow opening of files that have Unicode characters in their filenames?

Just use Unicode literals e.g., u".mp4" everywhere. IO functions in Python will return Unicode filenames back if you give them Unicode input (internally they might use Unicode-aware Windows API):
import os
import random
videodir = u"." # get videos from current directory
extensions = tuple(u".mkv .mp4 .tp .avi .ts .flv .mov".split())
files = [file for file in os.listdir(videodir) if file.endswith(extensions)]
if files: # at least one video file exists
random_file = random.choice(files)
os.startfile(os.path.join(videodir, random_file)) # start the video
else:
print('No %s files found in "%s"' % ("|".join(extensions), videodir,))
If you want to emulate how your web browser would open video files then you could use webbrowser.open() instead of os.startfile() though the former might use the latter internally on Windows anyway.

The error when running the BAT file is because the BAT file itself is saved as "UTF-8 with BOM". The "" bytes are not a corrupted filename, they are the literal first bytes stored in the BAT file. Re-save the BAT file as ANSI or UTF-16, which are the only encodings supported for BAT files.

Either use Unicode literals as described by J. F. Sebastian, or use Python 3, which always uses Unicode.
(For Python 3, your script will need a minor modification: print is a function now, so you have to put parentheses around the parameter list.)

please familiarize yourself to add # -*- coding: utf-8 -*- in your source code,
so python understanding about your unicode.

Passing a Hebrew file name as command line argument in Windows

I have a small Python program. I use the the Windows registry to enable the opening of files using the right-click context menu. My registry entry:
C:\Users\me\projects\mynotepad\notepad.exe "%1"
When I try and open a file with a Hebrew name using my right-click context menu, I get the file name as question marks, and I get an exception while trying to get the file size.
Here is my code:
file_name = sys.argv[1]
file_size = os.path.getsize(unicode(file_name))
I have tried this:
file_name = sys.argv[1].decode("cp1255").encode('utf-8')
file_size = os.path.getsize(unicode(file_name))
But it didn't work.
Any advice?

Turns out it's a problem. See here for the solution. You need to resort to Windows API to get the arguments.