We Keep Coding

Python doesn't release file after it is closed

What I need to do is to write some messages on a .txt file, close it and send it to a server. This happens in a infinite loop, so the code should look more or less like this:
from requests_toolbelt.multipart.encoder import MultipartEncoder
num = 0
while True:
num += 1
filename = f"example{num}.txt"
with open(filename, "w") as f:
f.write("Hello")
f.close()
mp_encoder = MultipartEncoder(
fields={
'file': ("file", open(filename, 'rb'), 'text/plain')
}
)
r = requests.post("my_url/save_file", data=mp_encoder, headers=my_headers)
time.sleep(10)
The post works if the file is created manually inside my working directory, but if I try to create it and write on it through code, I receive this response message:
500 - Internal Server Error
System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component.
I don't see the file appearing in the project window of PyCharm...I even used time.sleep(10) because at first, I thought it could be a time-related problem, but I didn't solve the problem. In fact, the file appears in my working directory only when I stop the code, so it seems the file is held by the program even after I explicitly called f.close(): I know the with function should take care of closing files, but it didn't look like that so I tried to add a close() to understand if that was the problem (spoiler: it was not)

I solved the problem by using another file
with open(filename, "r") as firstfile, open("new.txt", "a+") as secondfile:
secondfile.write(firstfile.read())
with open(filename, 'w'):
pass
r = requests.post("my_url/save_file", data=mp_encoder, headers=my_headers)
if r.status_code == requests.codes.ok:
os.remove("new.txt")
else:
print("File not saved")
I make a copy of the file, empty the original file to save space and send the copy to the server (and then delete the copy). Looks like the problem was that the original file was held open by the Python logging module

Firstly, can you change open(f, 'rb') to open("example.txt", 'rb'). In open, you should be passing file name not a closed file pointer.
Also, you can use os.path.abspath to show the location to know where file is written.
import os
os.path.abspath('.')
Third point, when you are using with context manager to open a file, you don't close the file. The context manger supposed to do it.
with open("example.txt", "w") as f:
f.write("Hello")

Why a new NamedTemporaryFile object has a path, but a file is not available? [duplicate]

I am attempting to create and write to a temporary file on Windows OS using Python. I have used the Python module tempfile to create a temporary file.
But when I go to write that temporary file I get an error Permission Denied. Am I not allowed to write to temporary files?! Am I doing something wrong? If I want to create and write to a temporary file how should should I do it in Python? I want to create a temporary file in the temp directory for security purposes and not locally (in the dir the .exe is executing).
IOError: [Errno 13] Permission denied: 'c:\\users\\blah~1\\appdata\\local\\temp\\tmpiwz8qw'
temp = tempfile.NamedTemporaryFile().name
f = open(temp, 'w') # error occurs on this line

NamedTemporaryFile actually creates and opens the file for you, there's no need for you to open it again for writing.
In fact, the Python docs state:
Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).
That's why you're getting your permission error. What you're probably after is something like:
f = tempfile.NamedTemporaryFile(mode='w') # open file
temp = f.name # get name (if needed)

Use the delete parameter as below:
tmpf = NamedTemporaryFile(delete=False)
But then you need to manually delete the temporary file once you are done with it.
tmpf.close()
os.unlink(tmpf.name)
Reference for bug: https://github.com/bravoserver/bravo/issues/111
regards,
Vidyesh

Consider using os.path.join(tempfile.gettempdir(), os.urandom(24).hex()) instead. It's reliable, cross-platform, and the only caveat is that it doesn't work on FAT partitions.
NamedTemporaryFile has a number of issues, not the least of which is that it can fail to create files because of a permission error, fail to detect the permission error, and then loop millions of times, hanging your program and your filesystem.

The following custom implementation of named temporary file is expanded on the original answer by Erik Aronesty:
import os
import tempfile
class CustomNamedTemporaryFile:
"""
This custom implementation is needed because of the following limitation of tempfile.NamedTemporaryFile:
> Whether the name can be used to open the file a second time, while the named temporary file is still open,
> varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).
"""
def __init__(self, mode='wb', delete=True):
self._mode = mode
self._delete = delete
def __enter__(self):
# Generate a random temporary file name
file_name = os.path.join(tempfile.gettempdir(), os.urandom(24).hex())
# Ensure the file is created
open(file_name, "x").close()
# Open the file in the given mode
self._tempFile = open(file_name, self._mode)
return self._tempFile
def __exit__(self, exc_type, exc_val, exc_tb):
self._tempFile.close()
if self._delete:
os.remove(self._tempFile.name)

This issue might be more complex than many of you think. Anyway this was my solution:
Make use of atexit module
def delete_files(files):
for file in files:
file.close()
os.unlink(file.name)
Make NamedTemporaryFile delete=False
temp_files = []
result_file = NamedTemporaryFile(dir=tmp_path(), suffix=".xlsx", delete=False)
self.temp_files.append(result_file)
Register delete_files as a clean up function
atexit.register(delete_files, temp_files)

tempfile.NamedTemporaryFile() :
It creates and opens a temporary file for you.
f = open(temp, 'w') :
You are again going to open the file which is already open and that's why you are getting Permission Denied error.
If you really wants to open the file again then you first need to close it which will look something like this-
temp= tempfile.NamedTemporaryFile()
temp.close()
f = open(temp.name, 'w')

Permission was denied because the file is Open during line 2 of your code.
close it with f.close() first then you can start writing on your tempfile

Merging PDF's with python pypdf and deleting merged files

I'm trying to write a program in python that takes a PDF file and appends to it first any pdf which includes the name of a fruit to it(Mango, Orange or Apple), then appends the pdf's with the names of animals to the original file(Zebra, Monkey, Dog) and finally appends any remaining PDF's. This is the code I have:
import os
from PyPDF2 import PdfFileReader, PdfFileMerger
originalFile="C:/originalFile.pdf"
merger = PdfFileMerger()
merger.append(PdfFileReader(file(originalFile, 'rb')))
os.remove(originalFile)
for filename in os.listdir('C:/'):
if "Mango" in filename or "Apple" in filename or "Orange" in filename:
if ".pdf" in filename:
merger.append(PdfFileReader(file('C:/'+filename, 'rb')))
os.remove("C:/"+filename)
for filename in os.listdir('C:/'):
if "Zebra" in filename or "Monkey" in filename or "Dog" in filename:
if ".pdf" in filename:
merger.append(PdfFileReader(file('C:/'+filename, 'rb')))
os.remove("C:/"+filename)
for filename in os.listdir('C:/'):
if ".pdf" in filename:
merger.append(PdfFileReader(file('C:/TRIAL/'+filename, 'rb')))
os.remove("C:/TRIAL/"+filename)
merger.write(originalFile)
When I run this program I get the following Error:
os.remove(originalFile)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:/originalFile.pdf'
Could anyone explain me how to close the file after I've added it to my merger file?

You should close the file explicitly.
fd = file('C:/'+filename, 'rb')
merger.append(PdfFileReader(fd))
fd.close()
os.remove('C:/'+filename)
A safer version:
fd = None
try:
fd = file('C:/'+filename, 'rb')
merger.append(PdfFileReader(fd))
finally:
if fd: fd.close()
if os.path.exists('C:/'+filename): os.remove('C:/'+filename)
Which can be simplified in Python 2.5+ as:
with file('C:/'+filename, 'rb') as fd:
merger.append(PdfFileReader(fd))
if os.path.exists('C:/'+filename): os.remove('C:/'+filename)
Which will cause python to close the file automagically.

To close a file, you should have opened it with with statement, which always closes the file whatever happens to the code inside the with block:
with open(originalFile,'rb') as pdf:
merger.append(PdfFileReader(pdf))
os.remove(originalFile)
This works for me.
Just a reminder that, you can close the file since you have added the pdf into the merger. Note that if you just open it with PdfFileReader(pdf) and haven't done anything to it, you can't delete the file or the PdfFileReader object won't be able to read the file. This is because the PdfFileReader only actually reads the file if you call some read method on it like getPage

Become originalFile has been opened, therefore, you cannot delete the file until you close it.
You need to modify your code like this:
merger = PdfFileMerger()
fin = file(originalFile, 'rb')
merger.append(PdfFileReader(fin))
fin.close()
os.remove(originalFile)

PyPDF merger now has a close method in as of in version 1.26.0
close()
Shuts all file descriptors (input and output) and clears all memory usage.
https://pythonhosted.org/PyPDF2/PdfFileMerger.html

Pdf merging isn't that hard in python. I see that you are already using PdfFileMerger. That should work as long as the pdf file exists, and the user who forks the python process has privileges to access the pdfs being merged. Good luck.

check if a file is open in Python

In my app, I write to an excel file. After writing, the user is able to view the file by opening it. But if the user forgets to close the file before any further writing, a warning message should appear. So I need a way to check this file is open before the writing process. Could you supply me with some python code to do this task?

If all you care about is the current process, an easy way is to use the file object attribute "closed"
f = open('file.py')
if f.closed:
print 'file is closed'
This will not detect if the file is open by other processes!
source: http://docs.python.org/2.4/lib/bltin-file-objects.html

I assume that you're writing to the file, then closing it (so the user can open it in Excel), and then, before re-opening it for append/write operations, you want to check that the file isn't still open in Excel?
This is how you could do that:
while True: # repeat until the try statement succeeds
try:
myfile = open("myfile.csv", "r+") # or "a+", whatever you need
break # exit the loop
except IOError:
input("Could not open file! Please close Excel. Press Enter to retry.")
# restart the loop
with myfile:
do_stuff()

For windows only
None of the other provided examples would work for me when dealing with this specific issue with excel on windows 10. The only other option I could think of was to try and rename the file or directory containing the file temporarily, then rename it back.
import os
try:
os.rename('file.xls', 'tempfile.xls')
os.rename('tempfile.xls', 'file.xls')
except OSError:
print('File is still open.')

You could use with open("path") as file: so that it automatically closes, else if it's open in another process you can maybe try
as in Tims example you should use except IOError to not ignore any other problem with your code :)
try:
with open("path", "r") as file: # or just open
# Code here
except IOError:
# raise error or print

Using
try:
with open("path", "r") as file:#or just open
may cause some troubles when file is opened by some other processes (i.e. user opened it manually).
You can solve your poblem using win32com library.
Below code checks if any excel files are opened and if none of them matches the name of your particular one, openes a new one.
import win32com.client as win32
xl = win32.gencache.EnsureDispatch('Excel.Application')
my_workbook = "wb_name.xls"
xlPath="my_wb_path//" + my_workbook
if xl.Workbooks.Count > 0:
# if none of opened workbooks matches the name, openes my_workbook
if not any(i.Name == my_workbook for i in xl.Workbooks):
xl.Workbooks.Open(Filename=xlPath)
xl.Visible = True
#no workbooks found, opening
else:
xl.Workbooks.Open(Filename=xlPath)
xl.Visible = True
'xl.Visible = True is not necessary, used just for convenience'
Hope this will help

Try this method if the above methods corrupt your excel file.
This function attempts to rename the file with its own name. If the file has already been opened, the edit will be reject by the os and an OSError exception will be raised. It does not touch the inner code so it will not corrupt your excel files. LMK if it worked for you.
def check_file_status(self):
try:
os.rename("file1.xlsx", "file1.xlsx")
print("File is closed.")
except OSError:
print("File is opened.")

if myfile.closed == False:
print("File is still open ################")

Just use this function. It will close any already opened excel file
import os
def close():
try:
os.system('TASKKILL /F /IM excel.exe')
except Exception:
print("KU")
close()

How to use tempfile.NamedTemporaryFile() in Python

I want to use tempfile.NamedTemporaryFile() to write some contents into it and then open that file. I have written following code:
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(contents)
tf.flush()
but I am unable to open this file and see its contents in Notepad or similar application. Is there any way to achieve this? Why can't I do something like:
os.system('start notepad.exe ' + tfName)
at the end.
I don't want to save the file permanently on my system. I just want the contents to be opened as a text in Notepad or similar application and delete the file when I close that application.

This could be one of two reasons:
Firstly, by default the temporary file is deleted as soon as it is closed. To fix this use:
tf = tempfile.NamedTemporaryFile(delete=False)
and then delete the file manually once you've finished viewing it in the other application.
Alternatively, it could be that because the file is still open in Python Windows won't let you open it using another application.
Edit: to answer some questions from the comments:
As of the docs from 2 when using delete=False the file can be removed by using:
tf.close()
os.unlink(tf.name)

You can also use it with a context manager so that the file will be closed/deleted when it goes out of scope. It will also be cleaned up if the code in the context manager raises.
import tempfile
with tempfile.NamedTemporaryFile() as temp:
temp.write('Some data')
temp.flush()
# do something interesting with temp before it is destroyed

Here is a useful context manager for this.
(In my opinion, this functionality should be part of the Python standard library.)
# python2 or python3
import contextlib
import os
#contextlib.contextmanager
def temporary_filename(suffix=None):
"""Context that introduces a temporary file.
Creates a temporary file, yields its name, and upon context exit, deletes it.
(In contrast, tempfile.NamedTemporaryFile() provides a 'file' object and
deletes the file as soon as that file object is closed, so the temporary file
cannot be safely re-opened by another library or process.)
Args:
suffix: desired filename extension (e.g. '.mp4').
Yields:
The name of the temporary file.
"""
import tempfile
try:
f = tempfile.NamedTemporaryFile(suffix=suffix, delete=False)
tmp_name = f.name
f.close()
yield tmp_name
finally:
os.unlink(tmp_name)
# Example:
with temporary_filename() as filename:
os.system('echo Hello >' + filename)
assert 6 <= os.path.getsize(filename) <= 8 # depending on text EOL
assert not os.path.exists(filename)