We Keep Coding

Why a new NamedTemporaryFile object has a path, but a file is not available? [duplicate]

I am attempting to create and write to a temporary file on Windows OS using Python. I have used the Python module tempfile to create a temporary file.
But when I go to write that temporary file I get an error Permission Denied. Am I not allowed to write to temporary files?! Am I doing something wrong? If I want to create and write to a temporary file how should should I do it in Python? I want to create a temporary file in the temp directory for security purposes and not locally (in the dir the .exe is executing).
IOError: [Errno 13] Permission denied: 'c:\\users\\blah~1\\appdata\\local\\temp\\tmpiwz8qw'
temp = tempfile.NamedTemporaryFile().name
f = open(temp, 'w') # error occurs on this line

NamedTemporaryFile actually creates and opens the file for you, there's no need for you to open it again for writing.
In fact, the Python docs state:
Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).
That's why you're getting your permission error. What you're probably after is something like:
f = tempfile.NamedTemporaryFile(mode='w') # open file
temp = f.name # get name (if needed)

Use the delete parameter as below:
tmpf = NamedTemporaryFile(delete=False)
But then you need to manually delete the temporary file once you are done with it.
tmpf.close()
os.unlink(tmpf.name)
Reference for bug: https://github.com/bravoserver/bravo/issues/111
regards,
Vidyesh

Consider using os.path.join(tempfile.gettempdir(), os.urandom(24).hex()) instead. It's reliable, cross-platform, and the only caveat is that it doesn't work on FAT partitions.
NamedTemporaryFile has a number of issues, not the least of which is that it can fail to create files because of a permission error, fail to detect the permission error, and then loop millions of times, hanging your program and your filesystem.

The following custom implementation of named temporary file is expanded on the original answer by Erik Aronesty:
import os
import tempfile
class CustomNamedTemporaryFile:
"""
This custom implementation is needed because of the following limitation of tempfile.NamedTemporaryFile:
> Whether the name can be used to open the file a second time, while the named temporary file is still open,
> varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).
"""
def __init__(self, mode='wb', delete=True):
self._mode = mode
self._delete = delete
def __enter__(self):
# Generate a random temporary file name
file_name = os.path.join(tempfile.gettempdir(), os.urandom(24).hex())
# Ensure the file is created
open(file_name, "x").close()
# Open the file in the given mode
self._tempFile = open(file_name, self._mode)
return self._tempFile
def __exit__(self, exc_type, exc_val, exc_tb):
self._tempFile.close()
if self._delete:
os.remove(self._tempFile.name)

This issue might be more complex than many of you think. Anyway this was my solution:
Make use of atexit module
def delete_files(files):
for file in files:
file.close()
os.unlink(file.name)
Make NamedTemporaryFile delete=False
temp_files = []
result_file = NamedTemporaryFile(dir=tmp_path(), suffix=".xlsx", delete=False)
self.temp_files.append(result_file)
Register delete_files as a clean up function
atexit.register(delete_files, temp_files)

tempfile.NamedTemporaryFile() :
It creates and opens a temporary file for you.
f = open(temp, 'w') :
You are again going to open the file which is already open and that's why you are getting Permission Denied error.
If you really wants to open the file again then you first need to close it which will look something like this-
temp= tempfile.NamedTemporaryFile()
temp.close()
f = open(temp.name, 'w')

Permission was denied because the file is Open during line 2 of your code.
close it with f.close() first then you can start writing on your tempfile

Python File I/O operations leads to “[Errno 23] Too many open files in system”

I am trying to iteratively process a list of compressed files. Each compressed file contains a large number of XML documents, some even contain up to 92,000 XML documents. Inside the nested loop, I am iterating over the list of extracted XML documents and pulling specific XML field values from each XML document. The problem is that I keep running into following issue after 5 minutes of the script running: [Errno 23] Too many open files in system.
When I uncomment the call to ‘parse_xml_files’ from the main function below, the script goes to completion without any issues. I am confused what is causing this issue since the files are being read using a context manager which should by itself close each XML files after they have been processed. May be I am overlooking something.
Many solutions on stackoverflow seems to be revolving around permanently increasing the limit of number of open files using ‘ulimit -a’. However, this solution does not work for me as I am dealing with such as a large number of files.
Any insight into this is greatly appreciated!
from os import system
from os import walk, path as ospath, makedirs
from bs4 import BeautifulSoup
from os import walk
def find_all_xml_files(path):
file_queue = []
for (dirpath, _, filenames) in walk(path):
if len(filenames) == 0:
continue
for filename in filenames:
if filename.endswith('.xml'):
file_queue.append(f'{dirpath}/{filename}')
else:
# Ignore unknown file formats
pass
return file_queue
def parse_xml_files(xml_files):
for xml_file in xml_files:
with open(xml_file, 'r') as f:
handler = f.read()
xml_doc = BeautifulSoup(handler, 'lxml')
""" And then extract and store various parts of the XML document """
def main():
# Retrieve list of files to process
compressed_files = get_files_to_process()
for compressed_file in compressed_files:
temp_folder = f'.temp' # Temporary directory to store extracted contents
# Create temporary storage file
makedirs(temp_folder)
# Unzip the compressed file
unzip_cmd = f'unzip -o {compressed_file} -d {temp_folder}'
system(unzip_cmd)
# Retrieve list of extracted XML files for parsing
xml_files = find_all_xml_files(temp_folder)
parse_xml_files(xml_files) # Commenting this line helps the script go to completion without any errors
# Delete temporary direcory and its content
delete_cmd = f'rm -r {temp_folder}'
system(delete_cmd)
main()
Update:
This is the Traceback:
Traceback (most recent call last):
File "main.py", line 36, in <module>
File "main.py", line 30, in main
File "main.py", line 8, in main
OSError: [Errno 23] Too many open files in system: '.temp/xyz.xml'
After the above traceback is shown, I am having to reboot my entire system because everything running on my laptop becomes unresponsive.

Download .pdf files from a webiste using python

i am trying to download all the pdfs from the webiste provided and i am using the following code:
import mechanize
from time import sleep
br = mechanize.Browser()
br.open('http://www.nerc.com/comm/CCC/Pages/AgendasHighlightsandMinutes-.aspx')
f=open("source.html","w")
f.write(br.response().read())
filetypes=[".pdf"]
myfiles=[]
for l in br.links():
for t in filetypes:
if t in str(l):
myfiles.append(l)
def downloadlink(l):
f=open(l.text,"w")
br.click_link(l)
f.write(br.response().read())
print l.text," has been downloaded"
for l in myfiles:
sleep(1)
downloadlink(l)
keep on getting the following error and can't figure out the problem why.
legal and privacy has been downloaded
Traceback (most recent call last):
File "downloads-pdfs.py", line 29, in <module>
downloadlink(l)
File "downloads-pdfs.py", line 21, in downloadlink
f=open(l.text,"w")
IOError: [Errno 13] Permission denied: u'/trademark policy'

The problem you encounter arises because you use the link URL as a filename. The character '/' is not valid in a filename. Try to modify your downloadlink function to something like this:
def downloadlink(l):
filename = l.text.split('/')[-1]
with open(filename, "w") as f:
br.click_link(l)
f.write(br.response().read())
print l.text," has been downloaded"

copy the snippet java file in middle of temp file in python

I am a beginner in python3
I want to copy snippet java file in the middle of other temp file which i gain the address of this file from downloading URL.
my problem is when i execute my program i have this error:
RESTART: C:/Users/user/AppData/Local/Programs/Python/Python36/refactordwon.py
the Url is:
('C:\\Users\\user\\AppData\\Local\\Temp\\tmpq5m7m_og', <http.client.HTTPMessage object at 0x0000003A854879E8>)
Traceback (most recent call last):
File "C:/Users/user/AppData/Local/Programs/Python/Python36/refactordwon.py", line 14, in <module>
file_out = open("path_file" , "r")
FileNotFoundError: [Errno 2] No such file or directory: 'path_file'
>>>
i do not know why?
because when i download the url, url shows me this address:
the Url is:
('C:\\Users\\user\\AppData\\Local\\Temp\\tmpey3yovte', <http.client.HTTPMessage object at 0x0000002233347978>)
i tried to use this address in different way but anyway i have error. i found the temp file and copied in python address.
I am sure that i have this file and the address is correct but again i have error that can not find file.
could you help me, please?!
I hope my question is clear
my code is:
import urllib.request
import os
import tempfile
#download URL
#[-------------------------
url = 'http://pages.di.unipi.it/corradini/Didattica/AP-17/PROG-ASS/03/assignment3.html'
gt_url = urllib.request.urlretrieve(url)
print("the Url is: ")
print(gt_url)
#--------------------------]
#copy sniper java file inside remote file
#[--------------------------
path_file =r'C:/Users/user/AppData/Local/Programs/Python/Python36/tmpokv2s_dw'
file_out = open("path_file" , "r")
file_in = open("snip1.java", "r")
file_out.readlines()
open("file_back", "w")
file_back.write(file_out)
pos_fileout = file_back.tell()
file_back.seek(pos_fileout)
file_back.write(file_in)
print("the content of file is: ")
file_back.close()
file_out.close()
file_in.close()
open("file_back", "r")
file_back.readlines()
print(file_back)
file_back.close()

Close already open csv in Python

Is there a way for Python to close that the file is already open file.
Or at the very least display a popup that file is open or a custom written error message popup for permission error.
As to avoid:
PermissionError: [Errno 13] Permission denied: 'C:\\zf.csv'
I've seen a lot of solutions that open a file then close it through python. But in my case. Lets say I left my csv open and then tried to run the job.
How can I make it so it closes the currently opened csv?
I've tried the below variations but none seem to work as they expect that I have already opened the csv at an earlier point through python. I suspect I'm over complicating this.
f = 'C:\\zf.csv'
file.close()
AttributeError: 'str' object has no attribute 'close'
This gives an error as there is no reference to opening of file but simply strings.
Or even..
theFile = open(f)
file_content = theFile.read()
# do whatever you need to do
theFile.close()
As well as:
fileobj=open('C:\\zf.csv',"wb+")
if not fileobj.closed:
print("file is already opened")
How do I close an already open csv?
The only workaround I can think of would be to add a messagebox, though I can't seem to get it to detect the file.
filename = "C:\\zf.csv"
if not os.access(filename, os.W_OK):
print("Write access not permitted on %s" % filename)
messagebox.showinfo("Title", "Close your CSV")

Try using a with context, which will manage the close (__exit__) operation smoothly at the end of the context:
with open(...) as theFile:
file_content = theFile.read()

You can also try to copy the file to a temporary file, and open/close/remove it at will. It requires that you have read access to the original, though.
In this example I have a file "test.txt" that is write-only (chmod 444) and it throws a "Permission denied" error if I try writing to it directly. I copy it to a temporary file that has "777" rights so that I can do what I want with it:
import tempfile, shutil, os
def create_temporary_copy(path):
temp_dir = tempfile.gettempdir()
temp_path = os.path.join(temp_dir, 'temp_file_name')
os.chmod(temp_path, 0o777); # give full access to the tempfile so we can copy
shutil.copy2(path, temp_path) # copy the original into the temp one
os.chmod(temp_path, 0o777); # replace permissions from the original file
return temp_path
path = "./test.txt" # original file
copy_path = create_temporary_copy(path) # temp copy
with open(copy_path, "w") as g: # can do what I want with it
g.write("TEST\n")

f = open("C:/Users/amol/Downloads/result.csv", "r")
print(f.readlines()) #just to check file is open
f.close()
# here you can add above print statement to check if file is closed or not. I am using python 3.5