As part of a Python project on Windows, I need to use small files as a means to communicate different processes. Since the external process must be called with subprocess.run(program, args...), I can't simply obtain a file descriptor for my file and pass it as a parameter to the external process. Instead, I need a file with a name which can be accessed from the normal filesystem. Thus, I would like a simple way to create a temporary file which is stored in memory (RAM) instead of disk, and which has a name other external processes can use to access it. In Linux, this can be achieved with the function os.mkfifo(). However, this function is not available in Windows.
At the moment, I am simply using the tempfile module to create a temporary file which is stored in disk and deleted once it is no longer needed. Here is a small reproducible example for my code:
import tempfile
import subprocess
import os
fd = tempfile.NamedTemporaryFile(mode="w+t", delete=False) # Obtain file descriptor
file_path = fd.name # Obtain file path
# Write data (encoded as str) to the file and close it
fd.write(data)
fd.close()
# Access this same file from an external process
output = subprocess.run( [program, file_path], stdout=subprocess.PIPE).stdout.decode('utf-8')
# Delete the file
os.remove(file_path)
print("External process output:", output)
So, my question is the following: How can I change this code so that in line fd.write(data) the data is written to RAM instead of disk?
Related
I send many files over TCP from PC(windows) to Server(Linux).
When I process files on server sometimes I get error, since file is corrupted or has zero size, because it is still undergoes 'saving' to hard disc.
I process files in python, grabbing like this:
file_list = sorted(glob('*.bin'))
for file in file_list:
file_size = os.path.getsize(file)
if file_size > min_file_size:
do_process(file)
How to make it in proper way, i.e make sure, that file is ok.
I cant choose right min_file_size, since files have different sizes..
May be I should copy it to another folder ant then process them?
** I'm using SCP to copy files. So on the server side how can I be sure(some linux hints), that file is ok, to move it to directory, which will be processing? Sometimes by typing ls I see files, whch is not fully sent yet.. so how can I rename them?
You can use the fuser command to check whether the file is currently being accessed by any process, as follows:
import subprocess
...
file_list = sorted(glob('*.bin'))
for file in file_list:
result = subprocess.run(['fuser','--silent',file])
if result.returncode != 0:
do_process(file)
The fuser command will terminate with a non-0 return code if the file is not being accessed.
This has nothing to do with TCP. You are basically asking how to synchronize two processes in a way that if one writes the file the other will only use it once it is completely written and closed by the other.
One common way is to let the first process (writer) use a temporary file name which is not expected by the second process (reader) and to rename the file to the expected one after the file was closed. Other ways involve using file locking. One can also have a communication between the two processes (like a pipe or socketpair) which is used to explicitly inform the reader once the writer has finished and which file was written.
The following code works good on unix-base system, and use /tmp for store file, But how can I use it under windows OS family?
local_filename, headers = urllib.request.urlretrieve('http://127.0.0.1/translations/python-3.7.3.exe')
print(local_filename)
fd = open(local_filename)
fd.close()
From the docs:
Retrieve a URL into a temporary location on disk.
If you don't supply the filename argument, then urlretrieve() will call tempfile.NamedTemporaryFile() to create a file on whatever is the location for temporary files on your operating system. That works under Windows too (the file will be created under %TEMP%).
If you want a specific location, pass a filename argument.
How can my Python script reliably get its own location so it can write a file next to itself?
I made a Python script that writes a new file with some working statistics:
import os
NAME = __file__.split('/')[-1]
PATH = os.path.dirname(os.path.realpath(__file__))
PROPER_PATH = os.path.join(MY_PATH, MY_NAME)
with open(os.path.join(PATH, STATS_NAME), 'wb') as statsfile:
statsfile.write(mystats)
This works great and I can call it from anywhere and it writes the stats file in the same place, next to the script. Apart from when I call the script from a macro VBA script.
The script gets called okay but writes its data to:
"C:\Users\lsp\AppData\Roaming\Microsoft\Templates"
How do I make sure it writes to the correct path (same as the script path)?
As a work-around I can imagine giving the path as an argument and then providing it in the VBA but surely there's a Pythonic way to achieve the behavior?
As tempfile.mktemp is depreciated in Python 2.7 I generate a unique path to a temporary file as follows:
temp = tempfile.NamedTemporaryFile(suffix=".py")
path_to_generated_py = temp.name
temp.close()
# now I use path_to_gerated_py to create a python file
Is this the recommended way in Python 2.7? As I close the temp file immediately it looks like misusing NamedTemporaryFile....
The direct replacement for tempfile.mktemp() is tempfile.mkstemp(). The latter creates the file, like NamedTemporaryFile, so you must close it (as in your code snippet). The difference with NamedTemporaryFile is that the file is not deleted when closed. This is actually required: your version has a theoretical race condition where two processes might end up with the same temporary file name. If you use mkstemp() instead, the file is never deleted, and will likely be overwritten by the 3rd-party library you use --- but at any point in time, the file exists, and so there is no risk that another process would create a temporary file of the same name.
Is this usage of Python tempfile.NamedTemporaryFile secure (i.e. devoid security issues of deprecated tempfile.mktemp)?
def mktemp2():
"""Create and close an empty temporary file.
Return the temporary filename"""
tf = tempfile.NamedTemporaryFile(delete=False)
tfilename = tf.name
tf.close()
return tfilename
outfilename = mktemp2()
subprocess.call(['program_name','-o',outfilename])
What I need to run external command that requires output file name as one of the arguments. It overwrites the outfilename if that exists without warnings. I want to use temporary file as I just need to read its content, I don't need it later.
Totally unsafe. There is an opportunity for an attacker to create the file with whatever permissions they like (or a symlink) with that name between when it is deleted and opened by the subprocess
If you can instead create the file in a directory other than /tmp that is owned and onnly read/writeable by your process, you don't need to concern yourself with the security of the file as anything in the directory is protected