We Keep Coding

How to solve newly created directories not visible on another process in Python

I have a Python script that creates a directory on local NTFS SSD disk on a computer running Windows.
os.makedirs(path)
After creating the directory
os.path.isdir(path)
returns true.
But when i start a process using multiprocessing.Pool's starmap, sometimes os.path.isdir(path) returns false for the same path in the function executed in the pool process. The failure occurs quite randomly, and i haven't spotted any obvious reason for it.
The directory is created in the same thread that starts the processes. There are multiple processes writing files to the directory, but each of them has it's own file and none of them modifies the directory in any other way.
I've tried solving this using retries and delays, but would like to know whether there's a correct solution to this problem?
Are the filesystem's records somehow outdated in the other processes at this time? And if so, is there any way to ensure they're updated before attempting to access any files or directories?
Here's a minimal example with all the relevant bits. The system processes "Works". Whenever a new work arrives, the system launches a thread that schedules processing in a process pool. The processing itself consists of writing a file the the directory causing problems i.e. the processes don't seem to find the directory.
Note: i haven't been able to reproduce this problem in the original environment nor with this example. I just see in the logs, that it occurs every now and then.
#!/usr/bin/python
import os
import sys
import time
import random
import shutil
import threading
import multiprocessing
def create_dir(path):
os.makedirs(path)
# This never fails
if not os.path.isdir(path):
raise Exception(f"Unable to create directory '{path}'")
def clean_dir(path):
for f in os.listdir(path):
filepath = os.path.join(path, f)
if os.path.isdir(filepath):
shutil.rmtree(filepath, ignore_errors = True)
else:
os.remove(filepath)
# Creates a file to the problem directory. Crashes if
# the directory doesn't exist (or isn't visible).
# This function is executed in a process.
def process(i, tmp_dir):
print(f"Processing {i} / {tmp_dir}")
# Most of the work is done here before accessing path
time.sleep (1 + 3 * random.random ())
# Check Foo/Work-id/Temp
# This fails every now and then, and when
# it fails, it fails for every child process
if not os.path.isdir(tmp_dir):
raise Exception(f"Directory '{tmp_dir}' doesn't exist")
filepath = os.path.join(tmp_dir, f"File-{i}.txt")
with open(filepath, 'w') as file:
file.write(f"{i}")
# Starts 3 processes (from a pool of 8) each writing a single file.
# This function itself is executed in a thread.
def start_processing(target_dir, pool):
tmp_dir = os.path.join(target_dir, "Temp")
# This never fails, create Foo/Work-id/Temp
if not os.path.exists(tmp_dir):
create_dir(tmp_dir)
num_processes = 3
results = pool.starmap(process, enumerate([tmp_dir] * num_processes))
# Creates directory Foo to the current directory. And starts 20 works
# to be executed in a process pool, 8 at a time.
if __name__ == "__main__":
pool = multiprocessing.Pool(processes = 8)
root = "Foo"
seed = int (time.time())
print(f"Seed {seed}")
# This never fails
if not os.path.exists(root):
create_dir(root)
else:
clean_dir(root)
time.sleep(2)
num_works = 20
for work_id in range (1, num_works + 1):
name = f"Work-{work_id}"
print(f"Starting {name}/{num_works})")
target_dir = os.path.join(root, name)
# Create target directory Foo/Work-id
if not os.path.exists(target_dir):
create_dir(target_dir)
thread = threading.Thread(
target = start_processing,
args = (target_dir, pool),
name = "Processing Thread"
)
thread.start()
time.sleep(0.07)

How can I check if a file is actively open in another process outside of my python script with python3.9 in Windows 10?

I am using the following function to allow the OS to open a third party application associated with the filetype in question. For example: If variable 'fileToOpen' links to a file (it's full path of course) called flower.psd, this function would open up Photoshop in Windows and Gimp in Linux (typically).
def launchFile(fileToOpen):
if platform.system() == 'Darwin': # macOS
subprocess.call(('open', fileToOpen))
elif platform.system() == 'Windows': # Windows
os.startfile(fileToOpen)
else: # linux variants
subprocess.call(('xdg-open', fileToOpen))
While it is running, I want to have the same python script monitor the use of that file and delete it once the third party app is done using it (meaning...the 3rd party app closed the psd file or the third party app itself closed and released the file from use).
I've tried using psutil and pywin32 but neither seem to work in Windows 10 with Python3.9. Does anyone have any success with this? If so, how did you go about getting the process of the third party app while not getting a permission error from Windows?
Ideally, I would like to get a solution that works across Windows, Macs, and Linux but I'll take any help with Windows 10 for now since Mac and Linux can be found easier with commandline assistance with the ps -ax | grep %filename% command
Keep in mind, this would ideally track any file. TIA for your help.
Update by request:
I tried adding this code to mine (from a previous suggestion). Even this alone in a python test.py file spits out permission errors:
import psutil
for proc in psutil.process_iter():
try:
# this returns the list of opened files by the current process
flist = proc.open_files()
if flist:
print(proc.pid,proc.name)
for nt in flist:
print("\t",nt.path)
# This catches a race condition where a process ends
# before we can examine its files
except psutil.NoSuchProcess as err:
print("****",err)
The follow code below does not spit out an error but does not detect a file in use:
import psutil
from pathlib import Path
def has_handle(fpath):
for proc in psutil.process_iter():
try:
for item in proc.open_files():
if fpath == item.path:
return True
except Exception:
pass
return False
thePath = Path("C:\\Users\\someUser\\Downloads\\Book1.xlsx")
fileExists = has_handle(thePath)
if fileExists :
print("This file is in use!")
else :
print("This file is not in use")

Found it!
The original recommendation from another post forgot one function..."Path" The item.path from the process list is returned as a string. This needs to convert to a Path object for comparison of your own path object.
Therefore this line:
if fpath == item.path:
Should be:
if fpath == Path(item.path):
and here is the full code:
import psutil
from pathlib import Path
def has_handle(fpath):
for proc in psutil.process_iter():
try:
for item in proc.open_files():
print (item.path)
if fpath == Path(item.path):
return True
except Exception:
pass
return False
thePath = Path("C:\\Users\\someUser\\Downloads\\Book1.xlsx")
fileExists = has_handle(thePath)
if fileExists :
print("This file is in use!")
else :
print("This file is not in use")
Note: The reason to use Path objects rather than a string is to stay OS independant.

Based on #Frankie 's answer I put together this script. The script above took 16.1 seconds per file as proc.open_files() is quite slow.
The script below checks all files in a directory and returns the pid related to each open file. 17 files only took 2.9s to check. This is due to only calling proc.open_files() if the files default app is open in memory.
As this is used to check if a folder can be moved, the pid can be later used to force close the locking application but BEWARE that that application could have other documents open and all data would be lost.
This does not detect open txt files or may not detect files that dont have a default application
from pathlib import Path
import psutil
import os
import shlex
import winreg
from pprint import pprint as pp
from collections import defaultdict
class CheckFiles():
def check_locked_files(self, path: str):
'''Check all files recursivly in a directory and return a dict with the
locked files associated with each pid (proocess id)
Args:
path (str): root directory
Returns:
dict: dict(pid:[filenames])
'''
fnames = []
apps = set()
for root, _, f_names in os.walk(path):
for f in f_names:
f = Path(os.path.join(root, f))
if self.is_file_in_use(f):
default_app = Path(self.get_default_windows_app(f.suffix)).name
apps.add(default_app)
fnames.append(str(f))
if apps:
return self.find_process(fnames, apps)
def find_process(self, fnames: list[str], apps: set[str]):
'''find processes for each locked files
Args:
fnames (list[str]): list of filepaths
apps (set[str]): set of default apps
Returns:
dict: dict(pid:[filenames])
'''
open_files = defaultdict(list)
for p in psutil.process_iter(['name']):
name = p.info['name']
if name in apps:
try:
[open_files[p.pid].append(x.path) for x in p.open_files() if x.path in fnames]
except:
continue
return dict(open_files)
def is_file_in_use(self, file_path: str):
'''Check if file is in use by trying to rename it to its own name (nothing changes) but if
locked then this will fail
Args:
file_path (str): _description_
Returns:
bool: True is file is locked by a process
'''
path = Path(file_path)
if not path.exists():
raise FileNotFoundError
try:
path.rename(path)
except PermissionError:
return True
else:
return False
def get_default_windows_app(self, suffix: str):
'''Find the default app dedicated to a file extension (suffix)
Args:
suffix (str): ie ".jpg"
Returns:
None|str: default app exe
'''
try:
class_root = winreg.QueryValue(winreg.HKEY_CLASSES_ROOT, suffix)
with winreg.OpenKey(winreg.HKEY_CLASSES_ROOT, r'{}\shell\open\command'.format(class_root)) as key:
command = winreg.QueryValueEx(key, '')[0]
return shlex.split(command)[0]
except:
return None
old_dir = r"C:\path_to_dir"
c = CheckFiles()
r = c.check_locked_files(old_dir)
pp(r)

Python script won't print when called from another python script

I have two scripts. The first script outputs to a file a list of folders and files, then it calls a second python script to read that file and print it to screen. The second script is called but nothing ever prints to the screen and I'm not sure why. No error message is thrown.
First Script:
#!/bin/python
from subprocess import call
import os.path
import os
def main():
userRequest=raw_input("""Type the path and folder name that you'd like to list all files for.
The format should begin with a slash '/' and not have an ending slash '/'
Example (/var/log) *Remember capital vs. lower case does matter* :""")
userInputCheck(userRequest)
def userInputCheck(userRequest):
lastCharacter=userRequest[-1:]
if lastCharacter=="/":
userRequest=userRequest[:-1]
folderCheck=os.path.isdir(userRequest)
if folderCheck != True:
print("\nSorry, '"+userRequest+"' does not exist, please try again.\n")
requestUserInput()
else:
extractFileList(userRequest)
def extractFileList(userRequest):
fileList=open('/tmp/fileList.txt', 'a')
for folderName, subFolderName, listFiles in os.walk(userRequest):
fileList.write(folderName+":\n")
for fileName in listFiles:
fileList.write(fileName+"\n")
fileList.write("\n")
fileList.close
os.system("readFile.py /tmp/fileList.txt")
if os.path.isfile("/tmp/fileList.txt"):
os.remove("/tmp/fileList.txt")
if __name__ == "__main__":
main()
Second Script:
#!/bin/python
import sys
userFile=sys.argv[1]
f = open(userFile, 'r')
fileInfo=f.read()
sys.stdout.write(fileInfo)
sys.stdout.flush()
f.close

Code that only execute once, Python Startup folder

Hey and thanks for all of your answers. I try to write a piece of python code that only executes once, (first time the program is installed) and copies the program into the windows startup folders.
(C:\Users\ USER \AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup)
That's the code i wrote for this. (Please don't judge me. I know it's
very shitty code. But I'm very new to coding. (this is the second
little program i try to write)
import os
import shutil
#get username
user = str(os.getlogin())
user.strip()
file_in = ('C:/Users/')
file_in_2 = ('/Desktop/Py Sandbox/test/program.py')
file_in_com = (file_in + user + file_in_2)
folder_seg_1 = ('C:/Users/')
folder_seg_2 = ('/AppData/Roaming/Microsoft/Windows/Start Menu/Programs/Startup')
#create FolderPath
folder_com = (folder_seg_1 + user + folder_seg_2)
shutil.copy2(file_in_com, folder_com)
Because i got an error, that there is no such internal, external,
command, program or batch file named Installer. I tried to generate a batch file with
nothing in it that executes when the installation process is finished.(But the error is still there.)
save_path = 'C:/Windows/assembly/temp'
name_of_file = str("Installer")
completeName = os.path.join(save_path, name_of_file+".bat")
file1 = open(completeName, "w")
file1.close()
The main idea behind this that there is my main Program, you execute
it it runs the code above and copies itself to the startup folder.
Then the code the whole installer file gets deleted form my main
program.
import Installer
#run Installer File
os.system('Installer')
os.remove('Installer.py')
But maybe there's someone out there who knows the answer to this problem.
And as I said earlier, thanks for all of your answers <3.
BTW I'm currently using Python 3.5.

Okay guys now I finally managed to solve this problem. It's actually not that hard but you need to think from another perspective.
This is now the code i came up with.
import os
import sys
import shutil
# get system boot drive
boot_drive = (os.getenv("SystemDrive"))
# get current Username
user = str(os.getlogin())
user.strip()
# get script path
script_path = (sys.argv[0])
# create FolderPath (Startup Folder)
folder_seg_1 = (boot_drive + '/Users/')
folder_seg_2 = ('/AppData/Roaming/Microsoft/Windows/Start Menu/Programs/Startup')
folder_startup = (folder_seg_1 + user + folder_seg_2)
#check if file exits, if yes copy to Startup Folder
file_name = (ntpath.basename(script_path))
startup_file = (folder_startup + ("/") + file_name)
renamed_file = (folder_startup + ("/") + ("SAMPLE.py"))
# checkfile in Startup Folder
check_path = (os.path.isfile(renamed_file))
if check_path == True:
pass
else:
shutil.copy2(script_path, folder_startup)
os.rename(startup_file, renamed_file)
This script gets your username, your boot drive, the file location of
your file than creates all the paths needed. Like your personal
startup folder. It than checks if there is already a file in the
startup folder if yes it just does nothing and goes on, if not it
copies the file to the startup folder an than renames it (you can use that if you want but you don't need to).

It is not necessary to do an os.getenv("SystemDrive") or os.getlogin(), because os.getenv("AppData") already gets both. So the most direct way of doing it that I know of is this:
path = os.path.join(os.getenv("appdata"),"Microsoft","Windows","Start Menu","Programs","Startup")

python3 chdir() does not seem to work

As an newcomer to python I figured I'd write a little python3 script to help me switch directories on the command line (ubuntu trusty). Unfortunately os.chdir() does not seems to work.
I've tried tinkering with it in various ways such as placing quotes around the path, removing the leading slash (which obviously doesn't work) and even just hardcoding it, but I can't get it to work - can anybody tell me what I'm missing here?
The call to chdir() happens towards the end - you can see the code in github too
#!/usr/bin/env python3
# #python3
# #author sabot <sabot#inuits.eu>
"""Switch directories without wearing out your slash key"""
import sys
import os
import json
import click
__VERSION__ = '0.0.1'
# 3 params are needed for click callback
def show_version(ctx, param, value):
"""Print version information and exit."""
if not value:
return
click.echo('Goto %s' % __VERSION__)
ctx.exit() # quit the program
def add_entry(dictionary, filepath, path, alias):
"""Add a new path alias."""
print("Adding alias {} for path {} ".format(alias,path))
dictionary[alias] = path
try:
jsondata = json.dumps(dictionary, sort_keys=True)
fd = open(filepath, 'w')
fd.write(jsondata)
fd.close()
except Exception as e:
print('Error writing to dictionary file: ', str(e))
pass
def get_entries(filename):
"""Get the alias entries in json."""
returndata = {}
if os.path.exists(filename) and os.path.getsize(filename) > 0:
try:
fd = open(filename, 'r')
entries = fd.read()
fd.close()
returndata = json.loads(entries)
except Exception as e:
print('Error reading dictionary file: ', str(e))
pass
else:
print('Dictionary file not found or empty- spawning new one in', filename)
newfile = open(filename,'w')
newfile.write('')
newfile.close()
return returndata
#click.command()
#click.option('--version', '-v', is_flag=True, is_eager=True,
help='Print version information and exit.', expose_value=False,
callback=show_version)
#click.option('--add', '-a', help="Add a new path alias")
#click.option('--target', '-t', help="Alias target path instead of the current directory")
#click.argument('alias', default='currentdir')
#click.pass_context
def goto(ctx, add, alias, target):
'''Go to any directory in your filesystem'''
# load dictionary
filepath = os.path.join(os.getenv('HOME'), '.g2dict')
dictionary = get_entries(filepath)
# add a path alias to the dictionary
if add:
if target: # don't use current dir as target
if not os.path.exists(target):
print('Target path not found!')
ctx.exit()
else:
add_entry(dictionary, filepath, target, add)
else: # use current dir as target
current_dir = os.getcwd()
add_entry(dictionary, filepath, current_dir, add)
elif alias != 'currentdir':
if alias in dictionary:
entry = dictionary[alias]
print('jumping to',entry)
os.chdir(entry)
elif alias == 'hell':
print("Could not locate C:\Documents and settings")
else:
print("Alias not found in dictionary - did you forget to add it?")
if __name__ == '__main__':
goto()

The problem is not with Python, the problem is that what you're trying to do is impossible.
When you start a Python interpreter (script or interactive REPL), you do so from your "shell" (Bash etc.). The shell has some working directory, and it launches Python in the same one. When Python changes its own working directory, it does not affect the parent shell, nor do changes in the working directory of the shell affect Python after it has started.
If you want to write a program which changes the directory in your shell, you should define a function in your shell itself. That function could invoke Python to determine the directory to change to, e.g. the shell function could be simply cd $(~/myscript.py) if myscript.py prints the directory it wants to switch to.

Here's Python 3 version of #ephemient's C solution:
#!/usr/bin/env python3
"""Change parent working directory."""
#XXX DIRTY HACK, DO NOT DO IT
import os
import sys
from subprocess import Popen, PIPE, DEVNULL, STDOUT
gdb_cmd = 'call chdir("{dir}")\ndetach\nquit\n'.format(dir=sys.argv[1])
with Popen(["gdb", "-p", str(os.getppid()), '-q'],
stdin=PIPE, stdout=DEVNULL, stderr=STDOUT) as p:
p.communicate(os.fsencode(gdb_cmd))
sys.exit(p.wait())
Example:
# python3 cd.py /usr/lib && python3 -c 'import os; print(os.getcwd())'