We Keep Coding

Is it possible to create a HDF5 structure matching the file path on a local pc?

I am trying to automatically create a HDF5 structure by using the file paths on my local pc. I want to read through the subdirectories and create a HDF5 structure to match, that I can then save files to. Thank you

You can do this by combining os.walk() and h5py create_group(). The only complications are handling Linux vs Windows (and drive letter on Windows). Another consideration is relative vs absolute path. (I used absolute path, but my example can be modified. (Note: it's a little verbose so you can see what's going on.) Here is the example code (for Windows):
with h5py.File('SO_73879694.h5','w') as h5f:
cwd = os.getcwd()
for root, dirs, _ in os.walk(cwd, topdown=True):
print(f'ROOT: {root}')
# for Windows, modify root: remove drive letter and replace backslashes:
grp_name = root[2:].replace( '\\', '/')
print(f'grp_name: {grp_name}\n')
h5f.create_group(grp_name)

This is actually quite easy to do using HDFql in Python. Here is a complete script that does that:
# import HDFql module (make sure it can be found by the Python interpreter)
import HDFql
# create and use (i.e. open) an HDF5 file named 'my_file.h5'
HDFql.execute("CREATE AND USE FILE my_file.h5")
# get all directories recursively starting from the root of the file system (/)
HDFql.execute("SHOW DIRECTORY / LIKE **")
# iterate result set and create group in each iteration
while HDFql.cursor_next() == HDFql.SUCCESS:
HDFql.execute("CREATE GROUP \"%s\"" % HDFql.cursor_get_char())

Python script running from windows scheduled task is storing data in wrong folder

I have a script that scrapes data, it saves it to my local db and it takes screenshot from webpages and stores it in a local folder. Both the db and screenshots folder are located in a folder on my Desktop.
Here are some lines of code from my script.
connection = sqlite3.connect('NW.db')
if not os.path.exists('nw_' + req[1]):
os.mkdir('nw_' + req[1])
# folder = f'nw_{req[1]}\{Month}_{Hire_Start}'
folder = f'nw_{req[1]}\{Hire_Start}'
if not os.path.exists(folder):
os.mkdir(folder)
folder_w = f'nw_{req[1]}'
if not os.path.exists(folder_w):
os.mkdir(folder_w)
ss.full_Screenshot(driver, save_path=r'.', image_name=folder + "NW_" + req[1] + "_" + days3 + "_" + Hire_Start + ".jpg")
Everything is working fine and it is stored in the same folder where I am running my script from. But when I make a windows scheduled task to run this script at a specific time it stores the db file and the screenshots folder in the following location C:\Windows\System32 instead of my folder on the Desktop
I made a batch file and I added this to my scheduled task
"C:\Users\Administrator\AppData\Local\Programs\Python\Python310\python.exe" "C:\Users\Administrator\Desktop\Python_Kishen\NW_offer.py"
pause

You shouldn't rely on your "current" or "working" directory in a script run from the system. Specify the full path to where you want your output files.
You can use os.chdir() to make your preferred destination your current dir, but that's not the best way. Full paths.

The scheduler process likely has a different location it operates from, and it looks like you're using relative paths. I'd recommend using full paths if you have a place you want this all to go. Then it doesn't matter where the code is run from.
As a side note, it looks like you're doing a lot of repeat string building. Why not do that once and store the result in a variable? Just use the variable in the few places you need that value.

Where does python look for files in a script? [duplicate]

This question already has answers here:
How do I get the parent directory in Python?
(21 answers)
Closed 9 years ago.
So I've just coded this class for a title screen and it works well. However, one of the people I'm working with on the project mentioned that I shouldn't use:
os.chdir(os.getcwd() + "/..")
resource = (os.getcwd() + "/media/file name")
to get to the super directory. He did mention something about the pythonpath though. We're using Eclipse if this is of some help.
For more context we're making a multi-platform game so we can't just synchronize our directories and hard-code it (although we are using git so the working directory is synchronized). Basically, I need some way to get from a script file in a "src' folder to a "media" folder that's next to it (AKA There's a super (project) folder with both "src" and "media" folders in it).
Any help would be greatly appreciated, but please don't say "google it" because I tried that before coming here (I don't know if that's a frequent thing here, but I've seen it too many times elsewhere...when I've googled for answers, sorry if I sound jerkish for saying that)

Python programs do have the concept of a current working directory, which is generally the directory from which the script was run. This is "where they look for files" with a relative path.
However, since your program can be run from a different folder than the one it is in, your directory of reference needs to instead refer to the directory your script is in (the current directory is not related to the location of your script, in general). The directory where your script is found is obtained with
script_dir = os.path.dirname(__file__)
Note that this path can be relative (possibly empty), so it is still important that the current working directory of your script be the same as the directory when your script was read by the python interpreter (which is when __file__ is set). It is important to convert the possibly relative script_dir into an absolute path if the current working directory is changed later in the code:
# If script_dir is relative, the current working directory is used, here. This is correct if the current
# working directory is the same as when the script was read by the Python interpreter (which is
# when __file__ was set):
script_dir = os.path.abspath(script_dir)
You can then get to the directory media in the parent directory with the platform-independent
os.path.join(script_dir, os.path.pardir, 'media')
In fact os.path.pardir (or equivalently os.pardir) is the platform-independent parent directory convention, and os.path.join() simply joins paths in a platform independent way.

I'd recommend something like:
import os.path
base_folder = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
media_folder = os.path.join(base_folder, "media")
src_folder = os.path.join(base_folder, "src")
resource = os.path.join(media_folder, "filename")
for path in [base_folder, media_folder, src_folder, resource]:
print path
The main ingredients are:
__file__: gets the path to the current source file (unlike sys.argv[0], which gives the path the script that was called)
os.path.split(): splits a path into the relative file/folder name and the base folder containing it. Using it twice as in base_folder = ... will give the parent directory.
os.path.join: OS-independent and correct combination of path names. Is aware of missing or multiple /s or \s

Consider using os.path.dirname() and os.path.join(). These should work in a platform independent way.

Yapsy looks at entire file system for plugins

I've got an application that uses YAPSY for our plugin framework. The program specifies a default plugin directory and also lets the user specify a directory as well (which is saved in an XML file):
# Get the default plugin directory, using XML
path = os.path.expanduser('~')
xml = xml_controller.Controller(path + '\.cxvrc.xml')
xml.load_file()
xml.get_plugin_directory()
directory = ["plugins", xml.get_plugin_directory()]
# Load the plugins from the default plugin directory.
manager = PluginManager()
manager.setPluginPlaces(directory)
manager.setPluginInfoExtension('plugin')
manager.collectPlugins()
The problem with this is that it's loading the user's plugins, no matter where they are on their file system. Essentially, it's ignoring the resulting string of the XML file query. Why could this be? An example of what
xml.get_plugin_directory()
returns is a string: "C:\Documents and Settings\achilds". I thought that the collectPlugins() method should check the list of directories that I've supplied in setPluginPlaces(directory).
Any ideas why this finds all plugins on the user's file system (no matter what plugin directory they've specified)?
Thank you

Well, I've figured out the culprit. It seems that this was on my end, after closer inspection of how the YAPSY PluginManager works. PluginManager.locatePlugin() looks at the directories given, but then uses os.walk(given paths) to check every folder underneath the given paths.
Unknowingly, I set the directory to look in at "C:\Documents and Settings\achilds" and then placed my plugins on the desktop which happens to be at location: "C:\Documents and Settings\achilds\Desktop". So, PluginManager.locatePlugin() did what it's designed to do and searched through all directories within "C:\Documents and Settings\achilds", finding the directory that I had placed on the Desktop.

Bundling GTK resources with py2exe

I'm using Python 2.6 and PyGTK 2.22.6 from the all-in-one installer on Windows XP, trying to build a single-file executable (via py2exe) for my app.
My problem is that when I run my app as a script (ie. not built into an .exe file, just as a loose collection of .py files), it uses the native-looking Windows theme, but when I run the built exe I see the default GTK theme.
I know that this problem can be fixed by copying a bunch of files into the dist directory created by py2exe, but everything I've read involves manually copying the data, whereas I want this to be an automatic part of the build process. Furthermore, everything on the topic (including the FAQ) is out of date - PyGTK now keeps its files in C:\Python2x\Lib\site-packages\gtk-2.0\runtime\..., and just copying the lib and etc directories doesn't fix the problem.
My questions are:
I'd like to be able to programmatically find the GTK runtime data in setup.py rather than hard coding paths. How do I do this?
What are the minimal resources I need to include?
Update: I may have almost answered #2 by trial-and-error. For the "wimp" (ie. MS Windows) theme to work, I need the files from:
runtime\lib\gtk-2.0\2.10.0\engines\libwimp.dll
runtime\etc\gtk-2.0\gtkrc
runtime\share\icons\*
runtime\share\themes\MS-Windows
...without the runtime prefix, but otherwise with the same directory structure, sitting directly in the dist directory produced by py2exe. But where does the 2.10.0 come from, given that gtk.gtk_version is (2,22,0)?

Answering my own question here, but if anyone knows better feel free to answer too. Some of it seems quite fragile (eg. version numbers in paths), so comment or edit if you know a better way.
1. Finding the files
Firstly, I use this code to actually find the root of the GTK runtime. This is very specific to how you install the runtime, though, and could probably be improved with a number of checks for common locations:
#gtk file inclusion
import gtk
# The runtime dir is in the same directory as the module:
GTK_RUNTIME_DIR = os.path.join(
os.path.split(os.path.dirname(gtk.__file__))[0], "runtime")
assert os.path.exists(GTK_RUNTIME_DIR), "Cannot find GTK runtime data"
2. What files to include
This depends on (a) how much of a concern size is, and (b) the context of your application's deployment. By that I mean, are you deploying it to the whole wide world where anyone can have an arbitrary locale setting, or is it just for internal corporate use where you don't need translated stock strings?
If you want Windows theming, you'll need to include:
GTK_THEME_DEFAULT = os.path.join("share", "themes", "Default")
GTK_THEME_WINDOWS = os.path.join("share", "themes", "MS-Windows")
GTK_GTKRC_DIR = os.path.join("etc", "gtk-2.0")
GTK_GTKRC = "gtkrc"
GTK_WIMP_DIR = os.path.join("lib", "gtk-2.0", "2.10.0", "engines")
GTK_WIMP_DLL = "libwimp.dll"
If you want the Tango icons:
GTK_ICONS = os.path.join("share", "icons")
There is also localisation data (which I omit, but you might not want to):
GTK_LOCALE_DATA = os.path.join("share", "locale")
3. Piecing it together
Firstly, here's a function that walks the filesystem tree at a given point and produces output suitable for the data_files option.
def generate_data_files(prefix, tree, file_filter=None):
"""
Walk the filesystem starting at "prefix" + "tree", producing a list of files
suitable for the data_files option to setup(). The prefix will be omitted
from the path given to setup(). For example, if you have
C:\Python26\Lib\site-packages\gtk-2.0\runtime\etc\...
...and you want your "dist\" dir to contain "etc\..." as a subdirectory,
invoke the function as
generate_data_files(
r"C:\Python26\Lib\site-packages\gtk-2.0\runtime",
r"etc")
If, instead, you want it to contain "runtime\etc\..." use:
generate_data_files(
r"C:\Python26\Lib\site-packages\gtk-2.0",
r"runtime\etc")
Empty directories are omitted.
file_filter(root, fl) is an optional function called with a containing
directory and filename of each file. If it returns False, the file is
omitted from the results.
"""
data_files = []
for root, dirs, files in os.walk(os.path.join(prefix, tree)):
to_dir = os.path.relpath(root, prefix)
if file_filter is not None:
file_iter = (fl for fl in files if file_filter(root, fl))
else:
file_iter = files
data_files.append((to_dir, [os.path.join(root, fl) for fl in file_iter]))
non_empties = [(to, fro) for (to, fro) in data_files if fro]
return non_empties
So now you can call setup() like so:
setup(
# Other setup args here...
data_files = (
# Use the function above...
generate_data_files(GTK_RUNTIME_DIR, GTK_THEME_DEFAULT) +
generate_data_files(GTK_RUNTIME_DIR, GTK_THEME_WINDOWS) +
generate_data_files(GTK_RUNTIME_DIR, GTK_ICONS) +
# ...or include single files manually
[
(GTK_GTKRC_DIR, [
os.path.join(GTK_RUNTIME_DIR,
GTK_GTKRC_DIR,
GTK_GTKRC)
]),
(GTK_WIMP_DIR, [
os.path.join(
GTK_RUNTIME_DIR,
GTK_WIMP_DIR,
GTK_WIMP_DLL)
])
]
)
)