We Keep Coding

File not found Jupyter notebook

I am having trouble loading a file in jupyter notebook.
Here is my project tree:
-- home
---- cdsw
------ my_main.py
------ notebooks
-------- my_notebook.ipynb
------ dns
-------- assets
---------- stopwords.txt
-------- bilans
---------- my_module.py
Know that '/home/cdsw/" is in my PYTHONPATH - the same interpreter in which I launch jupyter -.
In my_module.py I have these lines:
PATH_STOPWORDS: Final = os.path.join("dns", "assets", "stopwords.txt")
STOPWORDS: Final = load_stopwords(PATH_STOPWORDS)
load_stopwords is basically just a open(PATH_STOPWORDS, 'r').
So my problem is that when I import dns.bilans.my_module inside my_main.py it works fine: file is correctly loaded.
Yet, when I import it from my_notebook.ipynb, it does not :
FileNotFoundError: [Errno 2] No such file or directory: 'dns/assets/stopwords.txt'
So my_module is indeed founded by jupyter kernel (because it reads the code lines of the file) but can't use the relative path provided like it does from a run in a terminal.
When I use a open(relpath, 'r') inside a module, I don't need to go all through the project tree right ? Indeed it DOES work in my_main.py ...
I really don't get it ...
The output of os.getcwd() in jupyter is "/home/cdsw/notebooks".

This existing SO question suggests how to find files relative to the position of a Python code file. It isn't exactly the same question, however, and I believe that this technique is so important for every Python programmer to understand, that I'm going to provide a more thorough answer.
Given a piece of Python code, one can compute the path of the directory of the source file containing that code via:
here = os.path.dirname(__file__)
Having the position of the relevant source file, it is easy to compute an absolute path to any data file that has a well known location relative to that source file. In this case, the way to do that is:
stopwords_path = os.path.join(here, '..', '..', 'assets', 'stopwords.txt')
This path can be supplied to open() or used in any other way to refer to the stopwords.txt data file. Here, the way to use this path would be:
load_stopwords(stopwords_path)
I use this technique to not only find files that accompany code in a particular module, but also to find files that are in other locations throughout my source tree. As long as the code and data file exist in the same source repository, or are shipped together in a single Python package, the relative path will not change from installation to installation, and so this technique will work.
In general, you should avoid the use of relative paths. Whenever possible, you should also avoid having to tell your code where to find something. For any situation, ask yourself how you can obtain a reliable absolute path that you can then use to then locate whatever it is you're wanting to access.

Navigating directories

After getting the path to the current working directory using:
cwd = os.getcwd()
How would one go up one folder: C:/project/analysis/ to C:/project/ and enter a folder called data (C:/project/data/)?

In general it a bad idea to 'enter' a directory (ie change the current directory), unless that is explicity part of the behaviour of the program.
In general to open a file in one directory 'over from where you are you can do .. to navigate up one level.
In your case you can open a file using the path ../data/<filename> - in other words use relative file names.
If you really need to change the current working directory you can use os.chdir() but remember this could well have side effects - for example if you import modules from your local directory then using os.chdir() will probably impact that import.

As per Python documentation, you could try this:
os.chdir("../data")

Following a nested directory structure until the end

I have a some directories that contain some other directories which, at the lowest level, contain bunch of csv files such as (folder) a -> b -> c -> (csv files). There is usually only one folder at each level. When I process a directory how can I follow this structure until the end to get the csv files ? I was thinking maybe a recursive solution but I think there may be better ways to do this. I am using python. Hope I was clear.

The os package has a walk function that will do exactly what you need:
for current_path, directory, files in walk("/some/path"):
# current_path is the full path of the directory we are currently in
# directory is the name of the directory
# files is a list of file names in this directory
You can use os.path's to derive the full path to each file (if you need it).
Alternately, you might find the glob module to be of more use to you:
for csv_file in glob(/some/path/*/*.csv"):
# csv_file is the full path to the csv file.

Yapsy looks at entire file system for plugins

I've got an application that uses YAPSY for our plugin framework. The program specifies a default plugin directory and also lets the user specify a directory as well (which is saved in an XML file):
# Get the default plugin directory, using XML
path = os.path.expanduser('~')
xml = xml_controller.Controller(path + '\.cxvrc.xml')
xml.load_file()
xml.get_plugin_directory()
directory = ["plugins", xml.get_plugin_directory()]
# Load the plugins from the default plugin directory.
manager = PluginManager()
manager.setPluginPlaces(directory)
manager.setPluginInfoExtension('plugin')
manager.collectPlugins()
The problem with this is that it's loading the user's plugins, no matter where they are on their file system. Essentially, it's ignoring the resulting string of the XML file query. Why could this be? An example of what
xml.get_plugin_directory()
returns is a string: "C:\Documents and Settings\achilds". I thought that the collectPlugins() method should check the list of directories that I've supplied in setPluginPlaces(directory).
Any ideas why this finds all plugins on the user's file system (no matter what plugin directory they've specified)?
Thank you

Well, I've figured out the culprit. It seems that this was on my end, after closer inspection of how the YAPSY PluginManager works. PluginManager.locatePlugin() looks at the directories given, but then uses os.walk(given paths) to check every folder underneath the given paths.
Unknowingly, I set the directory to look in at "C:\Documents and Settings\achilds" and then placed my plugins on the desktop which happens to be at location: "C:\Documents and Settings\achilds\Desktop". So, PluginManager.locatePlugin() did what it's designed to do and searched through all directories within "C:\Documents and Settings\achilds", finding the directory that I had placed on the Desktop.

Bundling GTK resources with py2exe

I'm using Python 2.6 and PyGTK 2.22.6 from the all-in-one installer on Windows XP, trying to build a single-file executable (via py2exe) for my app.
My problem is that when I run my app as a script (ie. not built into an .exe file, just as a loose collection of .py files), it uses the native-looking Windows theme, but when I run the built exe I see the default GTK theme.
I know that this problem can be fixed by copying a bunch of files into the dist directory created by py2exe, but everything I've read involves manually copying the data, whereas I want this to be an automatic part of the build process. Furthermore, everything on the topic (including the FAQ) is out of date - PyGTK now keeps its files in C:\Python2x\Lib\site-packages\gtk-2.0\runtime\..., and just copying the lib and etc directories doesn't fix the problem.
My questions are:
I'd like to be able to programmatically find the GTK runtime data in setup.py rather than hard coding paths. How do I do this?
What are the minimal resources I need to include?
Update: I may have almost answered #2 by trial-and-error. For the "wimp" (ie. MS Windows) theme to work, I need the files from:
runtime\lib\gtk-2.0\2.10.0\engines\libwimp.dll
runtime\etc\gtk-2.0\gtkrc
runtime\share\icons\*
runtime\share\themes\MS-Windows
...without the runtime prefix, but otherwise with the same directory structure, sitting directly in the dist directory produced by py2exe. But where does the 2.10.0 come from, given that gtk.gtk_version is (2,22,0)?

Answering my own question here, but if anyone knows better feel free to answer too. Some of it seems quite fragile (eg. version numbers in paths), so comment or edit if you know a better way.
1. Finding the files
Firstly, I use this code to actually find the root of the GTK runtime. This is very specific to how you install the runtime, though, and could probably be improved with a number of checks for common locations:
#gtk file inclusion
import gtk
# The runtime dir is in the same directory as the module:
GTK_RUNTIME_DIR = os.path.join(
os.path.split(os.path.dirname(gtk.__file__))[0], "runtime")
assert os.path.exists(GTK_RUNTIME_DIR), "Cannot find GTK runtime data"
2. What files to include
This depends on (a) how much of a concern size is, and (b) the context of your application's deployment. By that I mean, are you deploying it to the whole wide world where anyone can have an arbitrary locale setting, or is it just for internal corporate use where you don't need translated stock strings?
If you want Windows theming, you'll need to include:
GTK_THEME_DEFAULT = os.path.join("share", "themes", "Default")
GTK_THEME_WINDOWS = os.path.join("share", "themes", "MS-Windows")
GTK_GTKRC_DIR = os.path.join("etc", "gtk-2.0")
GTK_GTKRC = "gtkrc"
GTK_WIMP_DIR = os.path.join("lib", "gtk-2.0", "2.10.0", "engines")
GTK_WIMP_DLL = "libwimp.dll"
If you want the Tango icons:
GTK_ICONS = os.path.join("share", "icons")
There is also localisation data (which I omit, but you might not want to):
GTK_LOCALE_DATA = os.path.join("share", "locale")
3. Piecing it together
Firstly, here's a function that walks the filesystem tree at a given point and produces output suitable for the data_files option.
def generate_data_files(prefix, tree, file_filter=None):
"""
Walk the filesystem starting at "prefix" + "tree", producing a list of files
suitable for the data_files option to setup(). The prefix will be omitted
from the path given to setup(). For example, if you have
C:\Python26\Lib\site-packages\gtk-2.0\runtime\etc\...
...and you want your "dist\" dir to contain "etc\..." as a subdirectory,
invoke the function as
generate_data_files(
r"C:\Python26\Lib\site-packages\gtk-2.0\runtime",
r"etc")
If, instead, you want it to contain "runtime\etc\..." use:
generate_data_files(
r"C:\Python26\Lib\site-packages\gtk-2.0",
r"runtime\etc")
Empty directories are omitted.
file_filter(root, fl) is an optional function called with a containing
directory and filename of each file. If it returns False, the file is
omitted from the results.
"""
data_files = []
for root, dirs, files in os.walk(os.path.join(prefix, tree)):
to_dir = os.path.relpath(root, prefix)
if file_filter is not None:
file_iter = (fl for fl in files if file_filter(root, fl))
else:
file_iter = files
data_files.append((to_dir, [os.path.join(root, fl) for fl in file_iter]))
non_empties = [(to, fro) for (to, fro) in data_files if fro]
return non_empties
So now you can call setup() like so:
setup(
# Other setup args here...
data_files = (
# Use the function above...
generate_data_files(GTK_RUNTIME_DIR, GTK_THEME_DEFAULT) +
generate_data_files(GTK_RUNTIME_DIR, GTK_THEME_WINDOWS) +
generate_data_files(GTK_RUNTIME_DIR, GTK_ICONS) +
# ...or include single files manually
[
(GTK_GTKRC_DIR, [
os.path.join(GTK_RUNTIME_DIR,
GTK_GTKRC_DIR,
GTK_GTKRC)
]),
(GTK_WIMP_DIR, [
os.path.join(
GTK_RUNTIME_DIR,
GTK_WIMP_DIR,
GTK_WIMP_DLL)
])
]
)
)