Using this general structure:
setup.py
/package
__init__.py
project.py
/data
client.log
I have a script that saves a list of names to client.log, so I don't have to reinitialize that list each time I need access to it or run the module. Before I set up this structure with pkg_resources, I used open('.../data/client.log', 'w') to update the log with explicit paths, but this doesn't work anymore.
Is there any way to edit data files within modules? Or is there a better way to save this list?
No, pkg_resources are for reading resources within a package. You can't use it to write log files, because it's the wrong place for log files. Your package directory should typically not be writeable by the user that loads the library. Also, your package may in fact be inside a ZIP-file.
You should instead store the logs in a log directory. Where to put that depends on a lot of things, the biggest issue is your operating system but also if it's system software or user software.
Related
I have a .py file that imports from other python modules that import from config files, other modules, etc.
I am to move the code needed to run that .py file, but only whatever the py file is reading from (I am not talking about packages installed by pip install, it's more about other python files in the project directory, mostly classes, functions and ini files).
Is there a way to find out only the external files used by that particular python script? Is it something that can be found using PyCharm for example?
Thanks!
Static analysis tools (such as PyCharm's refactoring tools) can (mostly) figure out the module import tree for a program (unless you do dynamic imports using e.g. importlib.import_module()).
However, it's not quite possible to statically definitively know what other files are required for your program to function. You could use Python's audit events (or strace/ptrace or similar OS-level functions) to look at what files are being opened by your program (e.g. during your tests being run (you do have tests, right?), or during regular program use), but it's likely not going to be exhaustive.
I have used gr_modtool to add custom blocks in python to an OOT module. It appears that all the source python I write must reside in the gr-my_oot_module/python directory.
I will be writing a lot of code spread over many python files. I would like to organize those files into sub-directories (presumably) under gr-my_oot_module/python. Simply creating those directories and putting code there does not lead to a successful installation.
What is the correct approach to organizing the python files I write for this module into sub-directories?
More specifically:
I added a block via gr_modtool. The associated python file with put in the python directory.
I then moved that .py file into a sub-directory (sub_dir) under python/.
I modified init.py and CMakeLists.txt under the python directory to reflect the sub-directory location and then did the install.
The block appears in GRC. When I try to use it, it complains
File "/home/my_name/devel/gnuradio3_8/lib/python3.6/dist-packages/my_module/__init__.py"
from .sub_dir.sub_dir_test_blk import sub_dir_test_blk
ModuleNotFoundError: No module named 'my_module.sub_dir' –
You're right, Python code resides under python/. Then, you should use gr_modtool add to add GNU Radio python blocks. That will also add them to the CMakeLists.txt, which will in turn make sure they get installed during installation.
No, just putting files in subdirectories doesn't make them part of the installed module. That is not different than for any other python code. If you want things to be part of a module, you need to have them in an __init__.py. The python.org tutorial is your friend!
To allow myself to have a clear filestructure in my project i am using the following code snippet to dynamically add the project main folder to the PYTHONPATH and therefore assure that I can import files even from above a files location.
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), "."))
Since I did this, when I start my main file, changes to the modules aren't recognized anymore until i manually delete any .pyc files. Thus I assume this for some reason prevented python from checking if the pyc files are up to date. Can I overcome this issue in any way?
Adding the path of an already imported module can get you into trouble if module names are no longer unique. Consider that you do import foo, which adds its parent package bar to sys.path - it's now possible to also do import bar.foo. Python will consider both to be different modules, which can mess up anything relying on module identity.
You should really consider why you need to do this hack in the first place. If you have an executable placed inside your package, you should not do
cd bardir/bar
python foo
but instead call it as part of the package via
cd bardir
python -m bar.foo
You could try to make python not write those *.pyc files.
How to avoid .pyc files?
For large projects this would matter slightly from a performance perspective. It's possible that you don't care about that, and then you can just not create the pyc files.
I have module that I am writing in python that needs to download data and store it in a particular directory. Currently, I am doing this by in the manner shown below,
import os
folder = 'd:\data' #location of the root folder directory on my system
DATAPATH = os.path.join(folder, 'download_data')
This works for my module on my system. I am interested in distributing this module to other machines and I am not sure how I can control the location of the root folder when I install the module to a different machine. Are there any best practices on how to do this? Is there some way to do this in the setup file?
Yes, it should be set by your installer and reside in a configuration file. Use module configparser to extract the value. Look at BasicInterpolation to see your need as an example.
From the sounds of things, you need to install your package with a known structure, including some non-Python files, then need to get the location of those files by using their known relative path from your package modules. In which case, you need to combine these two answers
Including non-Python files with setup.py
Retrieving python module path
And then use your existing logic to construct your full path
I have a Python project that has the following structure:
package1
class.py
class2.py
...
package2
otherClass.py
otherClass2.py
...
config
dev_settings.ini
prod_settings.ini
I wrote a setup.py file that converts this into an egg with the same file structure. (When I examine it using a zip program the structure seems identical.) The funny thing is, when I run the Python code from my IDE it works fine and can access the config files; but when I try to run it from a different Python script using the egg, it can't seem to find the config files in the egg. If I put the config files into a directory relative to the calling Python script (external to the egg), it works - but that sort of defeats the purpose of having a self-contained egg that has all the functionality of the program and can be called from anywhere. I can use any classes/modules and run any functions from the egg as long as they don't use the config files... but if they do, the egg can't find them and so the functions don't work.
Any help would be really appreciated! We're kind of new to the egg thing here and don't really know where to start.
The problem is, the config files are not files anymore - they're packaged within the egg. It's not easy to find the answer in the docs, but it is there. From the setuptools developer's guide:
Typically, existing programs manipulate a package's __file__ attribute in order to find the location of data files. However, this manipulation isn't compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs.
To access them, you need to follow the instructions for the Resource Management API.
In my own code, I had this problem with a logging configuration file. I used the API successfully like this:
from pkg_resources import resource_stream
_log_config_file = 'logging.conf'
_log_config_location = resource_stream(__name__, _log_config_file)
logging.config.fileConfig(_log_config_location)
_log = logging.getLogger('package.module')
See Setuptools' discussion of accessing pacakged data files at runtime. You have to get at your configuration file a different way if you want the script to work inside an egg. Also, for that to work, you may need to make your config directory a Python package by tossing in an empty __init__.py file.