GnuRadio: organizing OOT module python code in sub-directories - python

I have used gr_modtool to add custom blocks in python to an OOT module. It appears that all the source python I write must reside in the gr-my_oot_module/python directory.
I will be writing a lot of code spread over many python files. I would like to organize those files into sub-directories (presumably) under gr-my_oot_module/python. Simply creating those directories and putting code there does not lead to a successful installation.
What is the correct approach to organizing the python files I write for this module into sub-directories?
More specifically:
I added a block via gr_modtool. The associated python file with put in the python directory.
I then moved that .py file into a sub-directory (sub_dir) under python/.
I modified init.py and CMakeLists.txt under the python directory to reflect the sub-directory location and then did the install.
The block appears in GRC. When I try to use it, it complains
File "/home/my_name/devel/gnuradio3_8/lib/python3.6/dist-packages/my_module/__init__.py"
from .sub_dir.sub_dir_test_blk import sub_dir_test_blk
ModuleNotFoundError: No module named 'my_module.sub_dir' –

You're right, Python code resides under python/. Then, you should use gr_modtool add to add GNU Radio python blocks. That will also add them to the CMakeLists.txt, which will in turn make sure they get installed during installation.
No, just putting files in subdirectories doesn't make them part of the installed module. That is not different than for any other python code. If you want things to be part of a module, you need to have them in an __init__.py. The python.org tutorial is your friend!

Related

How do I get the list of imports and dependant files from python script?

I have a .py file that imports from other python modules that import from config files, other modules, etc.
I am to move the code needed to run that .py file, but only whatever the py file is reading from (I am not talking about packages installed by pip install, it's more about other python files in the project directory, mostly classes, functions and ini files).
Is there a way to find out only the external files used by that particular python script? Is it something that can be found using PyCharm for example?
Thanks!
Static analysis tools (such as PyCharm's refactoring tools) can (mostly) figure out the module import tree for a program (unless you do dynamic imports using e.g. importlib.import_module()).
However, it's not quite possible to statically definitively know what other files are required for your program to function. You could use Python's audit events (or strace/ptrace or similar OS-level functions) to look at what files are being opened by your program (e.g. during your tests being run (you do have tests, right?), or during regular program use), but it's likely not going to be exhaustive.

Dynamically updating the PYTHONPATH prevents .pyc update

To allow myself to have a clear filestructure in my project i am using the following code snippet to dynamically add the project main folder to the PYTHONPATH and therefore assure that I can import files even from above a files location.
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(__file__)), "."))
Since I did this, when I start my main file, changes to the modules aren't recognized anymore until i manually delete any .pyc files. Thus I assume this for some reason prevented python from checking if the pyc files are up to date. Can I overcome this issue in any way?
Adding the path of an already imported module can get you into trouble if module names are no longer unique. Consider that you do import foo, which adds its parent package bar to sys.path - it's now possible to also do import bar.foo. Python will consider both to be different modules, which can mess up anything relying on module identity.
You should really consider why you need to do this hack in the first place. If you have an executable placed inside your package, you should not do
cd bardir/bar
python foo
but instead call it as part of the package via
cd bardir
python -m bar.foo
You could try to make python not write those *.pyc files.
How to avoid .pyc files?
For large projects this would matter slightly from a performance perspective. It's possible that you don't care about that, and then you can just not create the pyc files.

Problem accessing config files within a Python egg

I have a Python project that has the following structure:
package1
class.py
class2.py
...
package2
otherClass.py
otherClass2.py
...
config
dev_settings.ini
prod_settings.ini
I wrote a setup.py file that converts this into an egg with the same file structure. (When I examine it using a zip program the structure seems identical.) The funny thing is, when I run the Python code from my IDE it works fine and can access the config files; but when I try to run it from a different Python script using the egg, it can't seem to find the config files in the egg. If I put the config files into a directory relative to the calling Python script (external to the egg), it works - but that sort of defeats the purpose of having a self-contained egg that has all the functionality of the program and can be called from anywhere. I can use any classes/modules and run any functions from the egg as long as they don't use the config files... but if they do, the egg can't find them and so the functions don't work.
Any help would be really appreciated! We're kind of new to the egg thing here and don't really know where to start.
The problem is, the config files are not files anymore - they're packaged within the egg. It's not easy to find the answer in the docs, but it is there. From the setuptools developer's guide:
Typically, existing programs manipulate a package's __file__ attribute in order to find the location of data files. However, this manipulation isn't compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs.
To access them, you need to follow the instructions for the Resource Management API.
In my own code, I had this problem with a logging configuration file. I used the API successfully like this:
from pkg_resources import resource_stream
_log_config_file = 'logging.conf'
_log_config_location = resource_stream(__name__, _log_config_file)
logging.config.fileConfig(_log_config_location)
_log = logging.getLogger('package.module')
See Setuptools' discussion of accessing pacakged data files at runtime. You have to get at your configuration file a different way if you want the script to work inside an egg. Also, for that to work, you may need to make your config directory a Python package by tossing in an empty __init__.py file.

External classes in Python

I'm just beginning Python, and I'd like to use an external RSS class. Where do I put that class and how do I import it? I'd like to eventually be able to share python programs.
About the import statement:
(a good writeup is at http://effbot.org/zone/import-confusion.htm and the python tutorial goes into detail at http://docs.python.org/tutorial/modules.html )
There are two normal ways to import code into a python program.
Modules
Packages
A module is simply a file that ends in .py. In order for python, it must exist on the search path (as defined in sys.path). The search path usually consists of the same directory of the .py that is being run, as well as the python system directories.
Given the following directory structure:
myprogram/main.py
myprogram/rss.py
From main.py, you can "import" the rss classes by running:
import rss
rss.rss_class()
#alternativly you can use:
from rss import rss_class
rss_class()
Packages provide a more structured way to contain larger python programs. They are simply a directory which contains an __init__.py as well as other python files.
As long as the package directory is on sys.path, then it can be used exactly the same as above.
To find your current path, run this:
import sys
print(sys.path)
I don't really like answering so late, but I'm not entirely satisfied with the existing answers.
I'm just beginning Python, and I'd like to use an external RSS class. Where do I put that class and how do I import it?
You put it in a python file, and give the python file an extension of .py . Then you can import a module representing that file, and access the class. Supposing you want to import it, you must put the python file somewhere in your import search path-- you can see this at run-time with sys.path, and possibly the most significant thing to know is that the site-packages (install-specific) and current directory ('') are generally in the import search path. When you have a single homogeneous project, you generally put it in the same directory as your other modules and let them import each other from the same directory.
I'd like to eventually be able to share python programs.
After you have it set up as a standalone file, you can get it set up for distribution using distutils. That way you don't have to worry about where, exactly, it should be installed-- distutils will worry for you. There are many other additional means of distribution as well, many OS-specific-- distutils works for modules, but if you want to distribute a proper program that users are meant to run, other options exist, such as using py2exe for Windows.
As for the modules/packages distinction, well, here it goes. If you've got a whole bunch of classes that you want divided up so that you don't have one big mess of a python file, you can separate it into multiple python files in a directory, and give the directory an __init__.py . The important thing to note is that from Python, there's no difference between a package and any other module. A package is a module, it's just a different way of representing one on the filesystem. Similarly, a module is not just a .py file-- if that were the case, sys would not be a module, since it has no .py file. It's built-in to the interpreter. There are infinitely many ways to represent modules on the filesystem, since you can add import hooks that can create ways other than directories and .py files to represent modules. One could, hypothetically, create an import hook that used spidermonkey to load Javascript files as Python modules.
from [module] import [classname]
Where the module is somewhere on your python path.
About modules and packages:
a module is a file ending with .py. You can put your class in such a file. As said by Andy, it needs to be in your python path (PYTHONPATH). Usually you will put the additional module in the same directory as your script is though which can be directly imported.
a package is a directory containing an __init__.py (can be empty) and contains module files. You can then import a la from <package>.<module> import <class>. Again this needs to be on your python path.
You can find more in the documenation.
If you want to store your RSS file in a different place use sys.append("") and pout the module in that directory and use
import or from import *
The first file, where you have created the class, is "first.py"
first.py:
class Example:
...
You create the second file, where you want to use the class contained in the "first.py", which is "second.py"
myprogram/first.py
myprogram/second.py
Then in the second file, to call the class contained in the first file, you simply type:
second.py:
from first import Example
...

Deploying a python application with shared package

I'm thinking how to arrange a deployed python application which will have a
Executable script located in /usr/bin/ which will provide a CLI to functionality implemented in
A library installed to wherever the current site-packages directory is.
Now, currently, I have the following directory structure in my sources:
foo.py
foo/
__init__.py
...
which I guess is not the best way to do things. During development, everything works as expected, however when deployed, the "from foo import FooObject" code in foo.py seemingly attempts to import foo.py itself, which is not the behaviour I'm looking for.
So the question is what is the standard practice of orchestrating situations like this? One of the things I could think of is, when installing, rename foo.py to just foo, which stops it from importing itself, but that seems rather awkward...
Another part of the problem, I suppose, is that it's a naming challenge. Perhaps call the executable script foo-bin.py?
This article is pretty good, and shows you a good way to do it. The second item from the Do list answers your question.
shameless copy paste:
Filesystem structure of a Python project
by Jp Calderone
Do:
name the directory something related to your project. For example, if your
project is named "Twisted", name the
top-level directory for its source
files Twisted. When you do releases,
you should include a version number
suffix: Twisted-2.5.
create a directory Twisted/bin and put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects.
If your project is expressable as a single Python source file, then put it
into the directory and name it
something related to your project. For
example, Twisted/twisted.py. If you
need multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and place
your source files in it. For example,
Twisted/twisted/internet.py.
put your unit tests in a sub-package of your package (note - this means
that the single Python source file
option above was a trick - you always
need at least one other file for your
unit tests). For example,
Twisted/twisted/test/. Of course, make
it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory called src or lib. This makes it hard
to run without installing.
put your tests outside of your Python package. This makes it hard to
run the tests against an installed
version.
create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module
instead of a package, it's simpler.
try to come up with magical hacks to make Python able to import your module
or package without having the user add
the directory containing it to their
import path (either via PYTHONPATH or
some other mechanism). You will not
correctly handle all cases and users
will get angry at you when your
software doesn't work in their
environment.
Distutils supports installing modules, packages, and scripts. If you create a distutils setup.py which refers to foo as a package and foo.py as a script, then foo.py should get installed to /usr/local/bin or whatever the appropriate script install path is on the target OS, and the foo package should get installed to the site_packages directory.
You should call the executable just foo, not foo.py, then attempts to import foo will not use it.
As for naming it properly: this is difficult to answer in the abstract; we would need to know what specifically it does. For example, if it configures and controls, calling it -config or ctl might be appropriate. If it is a shell API for the library, it should have the same name as the library.
Your CLI module is one thing, the package that supports it is another thing. Don't confuse the names withe module foo (in a file foo.py) and the package foo (in a directory foo with a file __init__.py).
You have two things named foo: a module and a package. What else do you want to name foo? A class? A function? A variable?
Pick a distinctive name for the foo module or the foo package. foolib, for example, is a popular package name.

Categories