We Keep Coding

Why import works in this ROS python script

I am working with ROS packages and coming from this tutorial. This import statement surprisingly works given the absence of AddTwoIntsResponse anywhere in the current working directory or any other directory listed in PATH. Also how come a .srv gets imported?
# add_two_ints_server.py
from beginner_tutorials.srv import AddTwoInts, AddTwoIntsResponse
The current working directory is ~/catkin_ws/src/beginner_tutorials/srv
This is my ROS directory layout:
catkin_ws
|-- src
| `-- beginner_tutorials
| |-- scripts
| | `-- add_two_ints_server.py
| `-- srv
| `-- AddTwoInts.srv
|-- build
`-- devel
The contents of AddTwoInts.srv are:
int64 a
int64 b
---
int64 sum
According to my understanding this should throw an ImportError: cannot import name 'AddTwoIntsResponse', but it doesn't. Importing any other file say: from beginner_tutorials.srv import foo throws an ImportError.
Where is my understanding going wrong?

|-- src
| `-- beginner_tutorials
| |-- scripts
| | `-- add_two_ints_server.py
| `-- srv
| `-- AddTwoInts.srv
|-- build
|-- devel // this is where your modules are imported from
When you build the package using catkin_make, 'catkin` generates the relevant python files for your service type defined in .srv file and puts them under catkin_ws/devel/lib/your-python-version/dist-packages/package-name/srv.
If your workspace is sourced, catkin_ws/devel/lib/your-python-version/dist-packages/ is already added to your PYTHONPATH and that is how you are able to import them successfully.
In case of the tutorial package that you are using, imports may work even when you haven't sourced your current catkin-directory, if you have the binaries of the tutorials installed. This way the python modules reside under /opt/ros/ros-version/lib/your-python-version/dist-packages/ and that is again part of the PYTHONPATH. (If ROS env is available)

Python: 'ImportError: No Module Named <my module>'

I'm using PyCharm CE to develop a project with a structure similar to this:
test_python/
|-- app
| |-- __init__.py
| |-- mymodule.py
| |-- mymodule.pyc
| `-- test_mymodule.py
|-- config.py
`-- tests
|-- __init__.py
|-- test_config.py
`-- test_models.py
When I try to run my test scripts such as test_config.py, I get:
$ python tests/test_config.py
Traceback (most recent call last):
File "tests/test_config.py", line 1, in <module>
from config import app_config
ImportError: No module named config
I have read a lot of other SO posts that talk about needing a init.py file in all directories that are packages (which I have done already). Many also suggest messing around with sys.path. My problem with this latter approach is that I never had to meddle with the paths previously. I'm not sure if it's something that changed with my dev environment setup, but here it is:
Python 2.7 | macOS Sierra | PyCharm CE
I have tried to install a virtual environment with virtualenv but didn't see a difference. Here is the sample project on github if you'd like to run it yourself

It seems that there is a problem with folder depth. Replace from ../config import app_config.

I think it will work if you change the working directory on your run configuration to test_python. All the packages in your import statements are relative to some entry in your Python path, and the working directory is usually in the Python path.

How to deal with submodule interdependency in Python?

I have a program with several submodules. I want the submodules to be usable independently, so that I can use some of them in other programs as well. However, the submodules have inter-dependency, requiring aspects of each other to run. What is the least problematic way to deal with this?
Right now, I have structured my program like this:
myapp/
|-- __init__.py
|-- app.py
|-- special_classes
| |-- __init__.py
| `-- tools
| `-- __init__.py
|-- special_functions
| |-- __init__.py
| `-- tools
| `-- __init__.py
`-- tools
|-- __init__.py
|-- classes.py
`-- functions.py
Where each submodule is a a git submodule of its parent.
The advantage of this is that I can manage and develop each submodule independently, and adding one of these submodules to a new project is as simple as git cloneing and git submodule adding it. Because I work in a managed, shared computing environment, this also makes it easy to run the program since user environment management & software versioning and installation are contentious issues.
The disadvantage is that in this example, I now have 3 copies of the tools submodule, which are independent of each other and have to be manually updated everytime there is a change in any of them. Doing any sort of development on the submodules becomes very cumbersome. Similarly, it has now tripled the number of unit tests I run, since tests get run in each submodule and there are 3 copies of the tools module.
I have seen various importing methods, such as those mentioned here but that does not seem like an ideal solution for this.
I have read about how to create a formal Python package here but this seems like a large undertaking and will make it much more difficult for my end users to actually install and run the program.
Another relevant question asked here

Better to have a single tool in the parent and import it to the submodule. that's by far feel best to me.

Create a python executable using setuptools

I have a small python application that I would like to make into a downloadable / installable executable for UNIX-like systems. I am under the impression that setuptools would be the best way to make this happen but somehow this doesn't seem to be a common task.
My directory structure looks like this:
myappname/
|-- setup.py
|-- myappname/
| |-- __init__.py
| |-- myappname.py
| |-- src/
| |-- __init__.py
| |-- mainclassfile.py
| |-- morepython/
| |-- __init__.py
| |-- extrapython1.py
| |-- extrapython2.py
The file which contains if __name__ == "__main__": is myappname.py. This file has a line at the top, import src.mainclassfile.
When this is downloaded, I would like for a user to be able to do something like:
$ python setup.py build
$ python setup.py install
And then it will be an installed executable which they can invoke from anywhere on the command line with:
$ myappname arg1 arg2
The important parts of my setup.py are like:
from setuptools import setup, find_packages
setup(
name='code2flow',
scripts=['myappname/myappname.py'],
package_dir={'myappname': 'myappname'},
packages=find_packages(),
)
Current state
By running:
$ sudo python setup.py install
And then in a new shell:
$ myapp.py
I am getting a No module named error

The problem here is that your package layout is broken.
It happens to work in-place, at least in 2.x. Why? You're not accessing the package as myappname—but the same directory that is that package's directory is also the top-level script directory, so you end up getting any of its siblings via old-style relative import.
Once you install things, of course, you'll end up with the myappname package installed in your site-packages, and then a copy of myappname.py installed somewhere on your PATH, so relative import can't possibly work.
The right way to do this is to put your top-level scripts outside the package (or, ideally, into a bin directory).
Also, your module and your script shouldn't have the same name. (There are ways you can make that work, but… just don't try it.)
So, for example:
myappname/
|-- setup.py
|-- myscriptname.py
|-- myappname/
| |-- __init__.py
| |-- src/
| |-- __init__.py
| |-- mainclassfile.py
Of course so far, all this makes it do is break in in-place mode the exact same way it breaks when installed. But at least that makes things easier to debug, right?
Anyway, your myscriptname.py then has to use an absolute import:
import myappname.src.mainclassfile
And your setup.py has to find the script in the right place:
scripts=['myscriptname.py'],
Finally, if you need some code from myscriptname.py to be accessible inside the module, as well as in the script, the right thing to do is to refactor it into two files—but if that's too difficult for some reason, you can always write a wrapper script.
See Arranging your file and directory structure and related sections in the Hitchhiker's Guide to Packaging for more details.
Also see PEP 328 for details on absolute vs. relative imports (but keep in mind that when it refers to "up to Python 2.5" it really means "up to 2.7", and "starting in 2.6" means "starting in 3.0".
For a few examples of packages that include scripts that get installed this way via setup.py (and, usually, easy_install and pip), see ipython, bpython, modulegraph, py2app, and of course easy_install and pip themselves.

How to run python test files

I am developing python in eclipse. As a result, python src files and test files are in different directories.
Question is: How do we run on command line specific test files in the test folder? These obviously depend on files in the src folder.
Cheers
Edit: if I run
python test/myTestFile.py
I get dependency errors, eg. ImportError: No module named SrcFile1

You need to make sure your PYTHONPATH is set correctly so the command-line interpreter can find your packages, or run your test cases from within Eclipse Pydev. Update: Another option: running your tests using nose might make things a bit easier, since it can auto-discover packages and test cases.
If your project is laid out like so:
/home/user/dev/
src/pkg1/
mod1.py
test/
mod1_test.py
Use: PYTHONPATH=$HOME/dev/src python test/mod1_test.py. I'd also recommend using distribute and virtualenv to set up your project for development.
Updated in response to question in comments:
This shows how the PYTHONPATH environment variable extends Python's package sear
ch path:
% PYTHONPATH=foo:bar python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
# exporting the variable makes it sticky for your current session. you can
# add this to your shell's resource file (e.g. ~/.profile) or source
# it from a textfile to save typing:
% export PYTHONPATH=bar:baz
% python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
% python -c 'import sys; print sys.path[:3]'
['', '/home/user/foo', '/home/user/bar']
The above should get you going in the short term. Using distribute and
virtualenv have a higher one-time setup cost but you get longer-term benefits
from using them. When you get a chance, read some of the many tutorials on SO for setting these up to see if they're a good fit for your project.

There are 2 principal solutions to this. Either, you need to use e.g. PYTHONPATH environment variable to tell the tests where the source is, or you need to make tests and production code part of the same module tree by inserting the relevant __init__.py files. In the latter approach, the tree may look something like this:
|-- qbit
| |-- __init__.py
| |-- master.py
| |-- policy.py
| |-- pool.py
| |-- synchronize.py
| `-- worker.py
`-- test
|-- __init__.py
|-- support.py
|-- test_policy.py
|-- test_synchronize.py
`-- test_worker.py
__init__.py can be an empty file.