I'm in the process of splitting up a monolithic project code base into several smaller projects. I'm having a hard time understanding how to handle dependencies amongst the different projects properly.
The structure looks somewhat like this:
SCM_ROOT
|-- core
| |-- src
| `-- setup.py
|-- project1
| |-- src
| `-- setup.py
|-- project2
| |-- src
| `-- setup.py
`-- project3
|-- src
`-- setup.py
What's the recommended way to handle dependencies between multi-package projects and setup a development environment? I'm using pip, virtualenv and requirements.txt files. Are there any tools that allow me bootstrap my environment from the repository quickly?
Using a build tool like Pybuilder or Pants was unnecessarily complicating the process. I ended up splitting it up into multiple projects in svn - each with it's own trunk/tags/branches directories. Dependencies are handled using a combination of install_requires and requirements.txt file based on information from here and here. Each project has a fabfile to run common tasks like clean, build, upload to pypi etc.
Related
I use pants to manage a Python project that uses protocol buffers. Pants places the generated _pb2.py and _pb2.pyi files under a separate dist/codegen tree. Is it possible to get VS Code autocomplete to work when using the _pb2 modules?
The file tree looks like this:
.
|-- dist/
| `-- codegen/
| `-- src/
| `-- project/
| |-- data_pb2.py
| `-- data_pb2.pyi
`-- src/
`-- project/
|-- __init__.py
|-- code.py
`-- data.proto
And in code.py I have import statements like this:
from project import data_pb2
I've tried setting python.analysis.extraPaths to ["dist/codegen/src"] in settings.json. This makes pylance stop complaining that data_pb2 is missing. But autocomplete still does not work, and pylance has no type information for members of data_pb2.
Replace your python.analysis.extraPaths with the following extent:
"python.analysis.extraPaths": [
"./dist/codegen/src"
],
And adding the following code to your code.py:
import sys
sys.path.append(".\dist\codegen\src")
You can use Python implicit namespace packages (PEP 420) to make this work. Namespace packages are allowed to have modules within the same package reside in different directories. Which allows pylance and other tools to work correctly when code is split between src and dist/codegen/src.
To use implicit namespace packages, you just need to remove src/package/__init__.py, and leave "python.analysis.extraPaths" set to ["dist/codegen/src"].
See also the GitHub issue microsoft/pylance-release#2855, which describes using implicit namespace packages to make pylance work correctly in a similar situation.
I am working with ROS packages and coming from this tutorial. This import statement surprisingly works given the absence of AddTwoIntsResponse anywhere in the current working directory or any other directory listed in PATH. Also how come a .srv gets imported?
# add_two_ints_server.py
from beginner_tutorials.srv import AddTwoInts, AddTwoIntsResponse
The current working directory is ~/catkin_ws/src/beginner_tutorials/srv
This is my ROS directory layout:
catkin_ws
|-- src
| `-- beginner_tutorials
| |-- scripts
| | `-- add_two_ints_server.py
| `-- srv
| `-- AddTwoInts.srv
|-- build
`-- devel
The contents of AddTwoInts.srv are:
int64 a
int64 b
---
int64 sum
According to my understanding this should throw an ImportError: cannot import name 'AddTwoIntsResponse', but it doesn't. Importing any other file say: from beginner_tutorials.srv import foo throws an ImportError.
Where is my understanding going wrong?
|-- src
| `-- beginner_tutorials
| |-- scripts
| | `-- add_two_ints_server.py
| `-- srv
| `-- AddTwoInts.srv
|-- build
|-- devel // this is where your modules are imported from
When you build the package using catkin_make, 'catkin` generates the relevant python files for your service type defined in .srv file and puts them under catkin_ws/devel/lib/your-python-version/dist-packages/package-name/srv.
If your workspace is sourced, catkin_ws/devel/lib/your-python-version/dist-packages/ is already added to your PYTHONPATH and that is how you are able to import them successfully.
In case of the tutorial package that you are using, imports may work even when you haven't sourced your current catkin-directory, if you have the binaries of the tutorials installed. This way the python modules reside under /opt/ros/ros-version/lib/your-python-version/dist-packages/ and that is again part of the PYTHONPATH. (If ROS env is available)
I'm working on a large code base and I'd like to set up rope projects so that rope is fast and does what I want. For what it's worth, I'm using rope with emacs, but if I understand correctly, rope's behavior should be independent of the editor.
The code base has many core libraries used by many apps. Each app depends on one or more core libraries but never on another app. Here's a simplified representation of the directory structure:
repo
|-- core
| |--CoreLib1
| | |-- CoreLib1.egg-info
| | `-- library_module
| | |-- __init__.py
| | `-- lib.py
| `--CoreLib2
| |-- CoreLib2.egg-info
| `-- library_module
| |-- __init__.py
| `-- lib.py
`-- apps
|-- AppA
| |-- AppA.egg-info
| `-- app_a_module
| |-- __init__.py
| `-- src.py
|-- AppB
| |-- AppB.egg-info
| `-- app_b_module
| |-- __init__.py
| `-- src.py
`-- AppC
|-- AppC.egg-info
`-- app_c_module
|-- __init__.py
`-- src.py
What I want to do
Currently I have repo/.ropeproject, and rope behaves how I want, but is slow. I believe the slowness is because it's analyzing all the code in all the apps at any given time. To solve this, I'm trying to create a rope project in each app (e.g. /repo/apps/AppA/.ropeproject) that knows about core but doesn't know about the other apps. The problem is, I can't get it to know about core. This means I can't do any rope operations on any names from core.
Works, but is slow:
Make sure there are no .ropeproject directories in the whole code base. Create a rope project in /repo/.
Put the following code in /repo/.ropeproject/config.py:
src_dirs = [
dirpath
for dirpath, dirnames, filenames in os.walk('core/')
if any(map(lambda dirname: dirname.endswith('.egg-info'), dirnames))
]
for src_dir in src_dirs:
prefs.add('python_path', src_dir)
Reload the rope project (to make sure it's using the freshly updated config.py)
Rope Generate Autoimport Cache. This takes upwards of 60 seconds and can't run in the background.
While editing repo/apps/AppA/app_a_module/src.py, attempt to use rope to auto-import a name from core. It works.
Fast, but doesn't work:
Make sure there are no .ropeproject directories in the whole code base. Create a rope project in /repo/apps/AppA/.
Put the following code in apps/AppA/.ropeproject/config.py:
src_dirs = [
dirpath
for dirpath, dirnames, filenames in os.walk('../../core/')
if any(map(lambda dirname: dirname.endswith('.egg-info'), dirnames))
]
for src_dir in src_dirs:
prefs.add('python_path', src_dir)
Reload the rope project (to make sure it's using the freshly updated config.py)
Rope Generate Autoimport Cache. This takes less than 1 second.
While editing repo/apps/AppA/app_a_module/src.py, attempt to use rope to auto-import a name from core. It fails.
Is what I want to do within the scope of rope's capabilities? Should it be able to work? If so, what am I doing wrong?
I work on a couple of different programs and packages in Python. They are each developed in their own Git repository, but frequently need to import modules defined in other packages. For instance, during development, the directory structure looks something like:
src/
|-- project-a/
| |-- client.py
| |-- server.py
| |-- package-a
| |-- __init__.py
| |-- module.py
|-- project-b/
| |-- package-b
| | |-- __init__.py
| | |-- other_module.py
| |-- package-c
| |-- __init__.py
| |-- third_module.py
|-- project-c/
|-- server1.py
|-- server2.py
|-- package-d/
|-- package-e/
|-- package-f/
When they are all installed, they work fine; they are all installed such that each package is in your Python path, and you can import from them as you need.
However, in development, I want the development version of each of these to be in my Python path, not the installed version. When making changes, I don't want to have to install each package that I'm changing to test it, I want the changes to take effect immediately. That means my Python path needs to include the directories project-a, project-b, etc.
Our current solution is just to have an environment.bash in the top level, which you source in your shell and it sets PYTHONPATH. That works OK, but I frequently forget to do so; since this is a client server application, with communications between servers, I need to have at least four windows open to different VMs to run this, and it happens pretty often that I forget to source environment.bash in at least one of those, leading me to try debugging strange behavior until I realize I'm importing the wrong things.
Another solution would be to set sys.path from within the top level client.py or server.py. This would work fine for launching them directly, but I would also need the path set up for running tools like Pylint or Sphinx, which that solution wouldn't cover. I'd also need a way to distinguish between running from source (when I want the path to include . and ../project-b) and running the installed version (which should use the standard path without modification).
Another choice would be to have a Makefile which sets up PYTHONPATH appropriately for various targets like make run-server, make lint, make doc, and so on. That's OK for those targets, which don't require any options, but would be inconvenient for running the client, which takes arguments. make run-client ARGS='foo bar' is a fairly cumbersome way to invoke it.
Is there any common way of setting up the Python path during development so that both my executables and tools like Pylint and Sphinx can pick it up appropriately, without interfering with how it will behave when installed?
A straightforward solution would be to simply symlink in the directories for each module in a separate folder, and run things from there. That way Python sees them all being in the same location, even though the actual sources are in different repositories.
src/
|-- project-a/
| |-- client.py
| |-- server.py
| |-- package-a
| |-- __init__.py
| |-- module.py
|-- project-b/
|-- package-b
| |-- __init__.py
| |-- other_module.py
|-- package-c
|-- __init__.py
|-- third_module.py
run/
|-- client.py --> ../src/project-a/client.py
|-- server.py --> ../src/project-a/server.py
|-- package-a/ --> ../src/project-a/package-a/
|-- package-b/ --> ../src/project-b/package-b/
|-- package-c/ --> ../src/project-b/package-c/
I have multiple projects that shares child apps with other projects.
When working within the project directory I want to be able to make changes to the app, update it, and pull those updates into the second project.
Requirement:
No use of symbolic links (my IDE's debugger doesn't work well with them)
No compiling/rerun a script. I would like to make changes to the app without having to rerun a script/buildout.
Apps must be within the project folder.
Here's the structure:
app_one (git repo)
|-- app_one (actual app uses by projects)
| +-- models.py
|-- README.md
+-- setup.py
project_one (git repo)
|-- project_one
| |-- apps
| | |-- app_one
| | | +-- models.py
| | | -- app_two
|-- setup.cfg
+-- setup.py
project_two (git repo)
|-- project_two
| |-- apps
| | |-- app_one (same app as project_one)
| | | +-- models.py
| | | -- app_two
|-- setup.cfg
+-- setup.py
Currently I'm using git-submodules for this; the downside is there is no way to link to a subfolder of a repo. I recently read about subtree, would this work better?
Ideally I would like to use buildout, but I haven't found a good way to accomplish this without the use of symbolic links. If there's a way to to do this please let me know.
Any suggestions would be greatly appreciated.
mr.developer. When a mr.developer source is checked out and activated, it becomes a setuptools/distribute develop egg in the buildout and so the checkout will be what is used by any scripts (such as a zc.recipe.egg generated python interpreter) in the buildout.