I have an issue where I am developing a Django project which includes other libraries we are also developing.
My current structure is as follows:
Main Project
App1
App2
Libraries
Library 1
Library 2
All libraries have their own setup scripts and are in separate git repositories, and we are adding them in PyCharm in the PYTHONPATH, and referencing them simply by their name. Which works good, but they are not in my current project, which means no re-factoring ( renaming, moving etc... ) and I have to use External search to find my class from the libraries.
How do I set some libraries as project related to make them view-able and refactorable like we do on the currently set project.
Well, you can add other directories as content roots:
Then simply mark the directory as a source root:
This should allow you to refactor, rename and do all the things you've wanted to do.
Another option would be to place libraries into separate project (or go even further and place each library in its own project) and then open this project/these projects side-by-side with the main project. This way you have clear separation between the main project and libraries used. This comes handy when you work on another project using some of the same libraries as then you only need to open already existing project containing libraries and you are done.
Related
I am new to Pycharm and need some help. I am working on a project that makes use of a large library of modules (specifically, Schrodinger; which allows for a lot of cool chemistry programs). Schrodinger requires the use of Python 2.7, if that makes any difference.
There are too many modules to install to the project directory. When I move the project directory to the location of the modules, my script becomes stuck on 'initializing'. I have attempted to import it as a package to no avail.
I have also tried to use the sys.path command, however a lot of the modules make use of other modules as well. So I that has become a pain very quickly.
How can I use these modules within Pycharm? And if there is no easy way, do you have a recommendation for an IDE that does have this feature?
Thanks
Pycharm doesn't identifies user defined modules which are not imported to Pycharm.
I usually mask the module as a Sources Root see the picture for more details. if the modules are in same project.
Alternative way: In your case import the external modules using File -> Open modules with open -> open in current window -> add to currently opened project this looks like two different projects. Now you can mark Sources Root for the complete module (i.e. learning) which you have imported.
import stackoverflow
Now pycharm can identifies the user defined modules.
I'm climbing my learning curve in Python and try to understand where to put everything.
I originally have a python module in a folder and then a sub folder src, in this src folder I will then have my main source files say main.py then I will have models folder storing my models codes.
/myproject/src/main.py
/myproject/src/models/a-model.py
/myproject/src/models/b-model.py
So my main will import the model like this:
from models.a-model import a
Then when I package the zip file I just zip the myproject folder with that folder structure and deploy and everything is fine.
Now I have another new module doing something different but need to use the same models.
I can easily duplicate them all and code separately and deploy. But I would like to share the codes to the models, so that when one model changes, I only need to update once, instead of 2 places.
My new module is like
/mynew/src/main-b.py
/mynew/src/models/a-model.py
/mynew/src/models/b-model.py
What is the best practise to do this?
Do I put like this?
/myproject/src/main.py
/mynew/src/main-b.py
/models/a-model.py
/models/b-model.py
And then update the import?
But I have doubt how do I deploy? Do I also have to setup the same folder structures?
One would be adding /myproject/src/models to the environment variable PYTHONPATH. Python adds the directories listed in PYTHONPATH environment variable to sys.path, the list of directories where Python searches when you try to import something. This is bad, because modifying PYTHONPATH has its own side effects, fortunately, virtual environments provide a way to get around those side effects.
Alternatively and much better you could add your modules to site-packages directory, site-packages is added to sys.pathby default, this obviates the need to modifyPYTHONPATH. To locate thesite-packages` directory, refer to this page from Python Documentation: Installing Python Modules (Legacy version).
You could also use LiClipse IDE which comes with Pydev already installed. Create source a folder from the IDE and link your previous project with your newer project. When you link your projects the IDE adds the source folders of your older project to the PYTHONPATH of your newer project and thus Python will be able to locate your modules.
I'm working toward adopting Python as part of my team's development tool suite. With the other languages/tools we use, we develop many reusable functions and classes that are specific to the work we do. This standardizes the way we do things and saves a lot of wheel re-inventing.
I can't seem to find any examples of how this is usually handled with Python. Right now I have a development folder on a local drive, with multiple project folders below that, and an additional "common" folder containing packages and modules with re-usable classes and functions. These "common" modules are imported by modules within multiple projects.
Development/
Common/
Package_a/
Package_b/
Project1/
Package1_1/
Package1_2/
Project2/
Package2_1/
Package2_2/
In trying to learn how to distribute a Python application, it seems that there is an assumption that all referenced packages are below the top-level project folder, not collateral to it. The thought also occurred to me that perhaps the correct approach is to develop common/framework modules in a separate project, and once tested, deploy those to each developer's environment by installing to the site-packages folder. However, that also raises questions re distribution.
Can anyone shed light on this, or point me to a resource that discusses this issue?
If you have common code that you want to share across multiple projects, it may be worth thinking about storing this code in a physically separate project, which is then imported as a dependency into your other projects. This is easily achieved if you host your common code project in github or bitbucket, where you can use pip to install it in any other project. This approach not only helps you to easily share common code across multiple projects, but it also helps protect you from inadvertently creating bad dependencies (i.e. those directed from your common code to your non common code).
The link below provides a good introduction to using pip and virtualenv to manage dependencies, definitely worth a read if you and your team are fairly new to working with python as this is a very common toolchain used for just this kind of problem:
http://dabapps.com/blog/introduction-to-pip-and-virtualenv-python/
And the link below shows you how to pull in dependencies from github using pip:
How to use Python Pip install software, to pull packages from Github?
The must-read-first on this kind of stuff is here:
What is the best project structure for a Python application?
in case you haven't seen it (and follow the link in the second answer).
The key is that each major package be importable as if "." was the top level directory, which means that it will also work correctly when installed in a site-packages. What this implies is that major packages should all be flat within the top directory, as in:
myproject-0.1/
myproject/
framework/
packageA/
sub_package_in_A/
module.py
packageB/
...
Then both you (within your other packages) and your users can import as:
import myproject
import packageA.sub_package_in_A.module
etc
Which means you should think hard about #MattAnderson's comment, but if you want it to appear as a separately-distributable package, it needs to be in the top directory.
Note this doesn't stop you (or your users) from doing an:
import packageA.sub_package_in_A as sub_package_in_A
but it does stop you from allowing:
import sub_package_in_A
directly.
...it seems that there is an assumption that all referenced packages
are below the top-level project folder, not collateral to it.
That's mainly because the current working directory is the first entry in sys.path by default, which makes it very convenient to import modules and packages below that directory.
If you remove it, you can't even import stuff from the current working directory...
$ touch foo.py
$ python
>>> import sys
>>> del sys.path[0]
>>> import foo
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named foo
The thought also occurred to me that perhaps the correct approach is
to develop common/framework modules in a separate project, and once
tested, deploy those to each developer's environment by installing to
the site-packages folder.
It's not really a major issue for development. If you're using version control, and all developers check out the source tree in the same structure, you can easily employ relative path hacks to ensure the code works correctly without having to mess around with environment variables or symbolic links.
However, that also raises questions re distribution.
This is where things can get a bit more complicated, but only if you're planning to release libraries independently of the projects which use them, and/or having multiple project installers share the same libraries. It that's the case, take a look at distutils.
If not, you can simply employ the same relative path hacks used in development to ensure you project works "out of the box".
I think that this is the best reference for creating a distributable python package:
link removed as it leads to a hacked site.
also, don't feel that you need to nest everything under a single directory. You can do things like
platform/
core/
coremodule
api/
apimodule
and then do things like from platform.core import coremodule, etc.
Is there a standard file in python which lists all the modules comprising the project, and other metadata?
Is this simply the 'package'? Or, do different IDEs use their own specific files?
There really isn't a single file in any package that consistently lists every module the entire package imports. Some people make entries to the __init__.py and some don't. Usually most python supported IDE's will make available to you whatever is on your pythonpath. Eclipse pydev, for instance, will add the specific project to the pythonpath of that project space.
If your project is on the pythonpath, then it should resolve.
Application builders like py2app/py2exe will scan the entire project and create an import graph to discover every module needed for that project
There's no real equivalent in Python by itself. Python packages are the way to encapsulate a set of modules and include metadata, but it isn't exactly equivalent to the notion of a "project."
Otherwise, are some projects which use project files in order to give you some of the features which IDEs provide. In particular, you should check out the rope library and the PyCharm IDE for some systems which implement a project file.
Here is the situation: the company that I'm working in right now gave me the freedom to work with either java or python to develop my applications. The company has mainly experience in java.
I have decided to go with python, so they where very happy to ask me to give maintenance to all the python projects/scripts related to the database maintenance that they have.
Its not that bad to handle all that stuff and its kind of fun to see how much free time I have compared to java programmers. There is just one but, the projects layout is a mess.
There are many scripts that simply lay in virtual machines all over the company. Some of them have complex functionality that is spread across a few modules(4 at maximum.)
While thinking about it about it, I realized that I don't know how to address that, so here are 3 questions.
Where do I put standalone scripts? We use git as our versioning system.
How do structure the project's layout in a way that the user do not need to dig deep into the folders to run the programs(in java I created a jar or a jar and a shell script to handle some bootstrap operations.)
What is a standard way to create modules that allow easy reusability(mycompany.myapp.mymodule?)
Where do I put standalone scripts?
You organize them "functionally" -- based on what they do and why people use them.
The language (Python vs. Java) is irrelevant.
You have to think of scripts as small applications focused on some need and create appropriate directory structures for that application.
We use /opt/thisapp and /opt/thatapp. If you want a shared mount-point, you might use a different path.
How do structure the project's layout in a way that the user do not need to dig deep into the folders to run the programs
You organize them "functionally" -- based on what they do and why people use them. At the top level of a /opt/thisapp directory, you might have an __init__.py (because it's a package) and perhaps a main.py script which starts the real work.
In Python 2.7 and Python 3, you have the runpy module. With this you would name your
top-level main script __main__.py
http://docs.python.org/library/runpy.html#module-runpy
What is a standard way to create modules that allow easy reusability(mycompany.myapp.mymodule?)
Read about packages. http://docs.python.org/tutorial/modules.html#packages
A package is a way of creating a module hierarchy: if you make a file called __init__.py in a directory, Python will treat that directory as a package and allow you to import its contents using dotted imports:
spam \
__init__.py
ham.py
eggs.py
import spam.ham
The modules inside a package can reference each other -- see the docs.
If these are all DB maintenance scripts, I would make a package called DB or something, and place them all in it. You can have subpackages for the more complicated ones. So if you had a script for, I don't know, cleaning up the transaction logs, you could put it in ourDB.clean and do
import ourDB.clean
ourDB.clean.transaction_logs( )