I'm still pretty new to Python.
I have a Django website with several apps and a /libs directory. I need to add a couple cron jobs that will use my Django models. I've already worked all that out, no big deal.
I have a problem with my imports, though.
I would like to include these scripts in the App they generally belong-to. But when I execute the script from the commandline, it's unable to load modules from the /libs directory.
To illustrate specifically the dir structure, imagine this pseudo-code:
import ./../libs/mysharedlib.py
In other words the scripts and sharedlib are:
~/project/myapp/myscript.py
~/project/libs/mysharedlib.py
I get that I could just add ~/project to my PYTHONPATH but then I have to worry about doing this during deployment and it just feels broken.
Should I move my scripts out of my apps and put them in the ~/project? Is there a better way?
There is a better way, custom management commands solve this.
https://docs.djangoproject.com/en/dev/howto/custom-management-commands/
These let you write stand alone utility scripts. You can run these as a cron or just as utilities. They use the exact same paths as any other module in your django app.
While these solve import problems does your libs directory have a __init__.py file? Can you import your lib directory in your views? Or is your import just not working in your cron scripts? Either way custom managment commands address this.
Related
I have a question about adding project path to python, for facilitating import effort.
Situation
When I write code in Python, I usually add necessary path to sys.path by using
import sys
sys.path.append("/path/to/dir/") # almost every `.py` need this
Sometimes, when my project gets bigger with many levels of directories, this approach seems bulky and error-prone (especially when I re-organize my files)
Recently, I start using a bash script (located at project root directory) that adding the sys.path.append with project root argument to .py file in the project. With this approach, I hardly have to manually care about importing a module.
Question
My question is: Is that a good practice? I find it convenient for myself, compared to my old method, but since the bash script is a separated file, I need 2 command to run any script in my project (one for the bash and one for the .py). I can include the command calling .py to the bash, but it far less flexible than directly call it from terminal.
Really want to hear some advices! Thanks in advance. Any suggestion will be gratefully appreciated!
It is generally not good practice to use manipulate sys.path within a python library or program. You should add the relevant paths to the PYTHONPATH in the calling environment for your python program:
PYTHONPATH="/path/to/other/projects/directory:$PYTHONPATH" python ...
or
export PYTHONPATH="/path/to/other/projects/directory:$PYTHONPATH"
python ...
This allows you to easily manipulate the paths that your program or library will search for dependencies easily without modifying your code.
It is also very easy to manage this in your personal development environment by modifying your bashrc or in your production environments in your init script (or other wrapper script) and provides you with one place to update each time you add or modify your project paths.
Given that you mention that you have almost one directory per .py file, you should also consider how your code might be organized into packages to further simplify your setup.
It's not a particularly good practice, though you could get away with it. Better to look into virtualenv though (or pipenv) for a smoother workflow.
This issue has been driving me insane for the past few days.
So basically, I'm trying to port over a Pure Python project to a proper PyCharm project. This is to basically improve code quality and project structure.
I wish it was as simple as basically creating a virtualenv to house everything, but it isn't. This project will eventually be developed simultaneously by multiple developers with Git as source control, and the default libraries will be modified. I presume this means that the libraries should ideally be tracked by Git in the end. Virtualenv shouldn't help here as far as I know because it's not portable between systems (or at least that's still being tested).
This project will also be, in the future, deployed to a Centos server.
So the only plan I can think of to successfully pull off this would be to simply bring in all of the libraries (which was done using pip install -t Libraries <ExampleLibrary>) into a single folder, with a __init__.py inside, and use them from other python files as a package within the Pycharm project.
Is this possible / recommended? I tried various methods to reference these libraries, but they all don't work during runtime. Somehow when the files in the library import something else from their own package, an ImportError is raised saying that there's no such module.
Will accept any other suggestions too.
Using Pycharm Community Edition.
EDIT: After having a good night's rest I think the crux of the issue is really just project organization. Before I ported it over to Pycharm the project worked as expected, but this had all of the python files in the root directory, and the libraries in a subfolder of the root, with every project file having the same boilerplate code:
import os, sys
absFilePath = os.path.dirname(os.path.abspath(__file__));
sys.path.insert(1, absFilePath + "/lib")
I was hoping that by using Pycharm to help me flesh out the packages, I could avoid having repeated boilerplate code.
Note: Not full solution.
The addition of the template code below forces the file containing the code to be in the same directory as the libs folder.
For Pycharm, all I had to do was mark the libs folder as a source folder. Even with the addition of the template code to the file, the modified libraries still work as expected.
For the Python Shell, this template code is still needed:
import os
import sys
absFilePath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(1, absFilePath + "/lib")
Currently, I'm starting to work on a Django project which it's already begun.
Don't misunderstand me, I already know how basic files and dependencies work and relate to each other on this framework (forms, views, models, urls, templates, etc)
But my problem is, that this is a project with lots of files, classes, models, constants, etc... And has lots of code not written by myself.
I'm not very familiar to these internal custom objects, so, I need a module or script on python which can read all folders and map all relationships and dependencies (internal project objects, not modules or requirements), so I can better understand what's already done and work more efficiently on these modifications I have to do.
Currently I'm studying it manually, but it's very unefficient, and a slow process, so, if anyone knows about a tool or maybe a script, of these characteristics, please let me know where I can find it, download it, etc...
Thanks in advance!
Python has a builtin module called modulefinder that can be run from the command line to trace dependencies. It can be invoked like
python -m modulefinder mymodule.py
to print module dependencies of mymodule.py
I'm writing a deployment engine for our code.
I plan to put a "deploy.py" in each project directory and then decide what projects need to be deployed, having the "deploy.py" of each project execute his logic for proper deployment configuration.
How would you recommend doing this? I've been thinking about "executing" the python code in each deploy.py file from within the "deployment_engine.py" code (I got this idea from scons method of doing stuff).
Alternativly I would like to somehow "import" the code (note that under each project the file is named "deploy.py" so a simple import deploy.py doesn't seem to do the job here)
Thank you,
Maxim.
You can use the __import__ function to do dynamic imports.
Here is the situation: the company that I'm working in right now gave me the freedom to work with either java or python to develop my applications. The company has mainly experience in java.
I have decided to go with python, so they where very happy to ask me to give maintenance to all the python projects/scripts related to the database maintenance that they have.
Its not that bad to handle all that stuff and its kind of fun to see how much free time I have compared to java programmers. There is just one but, the projects layout is a mess.
There are many scripts that simply lay in virtual machines all over the company. Some of them have complex functionality that is spread across a few modules(4 at maximum.)
While thinking about it about it, I realized that I don't know how to address that, so here are 3 questions.
Where do I put standalone scripts? We use git as our versioning system.
How do structure the project's layout in a way that the user do not need to dig deep into the folders to run the programs(in java I created a jar or a jar and a shell script to handle some bootstrap operations.)
What is a standard way to create modules that allow easy reusability(mycompany.myapp.mymodule?)
Where do I put standalone scripts?
You organize them "functionally" -- based on what they do and why people use them.
The language (Python vs. Java) is irrelevant.
You have to think of scripts as small applications focused on some need and create appropriate directory structures for that application.
We use /opt/thisapp and /opt/thatapp. If you want a shared mount-point, you might use a different path.
How do structure the project's layout in a way that the user do not need to dig deep into the folders to run the programs
You organize them "functionally" -- based on what they do and why people use them. At the top level of a /opt/thisapp directory, you might have an __init__.py (because it's a package) and perhaps a main.py script which starts the real work.
In Python 2.7 and Python 3, you have the runpy module. With this you would name your
top-level main script __main__.py
http://docs.python.org/library/runpy.html#module-runpy
What is a standard way to create modules that allow easy reusability(mycompany.myapp.mymodule?)
Read about packages. http://docs.python.org/tutorial/modules.html#packages
A package is a way of creating a module hierarchy: if you make a file called __init__.py in a directory, Python will treat that directory as a package and allow you to import its contents using dotted imports:
spam \
__init__.py
ham.py
eggs.py
import spam.ham
The modules inside a package can reference each other -- see the docs.
If these are all DB maintenance scripts, I would make a package called DB or something, and place them all in it. You can have subpackages for the more complicated ones. So if you had a script for, I don't know, cleaning up the transaction logs, you could put it in ourDB.clean and do
import ourDB.clean
ourDB.clean.transaction_logs( )