Update 2021: Solution is built into PyDev/Eclipse
See accepted answer for details
Original Question (and old answers) for below
As many comments/questions/rants on SO and other places will tell you, Python3 packages using relative imports want to be run from a central __main__.py file. In the case where a module, say "modA" within a package, say "packA", that uses relative imports needs to be run (for instance because a test package is run if __name__ == '__main__'), we are told to run instead run python3 -m modA.packA from the directory above modA if sys.path() does not contain the directory above modA. I may dislike this paradigm, but I can work around it.
When trying to run modA from Eclipse/PyDev, however, I can't figure out how to specify a run configuration that will execute the module properly with the -m flag. Has anyone figured out how to set up a run configuration that will do this properly?
References: Relative imports for the billionth time ; Relative import in Python 3 is not working ; Multilevel relative import
Nowadays (since PyDev 5.4.0 (2016-11-28)) you can go to the Settings > PyDev > Run and select Launch modules with python -m mod.name instead of python filename.py ;)
See: https://www.pydev.org/history_pydev.html
For older versions of PyDev (old answer)
Unfortunately, right now, it's not automatic running with -m in PyDev, so, I'll present 3 choices to work in PyDev with relative imports which are preceded by a dot (in PyDev version 4.3.0):
Don't use relative imports, only absolute imports in your __main__ modules.
Create a separate module for the __main__ which will do an absolute import for the module you want to run and run that module instead (if you're distributing your application, this is probably needed anyways as the usual way for people to launch your code in Python is by passing the script as an argument to Python and not using the -m switch).
Add the -m module to the vm arguments in your run configuration by doing:
Make the run (which will fail because of the relative import)
Right-click the editor > Copy Context Qualified Name
Open the run configuration: Alt, R, N (i.e.: Toolbar > Run > Run Configuration)
Open arguments tab and add the '-m Ctrl+V' (to add the -m and the module name you copied previously).
Although this is definitely not ideal: you'll now receive an argument with the filename (as PyDev will always pass that to run the file) and the whole process is a nuisance.
As a note, I do hope to provide a way to make runs within PyDev using the -m soon (hopefully for PyDev 4.4.0)... although this may not be possible if the file being run is not under the PYTHONPATH (i.e.: to run an external file it still has to support the option without the -m).
There's a bit nasty trick possible here to work around this issue. I'm using PyDev 9.2.0
Put your venv right in the workspace, say under the dir "venv".
Refresh your eclipse workspace and ensure that it uses this venv (through your interpreter setup).
After the refresh, go to the run configuration and edit the "Main Module" by clicking the Browse button.
The venv will now appear.
Browse into the venv/lib/python3.8/site-packages
There you will find the pip-installed module source codes and you can select the module you want to run.
Update 2021: This answer is no longer needed. See accepted answer for details.
Here's what I was able to do after Fabio's great suggestion.
Create a program called /usr/local/bin/runPy3M with world read/execute permissions, with the following code:
#!/usr/local/bin/python3 -u
'''
Run submodules inside packages (with relative imports) given
a base path and a path (relative or absolute) to the submodule
inside the package.
Either specify the package root with -b, or setenv ECLIPSE_PROJECT_LOC.
'''
import argparse
import os
import re
import subprocess
import sys
def baseAndFileToModule(basePath, pyFile):
'''
Takes a base path referring to the root of a package
and a (relative or absolute) path to a python submodule
and returns a string of a module name to be called with
python -m MODULE, if the current working directory is
changed to basePath.
Here the CWD is '/Users/cuthbert/git/t/server/tornadoHandlers/'.
>>> baseAndFileToModule('/Users/cuthbert/git/t/', 'bankHandler.py')
'server.tornadoHandlers.bankHandler'
'''
absPyFilePath = os.path.abspath(pyFile)
absBasePath = None
if basePath is not None:
absBasePath = os.path.abspath(basePath)
commonPrefix = os.path.commonprefix([absBasePath, absPyFilePath])
uncommonPyFile = absPyFilePath[len(commonPrefix):]
else:
commonPrefix = ""
uncommonPyFile = absPyFilePath
if commonPrefix not in sys.path:
sys.path.append(commonPrefix)
moduleize = uncommonPyFile.replace(os.path.sep, ".")
moduleize = re.sub("\.py.?$", "", moduleize)
moduleize = re.sub("^\.", "", moduleize)
return moduleize
def exitIfPyDevSetup():
'''
If PyDev is trying to see if this program is a valid
Python Interpreter, it expects to function just like Python.
This is a little module that does this if the last argument
is 'interpreterInfo.py' and then exits.
'''
if 'interpreterInfo.py' in sys.argv[-1]:
interarg = " ".join([sys.executable] + sys.argv[1:])
subprocess.call(interarg.split())
exit()
return
exitIfPyDevSetup()
parser = argparse.ArgumentParser(description='Run a python file or files as a module.')
parser.add_argument('file', metavar='pyfile', type=str, nargs=1,
help='filename to run, with .py')
parser.add_argument('-b', '--basepath', type=str, default=None, metavar='path',
help='path to directory to consider the root above the package')
parser.add_argument('-u', action='store_true', help='unbuffered binary stdout and stderr (assumed)')
args = parser.parse_args()
pyFile = args.file[0]
basePath = args.basepath
if basePath is None and 'ECLIPSE_PROJECT_LOC' in os.environ:
basePath = os.environ['ECLIPSE_PROJECT_LOC']
modName = baseAndFileToModule(basePath, pyFile)
allPath = ""
if basePath:
allPath += "cd '" + basePath + "'; "
allPath += sys.executable
allPath += " -m " + modName
subprocess.call(allPath, shell=True) # maybe possible with runpy instead...
Then add a new interpreter to PyDev -- call it what you'd like (e.g., runPy3M) and point the interpreter to /usr/local/bin/runPy3M. Click okay. Then move it up the list if need be. Then add to environment for the interpreter, Variable: ECLIPSE_PROJECT_LOC (name here DOES matter) with value ${project_loc}.
Now submodules inside packages that choose this interpreter will run as modules relative to the sub package.
I'd like to see baseAndFileToModule(basePath, pyFile) added to runpy eventually as another run option, but this will work for now.
EDIT: Unfortunately, after setting all this up, it appears that this configuration seems to prevent Eclipse/PyDev from recognizing builtins such as "None", "True", "False", "isinstnance," etc. So not a perfect solution.
Related
#!/usr/bin/python
import requests, zipfile, StringIO, sys
extractDir = "myfolder"
zip_file_url = "download url"
response = requests.get(zip_file_url)
zipDocument = zipfile.ZipFile(StringIO.StringIO(response.content))
zipinfos = zipDocument.infolist()
for zipinfo in zipinfos:
extrat = zipDocument.extract(zipinfo,path=extractDir)
System configuration
Ubuntu OS 16.04
Python 2.7.12
$ python extract.py
when I run the code on Terminal with above command, it works properly and create the folder and extract the file into it.
Similarly, when I create a cron job using sodu rights the code executes but don't create any folder or extracts the files.
crontab command:-
40 10 * * * /usr/bin/sudo /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
also tried
40 10 * * * /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
Notes :
I check the syslog, it says the cron is running successfully
The above code gives no errors
also made the python program executable by chmod +x filename.py
Please help where am I going wrong.
Oups, there is nothing really wrong in running a Python script in crontab, but many bad things can happen because the environment is not the one you are used to.
When you type in an interactive shell python directory.py, the PATH and all required PYTHON environment variable have been set as part of login and interactive shell initialization, and the current directory is your home directory by default or anywhere you currently are.
When the same command is run from crontab, the current directory is not specified (but may not be what you expect), PATH is only /bin:/usr/bin and python environment variables are not set. That means that you will have to tweak environment variables in crontab file until you get a correct Python environment, and set the current directory.
I had a very similar problem and it turned out cron didn’t like importing matplotlib, I ended up having to specify Agg backend. I figured it out by putting log statements after each line to see how far the program got before it crapped out. Of course, my log was empty which tipped me off that it crashed on imports.
TLDR: log each line inside the script
I have this script (this is the beginning of a wsgi script for openshift applications). This script activates a virtualenv using the in-python system environment.
#!/usr/bin/python
import os
virtenv = os.environ['OPENSHIFT_PYTHON_DIR'] + '/virtenv/'
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
pass
What's the opposite? i.e.: is there an analogous way to do a "deactivate_this"? (a file like that does not seem to exist in any of my created virtualenvs)
This means: I don't want to reinvent the wheel. I'm asking if there's a prepared command line for that.
I believe the short answer is No.
The medium answer is "Not unless you have saved the state of the environment before executing activate_this.py".
Perhaps a way to achieve what you want is to separate the parts of your app that need to run in the venv from those that don't. Then use a subprocess to activate the venv and run whatever you want, when the subprocess terminates, your original process (in the original environment) can resume.
NB This post (How to leave/exit/deactivate a python virtualenv?) suggests the activation may provide a shell function to put-everything-back. You'd need to check if your activate_this.py script provides anything similar, though it sounds like you have already checked this.
This is a complement to the answer #TomDalton gave. Althought there's no an automatically provided way to do this, there are means provided by the activate_this script.
First, it is important to remember that this line:
execfile(virtualenv, dict(__file__=virtualenv))
is calling the function without passing a dictionary for globals nor a dictionary for locals. This will imply that the execution context will be the current globals (i.e. the one from the calling line) and locals objects. such line will alter the variables we currently have -- will override the calling environment See here for docs about.
In that way, since variables are overriden, activate_this gives us some variables like:
old_os_path = os.environ['PATH']
#the previous PATH
prev_sys_path = list(sys.path)
#the old Python Path
if sys.platform == 'win32':
site_packages = os.path.join(base, 'Lib', 'site-packages')
else:
site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
#site_packages is the extended venv's packages dir.
#sys.path is affected here
sys.real_prefix = sys.prefix
#the old system prefix
So we can restore such variables if we want a manual deactivation:
import sys, os, site
sys.path[:0] = prev_sys_path #will also revert the added site-packages
sys.prefix = sys.real_prefix
os.setenv('PATH', old_os_path)
My Crontab -l
# m h dom mon dow command
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
00 8,20 * * * python /home/tomi/amaer/controller.py >>/tmp/out.txt 2>&1
My controller.py has config file settings.cfg also it uses other script in the folder it's located (I chmoded only controller.py)
The error
1;31mIOError^[[0m: [Errno 2] No such file or directory: 'settings.cfg'
I have no idea how to fix this? Please help me?
Edit: The part that read the config file
def main():
config=ConfigParser.ConfigParser()
config.readfp(open("settings.cfg"),"r")
As I initially wrote in my comment, this is because you are using relative path to the current working directory. However, that is not going to be the same when running all this via the cron executable rather than the python interpreter directly via the shebang.
Your current code would look for the "settings.cfg" in the current working directory which is where the cron executable resides, and not your script. Hence, you would need to change your code logic to using absolute paths by the help of the "os" built-in standard module.
Try to following line:
import os
...
def main():
config = ConfigParser.ConfigParser()
scriptDirectory = os.path.dirname(os.path.realpath(__file__))
settingsFilePath = os.path.join(scriptDirectory, "settings.cfg")
config.readfp(open(settingsFilePath,"r"))
This will get your the path of your script and then appends the "settings.cfg" with the appropriate dir separator for your operating system which is Linux in this particular case.
If the location of the config file changes any time in the future, you could use the argparse module for processing a command line argument to handle the config location properly, or even without it simply just using the first argument after the script name like sys.argv[1].
Your code is looking for settings.cfg in its current working directory.
This working directory will not be the same when cron executes the job, hence the error
You have two "easy" solutions:
Use an absolute path to the config file in your script (/home/tomi/amaer/config.cfg)
CD to the appropriate directory first in your crontab (cd /home/tomi/amaer/ && python /home/tomi/amaer/controller.py)
The "right" solution, though, would be to pass your script a parameter (or environment variable) that tells it where to look for the config file.
It's not exactly good practice to assume your config file will always be lying just next to your script.
You might want to have alook at this question: https://unix.stackexchange.com/questions/38951/what-is-the-working-directory-when-cron-executes-a-job
I am distributing a package that has this structure:
mymodule:
mymodule/__init__.py
mymodule/code.py
scripts/script1.py
scripts/script2.py
The mymodule subdir of mymodule contains code, and the scripts subdir contains scripts that should be executable by the user.
When describing a package installation in setup.py, I use:
scripts=['myscripts/script1.py']
To specify where scripts should go. During installation they typically go in some platform/user specific bin directory. The code that I have in mymodule/mymodule needs to make calls to the scripts though. What is the correct way to then find the full path to these scripts? Ideally they should be on the user's path at that point, so if I want to call them out from the shell, I should be able to do:
os.system('script1.py args')
But I want to call the script by its absolute path, and not rely on the platform specific bin directory being on the PATH, as in:
# get the directory where the scripts reside in current installation
scripts_dir = get_scripts_dir()
script1_path = os.path.join(scripts_dir, "script1.py")
os.system("%s args" %(script1_path))
How can this be done? thanks.
EDIT removing the code outside of a script is not a practical solution for me. the reason is that I distribute jobs to a cluster system and the way I usually do it is like this: imagine you have a set of tasks you want to run on. I have a script that takes all tasks as input and then calls another script, which runs only on the given task. Something like:
main.py:
for task in tasks:
cmd = "python script.py %s" %(task)
execute_on_system(cmd)
so main.py needs to know where script.py is, because it needs to be a command executable by execute_on_system.
I think you should structure your code so that you don't need to call scripts from you code. Move code you need from scripts to your package and then you can call this code both from your scripts and from your code.
My use case for this was to check that the directory my scripts are installed into is in the user's path and give a warning if not (since it is often not in the path if installing with --user). Here is the solution I came up with:
from setuptools.command.easy_install import easy_install
class my_easy_install( easy_install ):
# Match the call signature of the easy_install version.
def write_script(self, script_name, contents, mode="t", *ignored):
# Run the normal version
easy_install.write_script(self, script_name, contents, mode, *ignored)
# Save the script install directory in the distribution object.
# This is the same thing that is returned by the setup function.
self.distribution.script_install_dir = self.script_dir
...
dist = setup(...,
cmdclass = {'build_ext': my_builder, # I also have one of these.
'easy_install': my_easy_install,
},
)
if dist.script_install_dir not in os.environ['PATH'].split(':'):
# Give a sensible warning message...
I should point out that this is for setuptools. If you use distutils, then the solution is similar, but slightly different:
from distutils.command.install_scripts import install_scripts
class my_install_scripts( install_scripts ): # For distutils
def run(self):
install_scripts.run(self)
self.distribution.script_install_dir = self.install_dir
dist = setup(...,
cmdclass = {'build_ext': my_builder,
'install_scripts': my_install_scripts,
},
)
I think the correct solution is
scripts=glob("myscripts/*.py"),
For example, I have a Python script using the Google App Engine SDK:
from google.appengine.ext import db
from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
from google.appengine.ext.webapp.util import run_wsgi_app
The module db has a submodule Key, so I try to use autocomplete on it:
db.KTab
But at the bottom of the Vim window, I get the following:
-- Omni completion (^O^N^P) Pattern not found
How do I include the path to non-standard Python libraries so that Vim autocompletion can find them? And also display their docstrings?
You need to add your library files to your tags file. For instance, if you have installed the Google App Engine via pip in a virtual environment located in env/:
virtualenv --no-site-package env/
source env/bin/activate
pip install google_appengine
... then you should execute:
ctags -R --python-kinds=-i -o tags env/
If you did not install google_appengine through pip, then you should locate the path to your python libraries (hint: it should be indicated by $PYTHONPATH. And according to this reference page: "on Unix, this is usually .:/usr/local/lib/python.") and replace env/ by the path you found.
Finally, your .vimrc file should parse your tags file. For instance, in my .vimrc, I have:
set tags+=/path/to/my/tags
I grabbed this from natw's vimrc (I think...maybe sontek), but it should do the trick, so long as your packages are findable by your current install of Python. This lets you use gf, but also sets up searching these files for autocompletion. Note the py <<EOF part, which starts a section interpreted in Python. This means you'd have to have the python interpreter installed in vim to use it.
function! LoadPythonPath()
py <<EOF
# load PYTHONPATH into vim, this lets you hover over a module name
# and type 'gf' (for goto file) and open that file in vim. Useful
# and easier than rope for simple tasks
import os.path
import sys
import vim
for p in sys.path:
if os.path.isdir(p):
vim.command(r"set path+=%s" % (p.replace(" ", r"\ ")))
EOF
endfunction
Btw, I don't like to have this load automatically, so I set it to a function that intelligently loads/unloads when I call it/first enter a Python doc. And I add a let g:PythonPathLoaded=1 to the previous function.
function! GetPythonPath()
if !exists("g:PythonPathLoaded")
call LoadPythonPath()
return
elseif g:PythonPathLoaded
return
else
call LoadPythonPath()
endif
endfunction
And I have an unload function too...though I'm not sure whether this makes a huge difference.
function! UnloadPythonPath()
py <<EOF
# load PYTHONPATH into vim, this lets you hover over a module name
# and type 'gf' (for goto file) and open that file in vim. Useful
# and easier than rope for simple tasks
for p in sys.path:
if os.path.isdir(p):
vim.command(r"set path-=%s" % (p.replace(" ", r"\ ")))
EOF
let g:PythonPathLoaded = 0
endfunction
Hope this helps! Plus, an added bonus is that this will load your packages regardless of whether you are using virtualenv (since it, I believe, runs whatever is set as 'python' at the moment).