I'm quite new to python and just created my first opensource python project - a daemon to create wifi hotspots on linux.
I've used distutils to build the package. To start the daemon once installation is done, I've registered the below script that simply starts the daemon by calling the corresponding python module:
#!/bin/bash
python -m hotspotd $*
And this is the setup.py that registers it:
#INSTALL IT
from distutils.core import setup
s = setup(name='hotspotd',
version='0.1',
description='Small daemon to create a wifi hotspot on linux',
license='MIT',
author='Prahlad Yeri',
author_email='prahladyeri#yahoo.com',
url='https://github.com/prahladyeri/hotspotd',
#py_modules=['hotspotd','cli'],
packages=['hotspotd'],
package_dir={'hotspotd': ''},
package_data={'hotspotd': ['run.dat']},
scripts=['hotspotd']
#data_files=[('config',['run.dat'])],
)
Now, this works fine on my machine and some other machines I've tested. However, as indicated by the open issue on the github, some users are unable to run that script. It gives the error:
No module named hotspotd.main; 'hotspotd' is a package and cannot be directly executed
Apparently it expects the entire package.module syntax which is hotspotd.hotspotd on their setups. However, on my machine the full syntax doesn't work, and only hotspotd works. Whats going on here?
I had to change my script, so that instead of directly passing the module as argument, I import this module in the script itself and call the hotspotd.main function from there:
#!/usr/bin/env python
##author: Prahlad Yeri
##description: Small daemon to create a wifi hotspot on linux
##license: MIT
#python -m hotspotd $*
import hotspotd.main
import sys
import os
import argparse
if __name__ == "__main__":
#check root or not
if os.getenv('USER') != 'root':
print "You need root permissions to do this, sloth!"
sys.exit(1)
parser = argparse.ArgumentParser(description='A small daemon to create a wifi hotspot on linux')
parser.add_argument('-v', '--verbose', required=False, action='store_true')
parser.add_argument('command', choices=['start', 'stop', 'configure'])
args = parser.parse_args()
hotspotd.main.main(args)
Related
I'm trying to run a Python setup script from within a python script. That is, instead of:
python path_to/setup.py build --build-base="build_dir" install --prefix="install_dir"
which works fine. I want to be able to call
from distutils.core import run_setup
args = ['build', '--build-base="build_dir"', 'install', '--prefix="install_dir"']
run_setup("path_to/setup.py", script_args=args, stop_after='run')
from within another Python script (as automated/on the fly installation). This does not seem to work at the moment as the call returns the error
RuntimeError: 'distutils.core.setup()' was never called
-- perhaps 'setup.py' is not a Distutils setup script?
which is strange since calling it from outside the script works. Is there something wrong with this approach? The setup.py script looks normal:
from distutils.core import setup
from distutils.command.build_py import build_py
# ...
if __name__ == '__main__':
setup( ... )
Alternatively, I'm thinking about just calling "subprocess" from the script, but then want to ensure I'm using the same Python instance/runtime as the running Python instance (in case many python versions are installed). How could I do this, ensure to call the top command from a subprocess with the same python?
Update 2021: Solution is built into PyDev/Eclipse
See accepted answer for details
Original Question (and old answers) for below
As many comments/questions/rants on SO and other places will tell you, Python3 packages using relative imports want to be run from a central __main__.py file. In the case where a module, say "modA" within a package, say "packA", that uses relative imports needs to be run (for instance because a test package is run if __name__ == '__main__'), we are told to run instead run python3 -m modA.packA from the directory above modA if sys.path() does not contain the directory above modA. I may dislike this paradigm, but I can work around it.
When trying to run modA from Eclipse/PyDev, however, I can't figure out how to specify a run configuration that will execute the module properly with the -m flag. Has anyone figured out how to set up a run configuration that will do this properly?
References: Relative imports for the billionth time ; Relative import in Python 3 is not working ; Multilevel relative import
Nowadays (since PyDev 5.4.0 (2016-11-28)) you can go to the Settings > PyDev > Run and select Launch modules with python -m mod.name instead of python filename.py ;)
See: https://www.pydev.org/history_pydev.html
For older versions of PyDev (old answer)
Unfortunately, right now, it's not automatic running with -m in PyDev, so, I'll present 3 choices to work in PyDev with relative imports which are preceded by a dot (in PyDev version 4.3.0):
Don't use relative imports, only absolute imports in your __main__ modules.
Create a separate module for the __main__ which will do an absolute import for the module you want to run and run that module instead (if you're distributing your application, this is probably needed anyways as the usual way for people to launch your code in Python is by passing the script as an argument to Python and not using the -m switch).
Add the -m module to the vm arguments in your run configuration by doing:
Make the run (which will fail because of the relative import)
Right-click the editor > Copy Context Qualified Name
Open the run configuration: Alt, R, N (i.e.: Toolbar > Run > Run Configuration)
Open arguments tab and add the '-m Ctrl+V' (to add the -m and the module name you copied previously).
Although this is definitely not ideal: you'll now receive an argument with the filename (as PyDev will always pass that to run the file) and the whole process is a nuisance.
As a note, I do hope to provide a way to make runs within PyDev using the -m soon (hopefully for PyDev 4.4.0)... although this may not be possible if the file being run is not under the PYTHONPATH (i.e.: to run an external file it still has to support the option without the -m).
There's a bit nasty trick possible here to work around this issue. I'm using PyDev 9.2.0
Put your venv right in the workspace, say under the dir "venv".
Refresh your eclipse workspace and ensure that it uses this venv (through your interpreter setup).
After the refresh, go to the run configuration and edit the "Main Module" by clicking the Browse button.
The venv will now appear.
Browse into the venv/lib/python3.8/site-packages
There you will find the pip-installed module source codes and you can select the module you want to run.
Update 2021: This answer is no longer needed. See accepted answer for details.
Here's what I was able to do after Fabio's great suggestion.
Create a program called /usr/local/bin/runPy3M with world read/execute permissions, with the following code:
#!/usr/local/bin/python3 -u
'''
Run submodules inside packages (with relative imports) given
a base path and a path (relative or absolute) to the submodule
inside the package.
Either specify the package root with -b, or setenv ECLIPSE_PROJECT_LOC.
'''
import argparse
import os
import re
import subprocess
import sys
def baseAndFileToModule(basePath, pyFile):
'''
Takes a base path referring to the root of a package
and a (relative or absolute) path to a python submodule
and returns a string of a module name to be called with
python -m MODULE, if the current working directory is
changed to basePath.
Here the CWD is '/Users/cuthbert/git/t/server/tornadoHandlers/'.
>>> baseAndFileToModule('/Users/cuthbert/git/t/', 'bankHandler.py')
'server.tornadoHandlers.bankHandler'
'''
absPyFilePath = os.path.abspath(pyFile)
absBasePath = None
if basePath is not None:
absBasePath = os.path.abspath(basePath)
commonPrefix = os.path.commonprefix([absBasePath, absPyFilePath])
uncommonPyFile = absPyFilePath[len(commonPrefix):]
else:
commonPrefix = ""
uncommonPyFile = absPyFilePath
if commonPrefix not in sys.path:
sys.path.append(commonPrefix)
moduleize = uncommonPyFile.replace(os.path.sep, ".")
moduleize = re.sub("\.py.?$", "", moduleize)
moduleize = re.sub("^\.", "", moduleize)
return moduleize
def exitIfPyDevSetup():
'''
If PyDev is trying to see if this program is a valid
Python Interpreter, it expects to function just like Python.
This is a little module that does this if the last argument
is 'interpreterInfo.py' and then exits.
'''
if 'interpreterInfo.py' in sys.argv[-1]:
interarg = " ".join([sys.executable] + sys.argv[1:])
subprocess.call(interarg.split())
exit()
return
exitIfPyDevSetup()
parser = argparse.ArgumentParser(description='Run a python file or files as a module.')
parser.add_argument('file', metavar='pyfile', type=str, nargs=1,
help='filename to run, with .py')
parser.add_argument('-b', '--basepath', type=str, default=None, metavar='path',
help='path to directory to consider the root above the package')
parser.add_argument('-u', action='store_true', help='unbuffered binary stdout and stderr (assumed)')
args = parser.parse_args()
pyFile = args.file[0]
basePath = args.basepath
if basePath is None and 'ECLIPSE_PROJECT_LOC' in os.environ:
basePath = os.environ['ECLIPSE_PROJECT_LOC']
modName = baseAndFileToModule(basePath, pyFile)
allPath = ""
if basePath:
allPath += "cd '" + basePath + "'; "
allPath += sys.executable
allPath += " -m " + modName
subprocess.call(allPath, shell=True) # maybe possible with runpy instead...
Then add a new interpreter to PyDev -- call it what you'd like (e.g., runPy3M) and point the interpreter to /usr/local/bin/runPy3M. Click okay. Then move it up the list if need be. Then add to environment for the interpreter, Variable: ECLIPSE_PROJECT_LOC (name here DOES matter) with value ${project_loc}.
Now submodules inside packages that choose this interpreter will run as modules relative to the sub package.
I'd like to see baseAndFileToModule(basePath, pyFile) added to runpy eventually as another run option, but this will work for now.
EDIT: Unfortunately, after setting all this up, it appears that this configuration seems to prevent Eclipse/PyDev from recognizing builtins such as "None", "True", "False", "isinstnance," etc. So not a perfect solution.
I use PyCharm/IntelliJ community editions from a wile to write and debug Python scripts, but now I'm trying to debug a Python module, and PyCharm does a wrong command line instruction parsing, causing an execution error, or maybe I'm making a bad configuration.
This is my run/debug configuration:
And this is executed when I run the module (no problems here):
/usr/bin/python3.4 -m histraw
But when I debug, this is the output in the IntelliJ console:
/usr/bin/python3.4 -m /opt/apps/pycharm/helpers/pydev/pydevd.py --multiproc --client 127.0.0.1 --port 57851 --file histraw
/usr/bin/python3.4: Error while finding spec for '/opt/apps/pycharm/helpers/pydev/pydevd.py' (<class 'ImportError'>: No module named '/opt/apps/pycharm/helpers/pydev/pydevd')
Process finished with exit code 1
As you can see, the parameters are wrong parsed, and after -m option a IntelliJ debug script is passed before the module name.
I also tried just put -m histraw in the Script field, but doesn't work, that field is only to put Python script paths, not modules.
Any ideas?
There is another way to make it work.You can write a python script to run your module.Then just configure PyCharm to run this script.
import sys
import os
import runpy
path = os.path.dirname(sys.modules[__name__].__file__)
path = os.path.join(path, '..')
sys.path.insert(0, path)
runpy.run_module('<your module name>', run_name="__main__",alter_sys=True)
Then the debugger works.
In PyCharm 2019.1 (professional), I'm able to select run as module option under configurations, as below
I found it easiest to create a bootstrap file (debuglaunch.py) with the following contents.
from {package} import {file with __main__}
if __name__ == '__main__':
{file with __main__}.main()
For example, to launch locustio in the pycharm debugger, I created debuglaunch.py like this:
from locust import main
if __name__ == '__main__':
main.main()
And configured pycharm as follows.
NOTE: I found I was not able to break into the debugger unless I added a breakpoint on main.main() . That may be specific to locustio, however.
The problem is already fixed since PyCharm 4.5.2. See corresponding issue in PyCharm tracker:
https://youtrack.jetbrains.com/issue/PY-15230
I want to force a script to be run with python -S, I'm defining the script using entry_points in the setup.py. Is there an option for this?
Thanks!
I don't think there is such option in setuptools. You could create a stub script and specify it in the scripts distutils option instead. Bases on Is it possible to set the python -O (optimize) flag within a script?:
#!/usr/bin/env python
from your_package.script import main
if __name__=="__main__":
import os, sys
sentinel_option = '--dont-add-no-site-option'
if sentinel_option not in sys.argv:
sys.argv.append(sentinel_option)
os.execl(sys.executable, sys.executable, '-S', *sys.argv)
else:
sys.argv.remove(sentinel_option)
main()
I created a nice python Twisted app with a plugin for the twistd runner, as specified in the Twisted Documentation: http://twistedmatrix.com/documents/current/core/howto/tap.html. I am having problems packaging this with PyInstaller: my twistd plugin is not found during execution of the frozen application.
To ship my project, I created my own top-level startup script using the twistd runner modules, e.g.
#!/usr/bin/env python
from twisted.scripts.twistd import run
from sys import argv
argv[1:] = [
'--pidfile', '/var/run/myapp.pid',
'--logfile', '/var/run/myapp.log',
'myapp_plugin'
]
run()
Next, I use PyInstaller to freeze this as a single directory deployment. Executing the frozen script above fails as it cannot find my twistd plugin (edited for brevity):
~/pyinstall/dist/bin/mystartup?16632/twisted/python/modules.py:758:
UserWarning: ~/pyinstall/dist/mystartup?16632 (for module twisted.plugins)
not in path importer cache (PEP 302 violation - check your local configuration).
~/pyinstall/dist/bin/mystartup: Unknown command: myapp_plugin
Normally, Twistd inspects the Python system path to discover my plugin in twisted/plugins/myapp_plugin.py. If I print the list of twistd plugins in my startup script, the list is empty in the executable resulting from PyInstaller, e.g.
from twisted.plugin import IPlugin, getPlugins
plugins = list(getPlugins(IPlugin))
print "Twistd plugins=%s" % plugins
I use a somewhat default PyInstaller spec file, no hidden imports or import hooks specified.
I like the functionality of twistd with logging, pid files, etc, so I would like to avoid having to abandon the twistd runner altogether to circumvent the plugin issue.
Is there a way to ensure my twistd plugin is found in the frozen executable?
I found a workaround by reverse engineering some of the twisted code. Here I hardcode the plugin import. This works fine with PyInstaller for me.
#!/usr/bin/env python
import sys
from twisted.application import app
from twisted.scripts.twistd import runApp, ServerOptions
import myapp_plugin as myplugin
plug = myplugin.serviceMaker
class MyServerOptions(ServerOptions):
"""
See twisted.application.app.ServerOptions.subCommands().
Override to specify a single plugin subcommand and load the plugin
explictly.
"""
def subCommands(self):
self.loadedPlugins = {plug.tapname:plug}
yield (plug.tapname,
None,
# Avoid resolving the options attribute right away, in case
# it's a property with a non-trivial getter (eg, one which
# imports modules).
lambda plug=plug: plug.options(),
plug.description)
subCommands = property(subCommands)
def run():
"""
Replace twisted.application.app.run()
To use our ServerOptions.
"""
app.run(runApp, MyServerOptions)
sys.argv[1:] = [
'--pidfile', '/var/run/myapp.pid',
'--logfile', '/var/run/myapp.log',
plug.tapname] + sys.argv[1:]
run()