A script is developed both on OS X and Windows using a virtualenv. The so-called developer has already installed all required packages using a requirements.txt file, but one problem remains:
If the script is running on OS X, the beginning of each Python file must start like this:
#!/Users/os-x-username/.virtualenvs/some_env/bin/python
#!C:\Users\windows-username\Envs\some_env\Scripts\python.exe
But if developing on Windows, the line order must be switched:
#!C:\Users\windows-username\Envs\some_env\Scripts\python.exe
#!/Users/os-x-username/.virtualenvs/some_env/bin/python
How can the so-called developer avoid this tediousness?
If you don't mind adding extra steps, ou can create a launcher script launcher.py like:
#!/usr/bin/env python
import subprocess
import sys
if __name__ != "__main__":
print("This is a launcher. Please run only as a main script.")
exit(-1)
INTERPRETERS = {
"win": r"C:\Users\windows-username\Envs\some_env\Scripts\python.exe", # Windows
"darwin": "/Users/os-x-username/.virtualenvs/some_env/bin/python", # OSX
"linux": "/home/linux-user/.virtualenvs/some_env/bin/python" # Linux
# etc.
}
TARGET_SCRIPT = "original_script_name.py"
interpreter = None
for i in INTERPRETERS: # let's find a suitable interpreter for the current platform
if sys.platform.startswith(i):
interpreter = i
break
if not interpreter:
print("No suitable interpreter found for platform:", sys.platform)
exit(-1)
main_proc = subprocess.Popen([interpreter, TARGET_SCRIPT] + sys.argv[1:]) # call virtualenv
main_proc.communicate() # wait for it to finish
exit(main_proc.poll()) # redirect the return code
Since this script is there only to run the original_script_name.py in the desired interpreter for the current platform, it doesn't matter what its shebang is - as long as it picks any Python interpreter it will be fine.
It would act as a drop-in replacement for your original script (original_script_name.py) so just call launcher.py instead and it will even redirect the CLI arguments if needed.
Related
I am trying to create a python setup script that will be converted to an executable file in windows using pyinstaller. Currently, I am trying to figure out how this script would check if python is installed on the system, if not, then it will go ahead & download the installation exe file from the python website.
My current code is as follows:
import subprocess, requests, platform
architecture = platform.uname()[4]
if architecture == 'AMD64':
executable_file = "https://www.python.org/ftp/python/3.11.1/python-3.11.1-amd64.exe"
elif architecture == 'ARM64':
executable_file = "https://www.python.org/ftp/python/3.11.1/python-3.11.1-arm64.exe"
r = requests.get(executable_file)
with open("python-3.11.1-amd64.exe",'wb') as f:
f.write(r.content)
cmd = "python-3.11.1-amd64.exe"
returned_value = subprocess.call(cmd, shell=True)
if returned_value == 1602:
print("Installation Cancelled. Exitting.")
quit()
print('returned value:', returned_value)
I currently have no idea how to check if python is installed so the exe will not be downloaded & ran by the script. So I would like some ideas on how I could go with this.
Directly execute the python command to see the output, or use the winreg module to obtain the installed applications of the local machine
When you run python manage.py shell, (or shell_plus if you have the extension), the script will import some stuff for you and then open a regular python shell for you to try commands out, as opposed to running the script and immediately closing.
I know that you can use the -i switch when calling a script to keep the command prompt open, for example python -i foo.py will run the contents of foo.py yet keep the shell open instead of immediately closing.
I would guess that when you call django's manage.py shell it somehow uses the same feature to get the command prompt to stay open and not immediately close out.
How does it work? Can I add something to any script file to make it behave the same way (without using the -i switch) ?
As of 2.2.3, the shell management command has three programs it can start -- ipython, bpython, and python. For the first two, you can start the shells using python:
def ipython(self, options):
from IPython import start_ipython
start_ipython(argv=[])
def bpython(self, options):
import bpython
bpython.embed()
For python, django sets up readline and then uses code.interact to start the python interpreter:
def python(self, options):
import code
# Set up a dictionary to serve as the environment for the shell, so
# that tab completion works on objects that are imported at runtime.
imported_objects = {}
try: # Try activating rlcompleter, because it's handy.
import readline
except ImportError:
pass
else:
# We don't have to wrap the following import in a 'try', because
# we already know 'readline' was imported successfully.
import rlcompleter
readline.set_completer(rlcompleter.Completer(imported_objects).complete)
# Enable tab completion on systems using libedit (e.g. macOS).
# These lines are copied from Python's Lib/site.py.
readline_doc = getattr(readline, '__doc__', '')
if readline_doc is not None and 'libedit' in readline_doc:
readline.parse_and_bind("bind ^I rl_complete")
else:
readline.parse_and_bind("tab:complete")
# We want to honor both $PYTHONSTARTUP and .pythonrc.py, so follow system
# conventions and get $PYTHONSTARTUP first then .pythonrc.py.
if not options['no_startup']:
for pythonrc in OrderedSet([os.environ.get("PYTHONSTARTUP"), os.path.expanduser('~/.pythonrc.py')]):
if not pythonrc:
continue
if not os.path.isfile(pythonrc):
continue
with open(pythonrc) as handle:
pythonrc_code = handle.read()
# Match the behavior of the cpython shell where an error in
# PYTHONSTARTUP prints an exception and continues.
try:
exec(compile(pythonrc_code, pythonrc, 'exec'), imported_objects)
except Exception:
traceback.print_exc()
code.interact(local=imported_objects)
I'm running on Windows Server 2012 R2 standard machine with 32Bit python.
I'm trying to run a powershell script for a 64Bit system which check if a key exists in the registry:
$path = "Registry::HKLM\Software\<folder>"
if (!(Test-Path $path))
{
write "$path does not exist!"
}
When running through powershell, script works perfect.
When I'm running it from python, it doesn't find the key:
from gevent.subprocess import call
call("powershell <script.ps1>, shell=True")
After some research, I found that the 32bit python process is calling the 32bit version of powershell.
I verified it with this simple script that checks if the process running is 32bit or 64bit:
powershell [System.Environment]::Is64BitProcess
Will return True for 64Bit process and False for a 32Bit process.
Manually checking this command works:
C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe [System.Environment]::Is64BitProcess
Returns False as this is the 32Bit version of powershell (yes, quite confusing due to the WOW64 folder).
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe [System.Environment]::Is64BitProcess
Returns True.
But running:
from gevent.subprocess import call
call("C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe [System.Environment]::Is64BitProcess", shell=True)
Returns False
What am I missing?
What you're missing is that Windows transparently redirects file system calls from 32-bit processes to C:\Windows\System32 to C:\Windows\SysWOW64 :)
There a simple solution to this - if a 32-bit process tries to fetch file system resources from C:\Windows\sysnative instead, Windows will not redirect to SysWOW64.
From PowerShell it's fairly easy to figure out whether your process will be affected by all of this SysWOW64 redirection, basically:
$IsSysWOW64Process = [Environment]::Is64BitOperatingSystem -and -not [Environment]::Is64BitProcess
In python however, I haven't been able to find a surefire way of doing this, so I suspect your best bet is to call kernel32!IsWow64Process2() via ctypes:
from ctypes import windll,c_ushort,byref
import platform
def is_syswow64_process():
if platform.architecture()[0] != '64bit':
# 32-bit OS, no syswow64 handling
return False
# Ok, 64-bit OS, let's see if the process is 32-bit
# Obtain process handle to self
this_process = windll.kernel32.GetCurrentProcess()
# Declare ref arguments
proc_type, platform_type = c_ushort(), c_ushort()
# Query Windows for the information
wow64_call = windll.kernel32.IsWow64Process2(this_process, byref(proc_type), byref(platform_type))
if wow64_call == 0:
# you'd probably want to call kernel32!GetLastError here
raise Exception("Problem querying kernel32!IsWow64Process2")
return proc_type.value == 0
And now you can conditionally replace the path when needed:
powershell = "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe"
if is_syswow64_process():
powershell = re.sub("(?i)syswow64|system32", "sysnative", powershell)
call("%s .\path\to\script.ps1" % powershell)
I am trying to load automatically my Python virtual environment and execute a python query at the same time in Powershell but I am not getting any luck at it. This is the code I have:
# my_testing_file.py
# Activate the virtual environment
import os
script_directory = os.path.dirname(os.path.abspath(__file__))
activate_this_file = os.path.join(script_directory, 'C:\Temp\Development\Python_scripts\env\Scripts\activate_this.py')
# Start executing the main code
import pypyodbc
from openpyxl.workbook import Workbook
def main():
cursor = initiate_connection_db()
unposted_hours_results = retrieve_results_query(cursor)
...
if __name__ == "__main__":
main()
All the code is in 1 single file and basically I want to do in Powershell python my_testing_file.py so it loads the virtual environment and executes the rest of the code.
When this code gets executed from Powershell, a command prompt appears for a few seconds and then it shuts down and the rest of the code never gets executed. Any help for fixing this code would be greatly appreciated.
As #dfundako indicated, a post-activate script could have done the trick to activate the virtual environment and execute the script all at once, however, I found another alternative which I believe it is not the best but it performed what I needed to achieve. I had to pack the python script and its libraries in an .exe file so once a user clicks on it, the program executes the desired task.
Inside of my virtual environment, I executed this file called setup.py to build my_testing_file.py to an .exe:
# Python 3.6.7
# The cx_Freeze library did the trick to pack all the libraries and the script.
# Put down all the scripts that your
import sys, os
import openpyxl
import cx_Freeze
import my_testing_file.py
from subprocess import call
base = None
if sys.platform == 'win32':
base = "Win32GUI"
# my_testing_file is the main script that executes my desired task.
executables = [cx_Freeze.Executable("my_testing_file.py", base=base)]
cx_Freeze.setup(
name = "Testing Deploy",
options = {"build_exe":
{"packages":
["openpyxl", "subprocess"]
}
},
version = "0.01",
description = " Testing executable alternative",
executables = executables
)
I believe something similar could have been done with Docker as well but I lack the required knowledge to achieve the same goal. Feel free to improve this answer or put the code for the Docker alternative or the aforementioned post-activate script.
Update 2021: Solution is built into PyDev/Eclipse
See accepted answer for details
Original Question (and old answers) for below
As many comments/questions/rants on SO and other places will tell you, Python3 packages using relative imports want to be run from a central __main__.py file. In the case where a module, say "modA" within a package, say "packA", that uses relative imports needs to be run (for instance because a test package is run if __name__ == '__main__'), we are told to run instead run python3 -m modA.packA from the directory above modA if sys.path() does not contain the directory above modA. I may dislike this paradigm, but I can work around it.
When trying to run modA from Eclipse/PyDev, however, I can't figure out how to specify a run configuration that will execute the module properly with the -m flag. Has anyone figured out how to set up a run configuration that will do this properly?
References: Relative imports for the billionth time ; Relative import in Python 3 is not working ; Multilevel relative import
Nowadays (since PyDev 5.4.0 (2016-11-28)) you can go to the Settings > PyDev > Run and select Launch modules with python -m mod.name instead of python filename.py ;)
See: https://www.pydev.org/history_pydev.html
For older versions of PyDev (old answer)
Unfortunately, right now, it's not automatic running with -m in PyDev, so, I'll present 3 choices to work in PyDev with relative imports which are preceded by a dot (in PyDev version 4.3.0):
Don't use relative imports, only absolute imports in your __main__ modules.
Create a separate module for the __main__ which will do an absolute import for the module you want to run and run that module instead (if you're distributing your application, this is probably needed anyways as the usual way for people to launch your code in Python is by passing the script as an argument to Python and not using the -m switch).
Add the -m module to the vm arguments in your run configuration by doing:
Make the run (which will fail because of the relative import)
Right-click the editor > Copy Context Qualified Name
Open the run configuration: Alt, R, N (i.e.: Toolbar > Run > Run Configuration)
Open arguments tab and add the '-m Ctrl+V' (to add the -m and the module name you copied previously).
Although this is definitely not ideal: you'll now receive an argument with the filename (as PyDev will always pass that to run the file) and the whole process is a nuisance.
As a note, I do hope to provide a way to make runs within PyDev using the -m soon (hopefully for PyDev 4.4.0)... although this may not be possible if the file being run is not under the PYTHONPATH (i.e.: to run an external file it still has to support the option without the -m).
There's a bit nasty trick possible here to work around this issue. I'm using PyDev 9.2.0
Put your venv right in the workspace, say under the dir "venv".
Refresh your eclipse workspace and ensure that it uses this venv (through your interpreter setup).
After the refresh, go to the run configuration and edit the "Main Module" by clicking the Browse button.
The venv will now appear.
Browse into the venv/lib/python3.8/site-packages
There you will find the pip-installed module source codes and you can select the module you want to run.
Update 2021: This answer is no longer needed. See accepted answer for details.
Here's what I was able to do after Fabio's great suggestion.
Create a program called /usr/local/bin/runPy3M with world read/execute permissions, with the following code:
#!/usr/local/bin/python3 -u
'''
Run submodules inside packages (with relative imports) given
a base path and a path (relative or absolute) to the submodule
inside the package.
Either specify the package root with -b, or setenv ECLIPSE_PROJECT_LOC.
'''
import argparse
import os
import re
import subprocess
import sys
def baseAndFileToModule(basePath, pyFile):
'''
Takes a base path referring to the root of a package
and a (relative or absolute) path to a python submodule
and returns a string of a module name to be called with
python -m MODULE, if the current working directory is
changed to basePath.
Here the CWD is '/Users/cuthbert/git/t/server/tornadoHandlers/'.
>>> baseAndFileToModule('/Users/cuthbert/git/t/', 'bankHandler.py')
'server.tornadoHandlers.bankHandler'
'''
absPyFilePath = os.path.abspath(pyFile)
absBasePath = None
if basePath is not None:
absBasePath = os.path.abspath(basePath)
commonPrefix = os.path.commonprefix([absBasePath, absPyFilePath])
uncommonPyFile = absPyFilePath[len(commonPrefix):]
else:
commonPrefix = ""
uncommonPyFile = absPyFilePath
if commonPrefix not in sys.path:
sys.path.append(commonPrefix)
moduleize = uncommonPyFile.replace(os.path.sep, ".")
moduleize = re.sub("\.py.?$", "", moduleize)
moduleize = re.sub("^\.", "", moduleize)
return moduleize
def exitIfPyDevSetup():
'''
If PyDev is trying to see if this program is a valid
Python Interpreter, it expects to function just like Python.
This is a little module that does this if the last argument
is 'interpreterInfo.py' and then exits.
'''
if 'interpreterInfo.py' in sys.argv[-1]:
interarg = " ".join([sys.executable] + sys.argv[1:])
subprocess.call(interarg.split())
exit()
return
exitIfPyDevSetup()
parser = argparse.ArgumentParser(description='Run a python file or files as a module.')
parser.add_argument('file', metavar='pyfile', type=str, nargs=1,
help='filename to run, with .py')
parser.add_argument('-b', '--basepath', type=str, default=None, metavar='path',
help='path to directory to consider the root above the package')
parser.add_argument('-u', action='store_true', help='unbuffered binary stdout and stderr (assumed)')
args = parser.parse_args()
pyFile = args.file[0]
basePath = args.basepath
if basePath is None and 'ECLIPSE_PROJECT_LOC' in os.environ:
basePath = os.environ['ECLIPSE_PROJECT_LOC']
modName = baseAndFileToModule(basePath, pyFile)
allPath = ""
if basePath:
allPath += "cd '" + basePath + "'; "
allPath += sys.executable
allPath += " -m " + modName
subprocess.call(allPath, shell=True) # maybe possible with runpy instead...
Then add a new interpreter to PyDev -- call it what you'd like (e.g., runPy3M) and point the interpreter to /usr/local/bin/runPy3M. Click okay. Then move it up the list if need be. Then add to environment for the interpreter, Variable: ECLIPSE_PROJECT_LOC (name here DOES matter) with value ${project_loc}.
Now submodules inside packages that choose this interpreter will run as modules relative to the sub package.
I'd like to see baseAndFileToModule(basePath, pyFile) added to runpy eventually as another run option, but this will work for now.
EDIT: Unfortunately, after setting all this up, it appears that this configuration seems to prevent Eclipse/PyDev from recognizing builtins such as "None", "True", "False", "isinstnance," etc. So not a perfect solution.