I am trying to load automatically my Python virtual environment and execute a python query at the same time in Powershell but I am not getting any luck at it. This is the code I have:
# my_testing_file.py
# Activate the virtual environment
import os
script_directory = os.path.dirname(os.path.abspath(__file__))
activate_this_file = os.path.join(script_directory, 'C:\Temp\Development\Python_scripts\env\Scripts\activate_this.py')
# Start executing the main code
import pypyodbc
from openpyxl.workbook import Workbook
def main():
cursor = initiate_connection_db()
unposted_hours_results = retrieve_results_query(cursor)
...
if __name__ == "__main__":
main()
All the code is in 1 single file and basically I want to do in Powershell python my_testing_file.py so it loads the virtual environment and executes the rest of the code.
When this code gets executed from Powershell, a command prompt appears for a few seconds and then it shuts down and the rest of the code never gets executed. Any help for fixing this code would be greatly appreciated.
As #dfundako indicated, a post-activate script could have done the trick to activate the virtual environment and execute the script all at once, however, I found another alternative which I believe it is not the best but it performed what I needed to achieve. I had to pack the python script and its libraries in an .exe file so once a user clicks on it, the program executes the desired task.
Inside of my virtual environment, I executed this file called setup.py to build my_testing_file.py to an .exe:
# Python 3.6.7
# The cx_Freeze library did the trick to pack all the libraries and the script.
# Put down all the scripts that your
import sys, os
import openpyxl
import cx_Freeze
import my_testing_file.py
from subprocess import call
base = None
if sys.platform == 'win32':
base = "Win32GUI"
# my_testing_file is the main script that executes my desired task.
executables = [cx_Freeze.Executable("my_testing_file.py", base=base)]
cx_Freeze.setup(
name = "Testing Deploy",
options = {"build_exe":
{"packages":
["openpyxl", "subprocess"]
}
},
version = "0.01",
description = " Testing executable alternative",
executables = executables
)
I believe something similar could have been done with Docker as well but I lack the required knowledge to achieve the same goal. Feel free to improve this answer or put the code for the Docker alternative or the aforementioned post-activate script.
Looking to use a include a redis server for storing application specific data with my pyinstaller bundled application.
Before getting into it hands-on, need some guidance.
Are these the steps to follow for it?
(1) Bundle redis-server executable. And run it as a standalone application via some script in my bundled package.
(2) Use redis client packages in python to connect to the redis-server
I guess (2) should surely work. But is there any easy way of doing (1).
You can bundle arbitrary binaries with the --add-binary option on the command line or the binaries argument to the Analysis call in your .spec file. Check out the manual for details, but one example:
pyinstaller -F main.py --add-binary=`which redis-server`:bin
I don't know of a way to run arbitrary executables, but you could have some python code in your app to detect when you're bundled, find the redis binary, and start it up. Again, you can check out the documentation for details on how to go about this but, again, a example of how this could look (optional contextmanager elegance stolen from another answer):
import sys
import os
import subprocess
from contextlib import contextmanager
#contextmanager
def bundledredis():
proc = subprocess.Popen(
[os.path.join(sys._MEIPASS, 'bin', 'redis-server')])
yield
proc.terminate()
#contextmanager
def optional(condition, context_manager):
if condition:
with context_manager:
yield
else:
yield
def realmain():
print('doing stuff')
def main():
with optional(getattr(sys, 'frozen', False), bundledredis()):
realmain()
if __name__ == '__main__':
main()
I am running the code below by python win_service.py install from the normal command prompt, where I get access denied error.
Installing service TestService
Error installing service: Access is denied. (5)
which I was able to resolve when I started the command prompt by starting as administrator.
I was able to install the service, but I was unable to start the service.
Service installed
Starting service TestService
Error starting service: The service did not respond to the start or control request in a timely fashion.
import win32serviceutil
import win32service
import win32event
import servicemanager
import socket
class AppServerSvc (win32serviceutil.ServiceFramework):
_svc_name_ = "TestService"
_svc_display_name_ = "Test Service"
def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
self.hWaitStop = win32event.CreateEvent(None,0,0,None)
socket.setdefaulttimeout(60)
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
win32event.SetEvent(self.hWaitStop)
def SvcDoRun(self):
servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
servicemanager.PYS_SERVICE_STARTED,
(self._svc_name_,''))
self.main()
def main(self):
print "running"
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(AppServerSvc)
What am doing wrong, is there any other way to install the service that would solve the issue and how to dynamically run it as administrator.
I know this is old but I was stuck on this forever. For me, this specific problem was solved by copying this file - pywintypes36.dll
From -> Python36\Lib\site-packages\pywin32_system32
To -> Python36\Lib\site-packages\win32
It's possible that your service is not starting because it's unable to find the executable. I had a similar issue that was solved by adding some pywin32 related directories to my system path. You can do this using setx:
setx /M PATH "%PATH%;C:\Python27;C:\Python27\Scripts;C:\Python27\Lib\site-packages\pywin32_system32;C:\Python27\Lib\site-packages\win32"
Try running this in a cmd window with admin privileges and adjust the paths to match your own python installation.
Finally, the solution for this.
First step:
USE pyinstaller to create a standalone executable file, i.e.:
pip install pyinstaller
pyinstaller yourproject.py
cd dist\yourproject
yourproject.exe install
Second step:
Note that. When the Windows Service calls "your program", it gives a time to answer according the Service Development Protocol. All of the codes above, are not starting the service. Please, change your code as below:
if __name__ == '__main__':
if len(sys.argv) > 1:
# Called by Windows shell. Handling arguments such as: Install, Remove, etc.
win32serviceutil.HandleCommandLine(JobManager)
else:
# Called by Windows Service. Initialize the service to communicate with the system operator
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(JobManager)
servicemanager.StartServiceCtrlDispatcher()
As noted by Chip (and which I've missed while trying to figure this out), pythonservice.exe is going to run as a system service, so it will have a different environment than you do as a user. Running python service.py debug will run just fine because it's still running with your user environment, but if you run python service.py start, it may now fail instead due to the difference in environment variables. An instant timeout is most likely due to pythonservice.exe failing to execute, and it will fail to execute it it's missing either PythonXX.dll or pywintypesXX.dll.
PythonXX.dll is likely to be in your system path already (depending on how Python was installed), but if you're like me and trying to be extra careful to not alter the environment, that's going to be a problem. I was running something like .\.pyenv37\Scripts\python.exe service.py start, gets Python37.dll from the PATH, not the venv like I assumed, so it's no longer known when pythonservice.exe starts running using a different PATH, which causes it to immediately fail Windows reports it as an instant timeout.
The same goes for pywintypesXX.dll, except instead of installing it somewhere in your search path, the more portable solution is to drop it in the same directory as pythonservice.exe since the deafult DLL search path includes it. It will also cause your service to immediately timeout if it if pythonservice.exe can't find it.
Figuring this out without any sort of logs is an absolute nightmare, let me tell you!
EDIT: Here's what I'm using to verify all of that on script installation/update:
# customOptionHandler will only run after service install/update
if __name__=='__main__':
win32serviceutil.HandleCommandLine(AppServerSvc, customOptionHandler=post_service_update)
.
def post_service_update(*args):
import win32api, win32con, win32profile, pywintypes
from contextlib import closing
env_reg_key = "SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Environment"
hkey = win32api.RegOpenKeyEx(win32con.HKEY_LOCAL_MACHINE, env_reg_key, 0, win32con.KEY_ALL_ACCESS)
with closing(hkey):
system_path = win32api.RegQueryValueEx(hkey, 'PATH')[0]
# PATH may contain %SYSTEM_ROOT% or other env variables that must be expanded
# ExpandEnvironmentStringsForUser(None) only expands System variables
system_path = win32profile.ExpandEnvironmentStringsForUser(None, system_path)
system_path_list = system_path.split(os.pathsep)
core_dll_file = win32api.GetModuleFileName(sys.dllhandle)
core_dll_name = os.path.basename(core_dll_file)
for search_path_dir in system_path_list:
try:
dll_path = win32api.SearchPath(search_path_dir, core_dll_name)[0]
print(f"System python DLL: {dll_path}")
break
except pywintypes.error as ex:
if ex.args[1] != 'SearchPath': raise
continue
else:
print("*** WARNING ***")
print(f"Your current Python DLL ({core_dll_name}) is not in your SYSTEM PATH")
print("The service is likely to not launch correctly.")
from win32serviceutil import LocatePythonServiceExe
pythonservice_exe = LocatePythonServiceExe()
pywintypes_dll_file = pywintypes.__spec__.origin
pythonservice_path = os.path.dirname(pythonservice_exe)
pywintypes_dll_name = os.path.basename(pywintypes_dll_file)
try:
return win32api.SearchPath(pythonservice_path, pywintypes_dll_name)[0]
except pywintypes.error as ex:
if ex.args[1] != 'SearchPath': raise
print("*** WARNING ***")
print(f"{pywintypes_dll_name} is not is the same directory as pythonservice.exe")
print(f'Copy "{pywintypes_dll_file}" to "{pythonservice_path}"')
print("The service is likely to not launch correctly.")
It may seem like a lot, but it will at leastkeep you from forgetting to do those steps when deploying the service on a new machine/virtual environment or when updating python.
Before you start debugging, I recommend to make sure that the two steps described at https://github.com/mhammond/pywin32, which are
pip install pywin32
and
python Scripts/pywin32_postinstall.py -install
were completed.
if you are using anaconda python, be sure python36.dll is in your system path. This took me a long time to find.
credit: Can't start Windows service written in Python (win32serviceutil)
Make sure you run the application with a different user than the default Local System user. Replace it with the user you successfully be able to run the debug command with.
To replace the user go to the windows services (start > services.msc)
Right click on the service you created > properties > Log On
Uncheck the Local System Account and enter your own.
from all python windows service can not start{error 1053} worked for me.
Because I just set PATH for user login, not for System. You can recheck your PATH for System Variables.
I was also facing the same problem but, After messing out for 4 days, Finally I'm able to find the solution for this problem.
So, Here is the step by step Beginner Guide : https://github.com/PushpenderIndia/PythonWindowsService
Also Posting the same solution here as well.
Steps To Create an Python Windows Service
(1) Copy Paste These Codes to a Python File (e.g. server.py)
import servicemanager
import sys
import win32serviceutil
from mainserver import FlaskServer # Import your code, I've written a module called mainserver which contains FlaskServer Code using OOPs
import threading
import concurrent.futures
import time
class workingthread(threading.Thread):
def __init__(self, quitEvent):
self.quitEvent = quitEvent
self.waitTime = 1
threading.Thread.__init__(self)
def run(self):
try:
# Running start_flask() function on different thread, so that it doesn't blocks the code
executor = concurrent.futures.ThreadPoolExecutor(max_workers=5)
executor.submit(self.start_flask)
except:
pass
# Following Lines are written so that, the program doesn't get quit
# Will Run a Endless While Loop till Stop signal is not received from Windows Service API
while not self.quitEvent.isSet(): # If stop signal is triggered, exit
time.sleep(1)
def start_flask(self):
# This Function contains the actual logic, of windows service
# This is case, we are running our flaskserver
test = FlaskServer()
test.start()
class FlaskService(win32serviceutil.ServiceFramework):
_svc_name_ = "AA Testing"
_svc_display_name_ = "AAA Testing"
_svc_description_ = "This is my service"
def __init__(self, args):
win32serviceutil.ServiceFramework.__init__(self, args)
self.hWaitStop = threading.Event()
self.thread = workingthread(self.hWaitStop)
def SvcStop(self):
self.hWaitStop.set()
def SvcDoRun(self):
self.thread.start()
self.hWaitStop.wait()
self.thread.join()
if __name__ == '__main__':
if len(sys.argv) == 1:
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(FlaskService)
servicemanager.StartServiceCtrlDispatcher()
else:
win32serviceutil.HandleCommandLine(FlaskService)
(2) Install the latest pywin32.exe
NOTE: If you install pywin32 using pip, then it will not going to install properly, thus will show you some errors,
Visit https://github.com/mhammond/pywin32/releases
And Download the latest exe, If you are using Python 32bit then download pywin32-302.win32-py3.x.exe
If using Python 64 bit, then download pywin32-302.win-amd64-py3.x.exe
(3) Compile your server.py using pyinstaller
Compiling service executable
C:\Users\Pushpender\Desktop> python -m pip install servicemanager
C:\Users\Pushpender\Desktop> pyinstaller --onefile server.py --hidden-import=win32timezone --clean --uac-admin
Installing & Running service executable (Run CMD as Administrator)
C:\WINDOWS\system32>cd C:\Users\Pushpender\Desktop>
C:\WINDOWS\system32>d:
C:\Users\Pushpender\Desktop>server.exe --startup=auto install # Installing service with startup == Automatic
C:\Users\Pushpender\Desktop>server.exe start # For starting service (You can start from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>server.exe stop # For stopping service (You can stop from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>server.exe remove # For removing installed service
(4) You can run server.py directly without compiling (Run CMD as Administrator)
C:\WINDOWS\system32>cd C:\Users\Pushpender\Desktop>
C:\WINDOWS\system32>d:
C:\Users\Pushpender\Desktop>python server.py --startup=auto install # Installing service with startup == Automatic
C:\Users\Pushpender\Desktop>python server.py start # For starting service (You can start from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>python server.py stop # For stopping service (You can stop from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>python server.py remove # For removing installed service
NOTE:
You can tweak the above code, for example, you can change the following things
_svc_display_name_ = "AAA Testing"
_svc_description_ = "This is my service"
Also you can change the classes names of FlaskService , workingthread
Please change this line in your code from mainserver import FlaskServer
You can change the startup type to lots of things, type server.exe --help in order to know more about settings
If you want to set the StartUp= Manaull, then don't use --startup=auto, while installing service
If you want to set the StartUp= Automatic (Delayed), then use --startup=delayed, while installing service
Use --startup argument before install argument
(4) Integrate Windows server with Inno Setup Builder
If you want to create a Installer which will Install your service at Installion process & will remove it at Uninstall, then
Add these block of code in your script.iss
[Run]
Filename: "{app}\{#MyAppExeName}"; StatusMsg: "Installing Windows Service ... "; Parameters: "--startup=delayed install"; Flags: runhidden waituntilterminated
Filename: "{app}\{#MyAppExeName}"; StatusMsg: "Running Windows Service ... "; Parameters: "start"; Flags: runhidden waituntilterminated
[UninstallRun]
Filename: "{app}\{#MyAppExeName}"; Parameters: "remove"; Flags: runhidden
Please ensure the below paths are added in your system variables path. The below paths are added for Python 3.7. ensure to add the path as per your python installed version.
C:\Users\1022226\AppData\Local\Programs\Python\Python37\Scripts
C:\Users\1022226\AppData\Local\Programs\Python\Python37
If you use embedded python on windows (portable python) add the folder to path.
C:\[...]\python-3.9.5-embed-amd64
This solved in my case.
My project looks like this:
running-pycharm-project-at-cmd
- main.py
- c
- run_project.py
- z
- __init__.py
- the_module.py
- y
- __init__.py
- template.md
- the_module_module.py
- the_support_function.py
The contents of the .py files are shown below:
main.py
from c.run_project import run
print('running main.py...')
run()
c/run_project.py
from c.z.the_module import the_module_function, the_module_write_function
def run():
print('Running run_project.py!')
the_module_function()
# write a file:
the_module_write_function(read_file='./z/y/template.md', write_file='../README.md')
if __name__ == '__main__':
run()
c/z/the_module.py
from c.z.y.the_module_module import the_module_module_function
def the_module_function():
print('the_module_function is running!')
the_module_module_function()
pass
def the_module_write_function(read_file, write_file):
with open(read_file, 'r') as fid:
with open(write_file, 'w') as fid_out:
contents = fid.read()
contents.replace('{}', 'THE-FINAL-PRODUCT!')
fid_out.write(contents)
c/z/y/the_module_module.py
from .the_support_function import this_support_data
def the_module_module_function():
print('The module module function is running!')
print("Here is support data: {}".format(this_support_data))
pass
c/z/y/the_support_function.py
this_support_data = "I am the support data!"
GitHub repo to make replication easier: running-pycharm-project-at-cmd
The problem: in Pycharm load up the project with running-pycharm-project-at-cmd as the root. Then I right click and run the run_project.py file. Everything works fine. I cannot figure out how to run run_project.py from the command line in the same way. Creating main.py was a workaround suggested elsewhere, but it fails due to the relative references to the template file (in actual project, the y folder is a git submodule and therefore cannot have absolute references in it).
If you copy your main.py file into the c folder and rename it __init__.py, then you could run:
$ python -m c
The -m argument tells python to run a module or package (in this case c). Python will look in the c's folder for an __init__.py file and run that. You just need to ensure that the folder c is either in your PYTHONPATH or is a subfolder of the current working directory.
I circled back to this and was able to get it working. Note that I am on Windows 7, and some of this might be platform dependent. In order to get this working without changing any code, the key is setting PYTHONPATH environment variable before running. PyCharm does this automatically (by default, but it is configurable in the Run Configuration options).
Steps:
Recreating the issue:
Clone the github repo to G:\TEST.
Navigate to G:\TEST\running-pycharm-project-at-cmd-master\c
python run_project.py
Traceback (most recent call last):
File "run_project.py", line 1, in <module>
from c.z.the_module import the_module_function, the_module_write_function
ImportError: No module named 'c'
Simplest Solution (if able to run Python from the script location):
set the PYTHONPATH environment variable before running. e.g.
SET PYTHONPATH=G:\TEST\running-pycharm-project-at-cmd-master
Now python run_project.py
Running run_project.py!
the_module_function is running!
The module module function is running!
Here is support data: I am the support data!
To run from a remote location (this is definitely Windows specific):
Create a .bat file that navigates to correct working directory before running Python. i.e.:
START cmd /K "cd G:\TEST\running-pycharm-project-at-cmd-master\c & python run_project.py"
G:\TEST\myotherlocation
run_it.bat
Running run_project.py!
the_module_function is running!
The module module function is running!
Here is support data: I am the support data!
I want to use an embedded IPython shell with a user_ns dictionary and a my profile configuration (ipython_config.py and the startup files). The purpose is to run a Django shell with models imported on startup. django-extensions implements a command called shell_plus that does this:
https://github.com/django-extensions/django-extensions/blob/master/django_extensions/management/commands/shell_plus.py
from IPython import embed
embed(user_ns=imported_objects)
The problem is that this does not load my startup files. embed() calls load_default_config() which I figure loads ipython_config.py.
How do I make the embedded IPython instance run my profile startup files?
I used the following workaround to run my own IPython startup script but still take advantage of shell_plus:
Create a file called shell_plus_startup.py in the same directory as manage.py. For example:
# File: shell_plus_startup.py
# Extra python code to run after shell_plus starts an embedded IPython shell.
# Run this file from IPython using '%run shell_plus_startup.py'
# Common imports
from datetime import date
# Common variables
tod = date.today()
Launch shell plus (which launches an embedded IPython shell).
python manage.py shell_plus
Manually run the startup script.
In [1]: %run shell_plus_startup.py
Then you can use variables you've defined, modules you've imported, etc.
In [2]: tod
Out[2]: datetime.date(2012, 7, 14)
Also see this answer: scripting ipython through django's shell_plus
I found a way that works if you are using django-extensions-shell_plus. It is a bit hacky, but with this way your startup file is loaded fully automatically and you don't have to type any run-command at the beginning of your ipython-session.
Therefore I edited the file shells.py from the django_extensions dir, which is in my case located in /usr/local/lib/python2.7/dist-packages/django_extensions/management/shells.py. I added these lines inside the function import_objects(options, style):, so it imports the content of the file startup.py defined by the environment param PYTHONSTARTUP.
def import_objects(options, style):
# (...)
import os, sys, pkgutil
if 'PYTHONSTARTUP' in os.environ:
try:
sys.path.append(os.environ['PYTHONSTARTUP'])
import startup
content = [element for element in dir(startup) if not element.startswith('__')]
for element in content:
imported_objects[element] = getattr(startup, element)
except Exception, ex:
sys.exit("Could not import startup module content, Error:\n%s" % ex)
Now when I launch the shell_plus-shell, I give it the environment variable to my startup python script. My bash script to launch the shell with everything in place looks like this:
#!/bin/bash
export PYTHONSTARTUP=/home/ifischer/src/myproject/startup.py # tells shell_plus to load this file
python /home/ifischer/src/myproject/manage.py shell_plus --ipython
Now I have access to all methods and variables defined in startup.py from the beginning of the ipython session.
So you can reuse that and have custom startup files for every project, pre-loading different aspects.
Maybe there is a cleaner way where to include the lines I added to the shells.py? But this approach works fine for me for now.
It does automatically load your ipython configuration starting from django-extensions==1.5.6. You can also pass additional arguments to ipython via IPYTHON_ARGUMENTS. Docs:
http://django-extensions.readthedocs.org/en/latest/shell_plus.html#configuration
Another way is using a class that derives from InteractiveShellEmbed and InteractiveShellApp. Sample, incomplete code:
from IPython.terminal.embed import InteractiveShellEmbed
from IPython.terminal.ipapp import InteractiveShellApp
class ISE(InteractiveShellEmbed, InteractiveShellApp):
def init_shell(self):
self.shell = self.instance()
self.extra_args = []
ise = ISE()
ise.init_path()
ise.init_shell()
ise.init_code()
ise.shell()