I am trying to run a python program, written using Anaconda, as a Windows Service. The complexity is that I'd like to run the Windows Service from a specific conda virtual environment. The idea being that in the future, we may develop more python based windows services that may have different module dependencies, thus keeping each within its own virtual environment would be ideal.
I've found several excellent articles about how to write a python program as a windows service and they work fine. I created a very simple test program that simply writes some messages to a text file after the service starts. I can successfully install this test python program as a windows service and I see the various text messages in my file. However, when I try to import modules like Numpy or TensorFlow into my simple test python program, the service won't start and I get failure messages that their respective DLL's can't be found.
I am sure the problem is because the required conda virtual environment hasn't been activated. At the same time, I've tried replicating the various conda environment variables at the system level; tried adding all of the required python library paths from the virtual environment to the system path and system-wide python path, but to no avail.
I suspect that if I could activate the conda virtual environment as part of my python code, that would solve the problem. (I also suspect that installing all of the required modules into my base configuration would solve the problem, but i'd like to avoid that).
Here is the little test program I've written. This program works just fine with the basic Python modules like sys, os and so forth. It fails with the following error message when I try to run it and include Numpy or TensorFlow:
(This is from the Windows Event Viewer after I try and start my service - which does install correctly):
Python could not import the service's module
Traceback (most recent call last):
File "D:\TFS\Projects\DEV\AEPEnrollmentForms\src\aepenrl\Windows_Service_Example.py", line 35, in
import numpy as np
File "C:\Users\pboerner\AppData\Local\conda\conda\envs\aepenr\lib\site-packages\numpy__init__.py", line 140, in
from . import _distributor_init
File "C:\Users\pboerner\AppData\Local\conda\conda\envs\aepenr\lib\site-packages\numpy_distributor_init.py", line 34, in
from . import _mklinit
ImportError: DLL load failed: The specified module could not be found.
%2: %3
Here is the code for the simple test program. (Most of the Windows Service integration work I took from an excellent article provided by Davide Mastromatteo)
import numpy as np
import socket
import sys
import time
import win32serviceutil
import servicemanager
import win32event
import win32service
class SimpleService(win32serviceutil.ServiceFramework):
'''Base class to create winservice in Python'''
_svc_name_ = 'TestPythonSrvc'
_svc_display_name_ = 'Test Python Service'
_svc_description_ = 'Test to see how to create a windows service with python'
#classmethod
def parse_command_line(cls):
'''
ClassMethod to parse the command line
'''
win32serviceutil.HandleCommandLine(cls)
def __init__(self, args):
'''
Constructor of the winservice
'''
self.isrunning=True
self.fid = open("D:\\temp\\simple_service.txt", "w")
self.fid.write("Initialize\n")
self.fid.flush()
win32serviceutil.ServiceFramework.__init__(self, args)
self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
socket.setdefaulttimeout(60)
def SvcStop(self):
'''
Called when the service is asked to stop
'''
self.stop()
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
win32event.SetEvent(self.hWaitStop)
def SvcDoRun(self):
'''
Called when the service is asked to start
'''
self.start()
servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
servicemanager.PYS_SERVICE_STARTED,
(self._svc_name_, ''))
self.main()
def start(self):
'''
Override to add logic before the start
eg. running condition
'''
self.isrunning = True
self.fid.write("Start method called\n")
self.fid.flush()
def stop(self):
'''
Override to add logic before the stop
eg. invalidating running condition
'''
self.isrunning = False
self.fid.write("STOP method called. Setting stop flag\n")
self.fid.flush()
def main(self):
'''
Main class to be ovverridden to add logic
'''
a = np.zeros((100,1))
while True:
if self.isrunning:
self.fid.write(f"Tick. Numpy array shape {a.shape}\n")
self.fid.flush()
time.sleep(1)
else:
self.fid.write("Breaking out of main loop\n")
self.fid.flush()
break;
self.fid.write("Closing the log file\n")
self.fid.flush()
self.fid.close()
if __name__ == '__main__':
# This code block was required to get this simple service example to run
# on a Windows 10 laptop with Admin privs. Only calling the
# HandleCommandLine method alone didn'd seem to work. Not sure why but this
# code was provided as a solution on the Web.
if len(sys.argv) == 1:
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(SimpleService)
servicemanager.StartServiceCtrlDispatcher()
else:
win32serviceutil.HandleCommandLine(SimpleService)
Unfortunately the service manager of Windows is not as fexible as, let's say, systemd. The only method I found was to do the following:
make a batch file holding all your logic, e.g.
call C:\ProgramData\Anaconda3\Scripts\activate.bat C:\ProgramData\Anaconda3
call activate yourenv
cd C:/path/to/your/wd
python yourservice.py and your args
NB: your activate.bat file could be under your home folder: ~\AppData\local\Continuum\anaconda3\Scripts
Use the NSSM (which is aptly named): http://nssm.cc/download ; see also Run batch file as a Windows service . You need to start the command prompt or powershell as administrator before calling nssm.
This works to serve a bokeh server really well (using bokeh serveinstead of python). I guess it works for python script of any complexity then.
You can run a command upon stopping or exiting within the "hooks" tab.
In my case the logic within the batch file depends on the machine, so I need to make an additional python script that is called upon setting up to write the batch file.
I was stuck in the exact same situation when I came across a blog (link below) and followed the steps in it
Create a run.bat file in your working directory with the following lines.
call conda activate env
call python app.py
So that when you call your batch file it will first activate the virtual environment and then run the python script.
Download NSSM, no need to install it. Once downloaded, go to the NSSM -> win32/win64 (as per your computer architecture) and run the following command in the command prompt as administrator:
C:\nssm-2.24\win64\nssm install <service name> "C:\path\to\your\run.bat"
Your service is installed at this point but not yet started.
You might need to configure it
C:\nssm-2.24\win64\nssm edit <service name>
NSSM service editor will open after executing the above command where you can set Display names, description, log files, etc.
Once done you can start the service as
C:\nssm-2.24\win64\nssm start <service name>
Link for reference:-
Run Windows Service in its own Python virtual environment
I simply followed these steps and it worked just fine for me. I hope it will work for you as well.
I have flask app with following extensions used:
Python 3.7.2
Flask 1.0.2
Celery 4.3.0rc2
dlib 19.6
Without celery, as a regular single-thread program for face detection, everything works fine, but when I'm starting it as a task - the following error occurs:
Task handler raised error: WorkerLostError('Worker exited prematurely:
signal 11 (SIGSEGV).')
on this piece of code:
dlib.cnn_face_detection_model_v1(model)
And I don't get why, but with this:
dlib.get_frontal_face_detector()
it works fine!
I know the conflict caused by dlib (or BLAS) not being thread-safe, but is there any way to disable multiprocessing either make celery workers work with this?
UPD:
I have following project structure:
./app/face_detector.py
./tasks.py
In tasks.py I'm using this on top of the file:
from app.face_detector import FaceDetector
the trick with import inside of task:
#app.task
def foo():
from app.face_detector import FaceDetector
Doesn't work at all and throws this:
no module named...
So, the solution from github thread was not understood by me or not working
Importing packages that are subsequently installed (not present by default in python distribution for rhel 7.6) not working when ran as cron job
Hi Team,
I have a python(2.7) script which imports paramiko package. The script can import the paramiko package successfully when ran as a user(root or ftpuser) after logging in but it cannot import it when ran from cron job. I have tried out various options as provided in the brilliant stack overflow pages like the below but unfortunately couldn't resolve the issue.
1) Crontab not running my python script
I have provided the path to the paramiko package and verified it is successfully received at the script end by logging it when run as cron job and also I have given chmod -R 777 permission to the paramiko folder in the /opt/rh/python27/root/usr/lib/python2.7/site-packages location. Still the import is not functioning when ran as cron job
I have created a shell script and tried to invoke python script from with in the script and configured the shell script in cron job but it seems python script was not invoked
I have verified that there is only one python installation present in the server and so I' am using the correct path
I have disabled selinux option and tried after rebooting but issue still persists
Please note the issue exists not for just paramiko package but for other packages as well that was installed subsequently like mysql.connector e t c
Update1
It has to be something to do with the way I install the paramiko package because the script can even import other packages in the same path as that of paramiko and the permissions for both of them look identical only difference is former comes with the python distribution that is deployed from using the url https://access.redhat.com/solutions/1519803. Cannot figure out what is wrong with the installation steps as I install it as root after doing sudo su and then setting umask to 0022. I do pip install of parmiko and python-crontab as mentioned in their sites and both have the same issue
Another interesting thing is though I have the code to log exception around the failing import statement it never logs the exception but script seems to halt/hang at the import statment
Please help to resolve this issue...
PYTHON CODE
#!/usr/bin/env python
import sys
import logging
import os
def InitLog():
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s',
filename=os.path.dirname(os.path.abspath(__file__)) + '/test_paramiko.log',
filemode='a'
)
logging.info('***************start logging****************')
InitLog()
logging.info('before import')
logging.info(sys.path)
try:
sys.path.append("/opt/rh/python27/root/usr/lib/python2.7/site-packages")
logging.info("sys path appended before import")
import paramiko
except ImportError:
logging.ERROR("Exception occured druing import")
logging.info('after import')
CRONTAB Entry
SHELL=/bin/bash
PATH=/opt/rh/python27/root/usr/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ftpuser/.local/bin:/home/ftpuser/bin
PYTHONPATH=/opt/rh/python27/root/usr/lib64/python27.zip:/opt/rh/python27/root/usr/lib64/python2.7:/opt/rh/python27/root/usr/lib64/python2.7/plat-linux2:/opt/rh/python27/root/usr/lib64/python2.7/lib-tk:/opt/rh/python27/root/usr/lib64/python2.7/lib-old:/opt/rh/python27/root/usr/lib64/python2.7/lib-dynload:/opt/rh/python27/root/usr/lib64/python2.7/site-packages:/opt/rh/python27/root/usr/lib/python2.7/site-packages
*/1 * * * * /opt/rh/python27/root/usr/bin/python /home/ftpuser/Ganesh/test_paramiko.py
#*/1 * * * * /home/ftpuser/Ganesh/test_cron.sh >> /home/ftpuser/Ganesh/tes_cron.txt 2>&1
#*/1 * * * * /home/ftpuser/Ganesh/test_cron.sh
Shell script
#!/opt/rh/python27/root/usr/bin/python
export PATH=$PATH:/opt/rh/python27/root/usr/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ftpuser/.local/bin:/home/ftpuser/bin
export PYTHONPATH=$PYTHONPATH:/opt/rh/python27/root/usr/lib64/python27.zip:/opt/rh/python27/root/usr/lib64/python2.7:/opt/rh/python27/root/usr/lib64/python2.7/plat-linux2:/opt/rh/python27/root/usr/lib64/python2.7/lib-tk:/opt/rh/python27/root/usr/lib64/python2.7/lib-old:/opt/rh/python27/root/usr/lib64/python2.7/lib-dynload:/opt/rh/python27/root/usr/lib64/python2.7/site-packages:/opt/rh/python27/root/usr/lib/python2.7/site-packages
python /home/ftpuser/Ganesh/test_paramiko.py
Expected result from my python script is to log the "after import" string
But currently it is printing only till "sys path appended before import" which also shows the normal python packages are getting imported successfully
This seems to be working now after adding one more environment variable to crontab as below
LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64
I am running the code below by python win_service.py install from the normal command prompt, where I get access denied error.
Installing service TestService
Error installing service: Access is denied. (5)
which I was able to resolve when I started the command prompt by starting as administrator.
I was able to install the service, but I was unable to start the service.
Service installed
Starting service TestService
Error starting service: The service did not respond to the start or control request in a timely fashion.
import win32serviceutil
import win32service
import win32event
import servicemanager
import socket
class AppServerSvc (win32serviceutil.ServiceFramework):
_svc_name_ = "TestService"
_svc_display_name_ = "Test Service"
def __init__(self,args):
win32serviceutil.ServiceFramework.__init__(self,args)
self.hWaitStop = win32event.CreateEvent(None,0,0,None)
socket.setdefaulttimeout(60)
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
win32event.SetEvent(self.hWaitStop)
def SvcDoRun(self):
servicemanager.LogMsg(servicemanager.EVENTLOG_INFORMATION_TYPE,
servicemanager.PYS_SERVICE_STARTED,
(self._svc_name_,''))
self.main()
def main(self):
print "running"
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(AppServerSvc)
What am doing wrong, is there any other way to install the service that would solve the issue and how to dynamically run it as administrator.
I know this is old but I was stuck on this forever. For me, this specific problem was solved by copying this file - pywintypes36.dll
From -> Python36\Lib\site-packages\pywin32_system32
To -> Python36\Lib\site-packages\win32
It's possible that your service is not starting because it's unable to find the executable. I had a similar issue that was solved by adding some pywin32 related directories to my system path. You can do this using setx:
setx /M PATH "%PATH%;C:\Python27;C:\Python27\Scripts;C:\Python27\Lib\site-packages\pywin32_system32;C:\Python27\Lib\site-packages\win32"
Try running this in a cmd window with admin privileges and adjust the paths to match your own python installation.
Finally, the solution for this.
First step:
USE pyinstaller to create a standalone executable file, i.e.:
pip install pyinstaller
pyinstaller yourproject.py
cd dist\yourproject
yourproject.exe install
Second step:
Note that. When the Windows Service calls "your program", it gives a time to answer according the Service Development Protocol. All of the codes above, are not starting the service. Please, change your code as below:
if __name__ == '__main__':
if len(sys.argv) > 1:
# Called by Windows shell. Handling arguments such as: Install, Remove, etc.
win32serviceutil.HandleCommandLine(JobManager)
else:
# Called by Windows Service. Initialize the service to communicate with the system operator
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(JobManager)
servicemanager.StartServiceCtrlDispatcher()
As noted by Chip (and which I've missed while trying to figure this out), pythonservice.exe is going to run as a system service, so it will have a different environment than you do as a user. Running python service.py debug will run just fine because it's still running with your user environment, but if you run python service.py start, it may now fail instead due to the difference in environment variables. An instant timeout is most likely due to pythonservice.exe failing to execute, and it will fail to execute it it's missing either PythonXX.dll or pywintypesXX.dll.
PythonXX.dll is likely to be in your system path already (depending on how Python was installed), but if you're like me and trying to be extra careful to not alter the environment, that's going to be a problem. I was running something like .\.pyenv37\Scripts\python.exe service.py start, gets Python37.dll from the PATH, not the venv like I assumed, so it's no longer known when pythonservice.exe starts running using a different PATH, which causes it to immediately fail Windows reports it as an instant timeout.
The same goes for pywintypesXX.dll, except instead of installing it somewhere in your search path, the more portable solution is to drop it in the same directory as pythonservice.exe since the deafult DLL search path includes it. It will also cause your service to immediately timeout if it if pythonservice.exe can't find it.
Figuring this out without any sort of logs is an absolute nightmare, let me tell you!
EDIT: Here's what I'm using to verify all of that on script installation/update:
# customOptionHandler will only run after service install/update
if __name__=='__main__':
win32serviceutil.HandleCommandLine(AppServerSvc, customOptionHandler=post_service_update)
.
def post_service_update(*args):
import win32api, win32con, win32profile, pywintypes
from contextlib import closing
env_reg_key = "SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Environment"
hkey = win32api.RegOpenKeyEx(win32con.HKEY_LOCAL_MACHINE, env_reg_key, 0, win32con.KEY_ALL_ACCESS)
with closing(hkey):
system_path = win32api.RegQueryValueEx(hkey, 'PATH')[0]
# PATH may contain %SYSTEM_ROOT% or other env variables that must be expanded
# ExpandEnvironmentStringsForUser(None) only expands System variables
system_path = win32profile.ExpandEnvironmentStringsForUser(None, system_path)
system_path_list = system_path.split(os.pathsep)
core_dll_file = win32api.GetModuleFileName(sys.dllhandle)
core_dll_name = os.path.basename(core_dll_file)
for search_path_dir in system_path_list:
try:
dll_path = win32api.SearchPath(search_path_dir, core_dll_name)[0]
print(f"System python DLL: {dll_path}")
break
except pywintypes.error as ex:
if ex.args[1] != 'SearchPath': raise
continue
else:
print("*** WARNING ***")
print(f"Your current Python DLL ({core_dll_name}) is not in your SYSTEM PATH")
print("The service is likely to not launch correctly.")
from win32serviceutil import LocatePythonServiceExe
pythonservice_exe = LocatePythonServiceExe()
pywintypes_dll_file = pywintypes.__spec__.origin
pythonservice_path = os.path.dirname(pythonservice_exe)
pywintypes_dll_name = os.path.basename(pywintypes_dll_file)
try:
return win32api.SearchPath(pythonservice_path, pywintypes_dll_name)[0]
except pywintypes.error as ex:
if ex.args[1] != 'SearchPath': raise
print("*** WARNING ***")
print(f"{pywintypes_dll_name} is not is the same directory as pythonservice.exe")
print(f'Copy "{pywintypes_dll_file}" to "{pythonservice_path}"')
print("The service is likely to not launch correctly.")
It may seem like a lot, but it will at leastkeep you from forgetting to do those steps when deploying the service on a new machine/virtual environment or when updating python.
Before you start debugging, I recommend to make sure that the two steps described at https://github.com/mhammond/pywin32, which are
pip install pywin32
and
python Scripts/pywin32_postinstall.py -install
were completed.
if you are using anaconda python, be sure python36.dll is in your system path. This took me a long time to find.
credit: Can't start Windows service written in Python (win32serviceutil)
Make sure you run the application with a different user than the default Local System user. Replace it with the user you successfully be able to run the debug command with.
To replace the user go to the windows services (start > services.msc)
Right click on the service you created > properties > Log On
Uncheck the Local System Account and enter your own.
from all python windows service can not start{error 1053} worked for me.
Because I just set PATH for user login, not for System. You can recheck your PATH for System Variables.
I was also facing the same problem but, After messing out for 4 days, Finally I'm able to find the solution for this problem.
So, Here is the step by step Beginner Guide : https://github.com/PushpenderIndia/PythonWindowsService
Also Posting the same solution here as well.
Steps To Create an Python Windows Service
(1) Copy Paste These Codes to a Python File (e.g. server.py)
import servicemanager
import sys
import win32serviceutil
from mainserver import FlaskServer # Import your code, I've written a module called mainserver which contains FlaskServer Code using OOPs
import threading
import concurrent.futures
import time
class workingthread(threading.Thread):
def __init__(self, quitEvent):
self.quitEvent = quitEvent
self.waitTime = 1
threading.Thread.__init__(self)
def run(self):
try:
# Running start_flask() function on different thread, so that it doesn't blocks the code
executor = concurrent.futures.ThreadPoolExecutor(max_workers=5)
executor.submit(self.start_flask)
except:
pass
# Following Lines are written so that, the program doesn't get quit
# Will Run a Endless While Loop till Stop signal is not received from Windows Service API
while not self.quitEvent.isSet(): # If stop signal is triggered, exit
time.sleep(1)
def start_flask(self):
# This Function contains the actual logic, of windows service
# This is case, we are running our flaskserver
test = FlaskServer()
test.start()
class FlaskService(win32serviceutil.ServiceFramework):
_svc_name_ = "AA Testing"
_svc_display_name_ = "AAA Testing"
_svc_description_ = "This is my service"
def __init__(self, args):
win32serviceutil.ServiceFramework.__init__(self, args)
self.hWaitStop = threading.Event()
self.thread = workingthread(self.hWaitStop)
def SvcStop(self):
self.hWaitStop.set()
def SvcDoRun(self):
self.thread.start()
self.hWaitStop.wait()
self.thread.join()
if __name__ == '__main__':
if len(sys.argv) == 1:
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(FlaskService)
servicemanager.StartServiceCtrlDispatcher()
else:
win32serviceutil.HandleCommandLine(FlaskService)
(2) Install the latest pywin32.exe
NOTE: If you install pywin32 using pip, then it will not going to install properly, thus will show you some errors,
Visit https://github.com/mhammond/pywin32/releases
And Download the latest exe, If you are using Python 32bit then download pywin32-302.win32-py3.x.exe
If using Python 64 bit, then download pywin32-302.win-amd64-py3.x.exe
(3) Compile your server.py using pyinstaller
Compiling service executable
C:\Users\Pushpender\Desktop> python -m pip install servicemanager
C:\Users\Pushpender\Desktop> pyinstaller --onefile server.py --hidden-import=win32timezone --clean --uac-admin
Installing & Running service executable (Run CMD as Administrator)
C:\WINDOWS\system32>cd C:\Users\Pushpender\Desktop>
C:\WINDOWS\system32>d:
C:\Users\Pushpender\Desktop>server.exe --startup=auto install # Installing service with startup == Automatic
C:\Users\Pushpender\Desktop>server.exe start # For starting service (You can start from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>server.exe stop # For stopping service (You can stop from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>server.exe remove # For removing installed service
(4) You can run server.py directly without compiling (Run CMD as Administrator)
C:\WINDOWS\system32>cd C:\Users\Pushpender\Desktop>
C:\WINDOWS\system32>d:
C:\Users\Pushpender\Desktop>python server.py --startup=auto install # Installing service with startup == Automatic
C:\Users\Pushpender\Desktop>python server.py start # For starting service (You can start from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>python server.py stop # For stopping service (You can stop from Windows Service or From Task Manager)
C:\Users\Pushpender\Desktop>python server.py remove # For removing installed service
NOTE:
You can tweak the above code, for example, you can change the following things
_svc_display_name_ = "AAA Testing"
_svc_description_ = "This is my service"
Also you can change the classes names of FlaskService , workingthread
Please change this line in your code from mainserver import FlaskServer
You can change the startup type to lots of things, type server.exe --help in order to know more about settings
If you want to set the StartUp= Manaull, then don't use --startup=auto, while installing service
If you want to set the StartUp= Automatic (Delayed), then use --startup=delayed, while installing service
Use --startup argument before install argument
(4) Integrate Windows server with Inno Setup Builder
If you want to create a Installer which will Install your service at Installion process & will remove it at Uninstall, then
Add these block of code in your script.iss
[Run]
Filename: "{app}\{#MyAppExeName}"; StatusMsg: "Installing Windows Service ... "; Parameters: "--startup=delayed install"; Flags: runhidden waituntilterminated
Filename: "{app}\{#MyAppExeName}"; StatusMsg: "Running Windows Service ... "; Parameters: "start"; Flags: runhidden waituntilterminated
[UninstallRun]
Filename: "{app}\{#MyAppExeName}"; Parameters: "remove"; Flags: runhidden
Please ensure the below paths are added in your system variables path. The below paths are added for Python 3.7. ensure to add the path as per your python installed version.
C:\Users\1022226\AppData\Local\Programs\Python\Python37\Scripts
C:\Users\1022226\AppData\Local\Programs\Python\Python37
If you use embedded python on windows (portable python) add the folder to path.
C:\[...]\python-3.9.5-embed-amd64
This solved in my case.
I'm writing a build phase script in xcode using python.
I want to abort building when the script encounters an error en increment the error count in xcode.
What would be the way to do that ?
#!/usr/bin/env python
aStringThatShouldNotBeEmpty = ""
if not aStringThatShouldNotBeEmpty:
# abort build process and print error in issue navigator
xcode expects a non-zero exit code as failure.
So:
#!/usr/bin/env python
import sys
aStringThatShouldNotBeEmpty = ""
if not aStringThatShouldNotBeEmpty:
# abort build process and print error in issue navigator
print 'Your own error, write whatever you want'
sys.exit(1)