We Keep Coding

Script behaves differently when run from cron job and from command line using django manage.py when using a cron supervisor

I know crons run in a different environment than command lines, but I'm using absolute paths everywhere and I don't understand why my script behaves differently. I believe it is somehow related to my cron_supervisor which runs the django "manage.py" within a sub process.
Cron:
0 * * * * /home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py cron_supervisor --command="/home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py envoyer_argent"
This will call the cron_supervisor, and it's call the script, but the script won't be executed as it would if I would run:
/home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py envoyer_argent
Is there something particular to be done for the script to be called properly when running it through another script?
Here is the supervisor, which basically is for error handling and making sure we get warned if something goes wrong within the cron scripts themselves.
import logging
import os
from subprocess import PIPE, Popen
from django.core.management.base import BaseCommand
from command_utils import email_admin_error, isomorphic_logging
from utils.send_slack_message import send_slack_message
CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = CURRENT_DIR + '/../../../'
logging.basicConfig(
level=logging.INFO,
filename=PROJECT_DIR + 'cron-supervisor.log',
format='%(asctime)s %(levelname)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
class Command(BaseCommand):
help = "Control a subprocess"
def add_arguments(self, parser):
parser.add_argument(
'--command',
dest='command',
help="Command to execute",
)
parser.add_argument(
'--mute_on_success',
dest='mute_on_success',
action='store_true',
help="Don't post any massage on success",
)
def handle(self, *args, **options):
try:
isomorphic_logging(logging, "Starting cron supervisor with command \"" + options['command'] + "\"")
if options['command']:
self.command = options['command']
else:
error_message = "Empty required parameter --command"
# log error
isomorphic_logging(logging, error_message, "error")
# send slack message
send_slack_message("Cron Supervisor Error: " + error_message)
# send email to admin
email_admin_error("Cron Supervisor Error", error_message)
raise ValueError(error_message)
if options['mute_on_success']:
self.mute_on_success = True
else:
self.mute_on_success = False
# running process
process = Popen([self.command], stdout=PIPE, stderr=PIPE, shell=True)
output, error = process.communicate()
if output:
isomorphic_logging(logging, "Output from cron:" + output)
# check for any subprocess error
if process.returncode != 0:
error_message = 'Command \"{command}\" - Error \nReturn code: {code}\n```{error}```'.format(
code=process.returncode,
error=error,
command=self.command,
)
self.handle_error(error_message)
else:
message = "Command \"{command}\" ended without error".format(command=self.command)
isomorphic_logging(logging, message)
# post message on slack if process isn't muted_on_success
if not self.mute_on_success:
send_slack_message(message)
except Exception as e:
error_message = 'Command \"{command}\" - Error \n```{error}```'.format(
error=e,
command=self.command,
)
self.handle_error(error_message)
def handle_error(self, error_message):
# log the error in local file
isomorphic_logging(logging, error_message)
# post message in slack
send_slack_message(error_message)
# email admin
email_admin_error("Cron Supervisor Error", error_message)
Example of script not executed properly when being called by the cron, through the cron_supervisor:
# -*- coding: utf-8 -*-
import json
import logging
import os
from django.conf import settings
from django.core.management.base import BaseCommand
from utils.lock import handle_lock
logging.basicConfig(
level=logging.INFO,
filename=os.path.join(settings.BASE_DIR, 'crons.log'),
format='%(asctime)s %(levelname)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
class Command(BaseCommand):
help = "Envoi de l'argent en attente"
#handle_lock
def handle(self, *args, **options):
logging.info("some logs that won't be log (not called)")
logging.info("Those logs will be correcly logged")
Additionally, I have another issue with the logging which I don't quite understand either, I specify to store logs within cron-supervisor.log but they don't get stored there, I couldn't figure out why. (but that's not related to my main issue, just doesn't help with debug)

Your cron job can't just run the Python interpreter in the virtualenv; this is completely insufficient. You need to activate the env just like in an interactive environment.
0 * * * * . /home/p1/.virtualenvs/prod/bin/activate; python /home/p1/p1/manage.py cron_supervisor --command="python /home/p1/p1/manage.py envoyer_argent"
This is already complex enough that you might want to create a separate wrapper script containing these commands.
Without proper diagnostics of how your current script doesn't work, it's entirely possible that this fix alone is insufficient. Cron jobs do not only (or particularly) need absoute paths; the main differences compared to interactive shells is that cron jobs run with a different and more spare environment, where e.g. the shell's PATH, various library paths, environment variables etc can be different or missing altogether; and of course, no interactive facilities are available.
The system variables will hopefully be taken care of by your virtualenv; if it's correctly done, activating it will set up all the variables (PATH, PYTHONPATH, etc) your script needs. There could still be things like locale settings which are set up by your shell only when you log in interactively; but again, without details, let's just hope this isn't an issue for you.
The reason some people recommend absolute paths is that this will work regardless of your working directory. But a correctly written script should work fine in any directory; if it matters, the cron job will start in the owner's home directory. If you wanted to point to a relative path from there, this will work fine inside a cron job just as it does outside.
As an aside, you probably should not use subprocess.Popen() if one of the higher-level wrappers from the subprocess module do what you want. Unless compatibility with legacy Python versions is important, you should probably use subprocess.run() ... though running Python as a subprocess of Python is also often a useless oomplication. See also my answer to this related question.

Have icon overlays persist after machine restart in Python

So, thanks to the Tim Golden guide and other questions here I have a script that will show overlays on files and folders based on their "state" similar to Tortoise SVN or Dropbox.
My problem is that once I restart the explorer.exe process or the OS itself and open explorer there are no longer any overlays.
My first thought:
Have the service that actually manages file state detect that no requests have come in and just re-register the overlay handler
The problem here is that registration requires elevated permissions which is acceptable on initial install of the application by the end user but not every time they restart their machine.
Can anyone suggest what I might be missing here?
I have the class BaseOverlay and its children in a single .py file and register from my main app by calling this script using subprocess.
subprocess.check_call(script_path, shell=True)
Is Explorer not able to re-load the script as it is Python? Do I need to compile into a DLL or EXE? Would that change the registration process?
Here's the registration call:
win32com.server.register.UseCommandLine(BaseOverlay)
Here's the class(simplified):
class BaseOverlay:
_reg_clsid_ = '{8D4B1C5D-F8AC-4FDA-961F-A0143CD97C41}'
_reg_progid_ = 'someoverlays'
_reg_desc_ = 'Icon Overlay Handler'
_public_methods_ = ['GetOverlayInfo', 'GetPriority', 'IsMemberOf']
_com_interfaces_ = [shell.IID_IShellIconOverlayIdentifier]
def GetOverlayInfo(self):
return icon_path, 0, shellcon.ISIOI_ICONFILE
def GetPriority(self):
return 50
def IsMemberOf(self, fname, attributes):
return winerror.S_OK

all python windows service can not start{error 1053}

all python code service can install but cannot start
Error 1053: The service did not respond to the start or control request in a timely fashion".
since my service can install and start in my server.
i think my code has no problem.
but i still wonder is there a solution that i can solve this error in code
my service:
import win32serviceutil
import win32service
import win32event
import time
import traceback
import os
import ConfigParser
import time
import traceback
import os
import utils_func
from memcache_synchronizer import *
class MyService(win32serviceutil.ServiceFramework):
"""Windows Service."""
os.chdir(os.path.dirname(__file__))
conf_file_name = "memcache_sync_service.ini"
conf_parser = ConfigParser.SafeConfigParser()
conf_parser.read(conf_file_name)
_svc_name_, _svc_display_name_, _svc_description_ = utils_func.get_win_service(conf_parser)
def __init__(self, args):
if os.path.dirname(__file__):
os.chdir(os.path.dirname(__file__))
win32serviceutil.ServiceFramework.__init__(self, args)
# create an event that SvcDoRun can wait on and SvcStop can set.
self.stop_event = win32event.CreateEvent(None, 0, 0, None)
def SvcDoRun(self):
self.Run()
win32event.WaitForSingleObject(self.stop_event, win32event.INFINITE)
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
win32event.SetEvent(self.stop_event)
LoggerInstance.log("memcache_sync service is stopped")
self.ReportServiceStatus(win32service.SERVICE_STOPPED)
sys.exit()
def Run(self):
try:
LoggerInstance.log("\n******\n\memcache_sync_service is running, configuration: %s\n******" % (self.conf_file_name,))
if ((not self.conf_parser.has_section('Memcache')) or
(not self.conf_parser.has_option('Memcache', 'check_interval'))):
LoggerInstance.log('memcache_sync_service : no Memcache service parameters')
self.SvcStop()
# set configuration parameters from ini configuration
self.check_interval = self.conf_parser.getint('Memcache', 'check_interval')
ms = MemcacheSynchronizer()
while 1:
ms.Sync()
time.sleep(self.check_interval)
except:
LoggerInstance.log("Unhandled Exception \n\t%s" % (traceback.format_exc(),))
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(MyService)
execute result of "sc query [name]" cmd:
SERVICE_NAME: NewsMonitoringMemcacheSynchronizer
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
(NOT_STOPPABLE,NOT_PAUSABLE,IGNORES_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
update:
i can run this service with debug mode, cmd:
memcache_syn_service.py debug

Had the same problem using pypiwin32 (version: 220) and python (version: 3.6). I had to copy :
"\Python36-32\Lib\site-packages\pypiwin32_system32\pywintypes36.dll"
to
"\Python36-32\Lib\site-packages\win32"
for the service to start (was working in debug mode)

If:
python your_service.py debug works, whilst
python your_service.py install + start it as a service fails with error 1053,
this command may help python C:\Python27\Scripts\pywin32_postinstall.py.

all my python coded windows service cannot run on my computer.
but all of them can start at our dev-server which means my code is correct.
but i found a alternative solution, run in debug mode:
any_service.py debug

Make sure you run the application with a different user than the default Local System user. Replace it with the user you successfully be able to run the debug command with.
To replace the user go to the windows services (start > services.msc)
Right click on the service you created > properties > Log On
Uncheck the Local System Account and enter your own.

All of the known fixes have failed me, and this one worked:
In services window:
right-click your installed service;
Go to Log On tab;
Select "This Account" and enter your user ID and pass;
Restart PC.
Has to do with Windows Permissions I was explained...
Method won't work if there's no password set for Windows User.

In my case the problem was from python37.dll not being at C:\Python37-x64\Lib\site-packages\win32.
Just copy it there and it will solve the problem

I had similar problem with a python service and found out that it was missing DLLs since the 'System Path' (not the user path) was not complete. Check the path in your dev-server and whether it matches the one at your computer (System path if service is installed as a LocalSystem service). For me I was missing python dlls' path c:\python27 (windows).

I had this issue and solved it two times in the same way, simply adding the Environment Variables.
I opened Environment Variables, and in system variable PATH added
C:\Users\MyUser\AppData\Local\Programs\Python\PythonXXX
C:\Users\MyUser\AppData\Local\Programs\Python\PythonXXX\Scripts
(Obviously change User name and XXX with Python version)

NTEventLogHandler from a Python executable

import logging, logging.handlers
def main():
ntl = logging.handlers.NTEventLogHandler("Python Logging Test")
logger = logging.getLogger("")
logger.setLevel(logging.DEBUG)
logger.addHandler(ntl)
logger.error("This is a '%s' message", "Error")
if __name__ == "__main__":
main()
The Python (2.7.x) script above writes "This is a 'Error' message" to the Windows Event Viewer. When I run it as a script, I get the expected output. If I convert the script to an executable via PyInstaller, I get an entry in the event log but it says something completely different.
The description for Event ID ( 1 ) in Source ( Python Logging Test ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: This is a 'Error' message.
This is the command I use to convert the script into an executable: pyinstaller.py --onefile --noconsole my_script.py though the command line parameters do not appear to have any impact on this behaviour and it will suffice to just call pyinstaller.py my_script.py.
I would appreciate any help in understanding what is going on and how I go about fixing this.
Final solution
I didn't want to go down the resource hacker route, as that is going to be a difficult step to automate. Instead, the approach I took was to grab the win32service.pyd file from c:\Python27\Lib\site-packages\win32 and place it next to my executable. The script was then modified pass the full path to the copy of the win32service.pyd file and this works in both script and exe form. The final script is included below:
import logging, logging.handlers
import os
import sys
def main():
base_dir = os.path.dirname(sys.argv[0])
dllname = os.path.join(base_dir, "win32service.pyd")
ntl = logging.handlers.NTEventLogHandler("Python Logging Test", dllname=dllname)
logger = logging.getLogger("")
logger.setLevel(logging.DEBUG)
logger.addHandler(ntl)
logger.error("This is a '%s' message", "Error")
if __name__ == "__main__":
main()

Usually, the Windows Event Log doesn't store error messages in plain text, but rather message ID references and insertion strings.
Instead of storing a message like Service foo crashed unexpectedly, it stores a message ID which points to a resource string stored in a DLL. In this case, the resource would be something like Service %s crashed unexpectedly and foo would be stored as insertion string. The program which writes the message registers the resource DLL.
The reason for this is localization. DLLs can store lots of different resources (dialog layout, strings, icons…), and one DLL can contain the same resource in many different languages. The operating system automatically chooses the right resources depending on the system locale. Resource DLLs are used by virtually all Microsoft utilities and core utilities.
Side note: Nowadays, the preferred (and cross-platform) way for localization is gettext.
This is used for the message log as well – ideally, you could open a log from an German Windows installation on an English one with all messages in English.
I suspect that the pywin32 implementation skips that mechanism by only having one single message ID (1) which is just something like "%s". It is stored in win32service.pyd and registered by pywin32. This works fine as long as this file exists on the file system, but breaks as soon as it's hidden inside a PyInstaller executable. I guess you have to embed the message ID into your executable directly.
Edit: suspicion confirmed, the message table is indeed stored inside win32service.pyd
Resource Hacker showing the message table http://media.leoluk.de/evlog_rh.png
Try to copy the message table resource from win32service.pyd to your PyInstaller executable (for example using Resource Hacker).
Looking at the logging handler implementation, this might work:
def __init__(self, appname, dllname=None, logtype="Application"):
logging.Handler.__init__(self)
try:
import win32evtlogutil, win32evtlog
self.appname = appname
self._welu = win32evtlogutil
if not dllname:
dllname = os.path.split(self._welu.__file__)
dllname = os.path.split(dllname[0])
dllname = os.path.join(dllname[0], r'win32service.pyd')
You'd have to set dllname to os.path.dirname(__file__). Use something like this if you want it to continue working for the unfrozen script:
if getattr(sys, 'frozen', False):
dllname = None
elif __file__:
dllname = os.path.dirname(__file__)
ntl = logging.handlers.NTEventLogHandler("Python Logging Test", dllname=dllname)

CherryPy3 and IIS 6.0

I have a small Python web application using the Cherrypy framework. I am by no means an expert in web servers.
I got Cherrypy working with Apache using mod_python on our Ubuntu server. This time, however, I have to use Windows 2003 and IIS 6.0 to host my site.
The site runs perfectly as a stand alone server - I am just so lost when it comes to getting IIS running. I have spent the past day Googling and blindly trying any and everything to get this running.
I have all the various tools installed that websites have told me to (Python 2.6, CherrpyPy 3, ISAPI-WSGI, PyWin32) and have read all the documentation I can. This blog was the most helpful:
http://whatschrisdoing.com/blog/2008/07/10/turbogears-isapi-wsgi-iis/
But I am still lost as to what I need to run my site. I can't find any thorough examples or how-to's to even start with. I hope someone here can help!
Cheers.

I run CherryPy behind my IIS sites. There are several tricks to get it to work.
When running as the IIS Worker Process identity, you won't have the same permissions as you do when you run the site from your user process. Things will break. In particular, anything that wants to write to the file system will probably not work without some tweaking.
If you're using setuptools, you probably want to install your components with the -Z option (unzips all eggs).
Use win32traceutil to track down problems. Be sure that in your hook script that you're importing win32traceutil. Then, when you're attempting to access the web site, if anything goes wrong, make sure it gets printed to standard out, it'll get logged to the trace utility. Use 'python -m win32traceutil' to see the output from the trace.
It's important to understand the basic process to get an ISAPI application running. I suggest first getting a hello-world WSGI application running under ISAPI_WSGI. Here's an early version of a hook script I used to validate that I was getting CherryPy to work with my web server.
#!python
"""
Things to remember:
easy_install munges permissions on zip eggs.
anything that's installed in a user folder (i.e. setup develop) will probably not work.
There may still exist an issue with static files.
"""
import sys
import os
import isapi_wsgi
# change this to '/myapp' to have the site installed to only a virtual
# directory of the site.
site_root = '/'
if hasattr(sys, "isapidllhandle"):
import win32traceutil
appdir = os.path.dirname(__file__)
egg_cache = os.path.join(appdir, 'egg-tmp')
if not os.path.exists(egg_cache):
os.makedirs(egg_cache)
os.environ['PYTHON_EGG_CACHE'] = egg_cache
os.chdir(appdir)
import cherrypy
import traceback
class Root(object):
#cherrypy.expose
def index(self):
return 'Hai Werld'
def setup_application():
print "starting cherrypy application server"
#app_root = os.path.dirname(__file__)
#sys.path.append(app_root)
app = cherrypy.tree.mount(Root(), site_root)
print "successfully set up the application"
return app
def __ExtensionFactory__():
"The entry point for when the ISAPIDLL is triggered"
try:
# import the wsgi app creator
app = setup_application()
return isapi_wsgi.ISAPISimpleHandler(app)
except:
import traceback
traceback.print_exc()
f = open(os.path.join(appdir, 'critical error.txt'), 'w')
traceback.print_exc(file=f)
f.close()
def install_virtual_dir():
import isapi.install
params = isapi.install.ISAPIParameters()
# Setup the virtual directories - this is a list of directories our
# extension uses - in this case only 1.
# Each extension has a "script map" - this is the mapping of ISAPI
# extensions.
sm = [
isapi.install.ScriptMapParams(Extension="*", Flags=0)
]
vd = isapi.install.VirtualDirParameters(
Server="CherryPy Web Server",
Name=site_root,
Description = "CherryPy Application",
ScriptMaps = sm,
ScriptMapUpdate = "end",
)
params.VirtualDirs = [vd]
isapi.install.HandleCommandLine(params)
if __name__=='__main__':
# If run from the command-line, install ourselves.
install_virtual_dir()
This script does several things. It (a) acts as the installer, installing itself into IIS [install_virtual_dir], (b) contains the entry point when IIS loads the DLL [__ExtensionFactory__], and (c) it creates the CherryPy WSGI instance consumed by the ISAPI handler [setup_application].
If you place this in your \inetpub\cherrypy directory and run it, it will attempt to install itself to the root of your IIS web site named "CherryPy Web Server".
You're also welcome to take a look at my production web site code, which has refactored all of this into different modules.

OK, I got it working. Thanks to Jason and all his help. I needed to call
cherrypy.config.update({
'tools.sessions.on': True
})
return cherrypy.tree.mount(Root(), '/', config=path_to_config)
I had this in the config file under [/] but for some reason it did not like that. Now I can get my web app up and running - then I think I will try and work out why it needs that config update and doesn't like the config file I have...