import logging, logging.handlers
def main():
ntl = logging.handlers.NTEventLogHandler("Python Logging Test")
logger = logging.getLogger("")
logger.setLevel(logging.DEBUG)
logger.addHandler(ntl)
logger.error("This is a '%s' message", "Error")
if __name__ == "__main__":
main()
The Python (2.7.x) script above writes "This is a 'Error' message" to the Windows Event Viewer. When I run it as a script, I get the expected output. If I convert the script to an executable via PyInstaller, I get an entry in the event log but it says something completely different.
The description for Event ID ( 1 ) in Source ( Python Logging Test ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: This is a 'Error' message.
This is the command I use to convert the script into an executable: pyinstaller.py --onefile --noconsole my_script.py though the command line parameters do not appear to have any impact on this behaviour and it will suffice to just call pyinstaller.py my_script.py.
I would appreciate any help in understanding what is going on and how I go about fixing this.
Final solution
I didn't want to go down the resource hacker route, as that is going to be a difficult step to automate. Instead, the approach I took was to grab the win32service.pyd file from c:\Python27\Lib\site-packages\win32 and place it next to my executable. The script was then modified pass the full path to the copy of the win32service.pyd file and this works in both script and exe form. The final script is included below:
import logging, logging.handlers
import os
import sys
def main():
base_dir = os.path.dirname(sys.argv[0])
dllname = os.path.join(base_dir, "win32service.pyd")
ntl = logging.handlers.NTEventLogHandler("Python Logging Test", dllname=dllname)
logger = logging.getLogger("")
logger.setLevel(logging.DEBUG)
logger.addHandler(ntl)
logger.error("This is a '%s' message", "Error")
if __name__ == "__main__":
main()
Usually, the Windows Event Log doesn't store error messages in plain text, but rather message ID references and insertion strings.
Instead of storing a message like Service foo crashed unexpectedly, it stores a message ID which points to a resource string stored in a DLL. In this case, the resource would be something like Service %s crashed unexpectedly and foo would be stored as insertion string. The program which writes the message registers the resource DLL.
The reason for this is localization. DLLs can store lots of different resources (dialog layout, strings, icons…), and one DLL can contain the same resource in many different languages. The operating system automatically chooses the right resources depending on the system locale. Resource DLLs are used by virtually all Microsoft utilities and core utilities.
Side note: Nowadays, the preferred (and cross-platform) way for localization is gettext.
This is used for the message log as well – ideally, you could open a log from an German Windows installation on an English one with all messages in English.
I suspect that the pywin32 implementation skips that mechanism by only having one single message ID (1) which is just something like "%s". It is stored in win32service.pyd and registered by pywin32. This works fine as long as this file exists on the file system, but breaks as soon as it's hidden inside a PyInstaller executable. I guess you have to embed the message ID into your executable directly.
Edit: suspicion confirmed, the message table is indeed stored inside win32service.pyd
Resource Hacker showing the message table http://media.leoluk.de/evlog_rh.png
Try to copy the message table resource from win32service.pyd to your PyInstaller executable (for example using Resource Hacker).
Looking at the logging handler implementation, this might work:
def __init__(self, appname, dllname=None, logtype="Application"):
logging.Handler.__init__(self)
try:
import win32evtlogutil, win32evtlog
self.appname = appname
self._welu = win32evtlogutil
if not dllname:
dllname = os.path.split(self._welu.__file__)
dllname = os.path.split(dllname[0])
dllname = os.path.join(dllname[0], r'win32service.pyd')
You'd have to set dllname to os.path.dirname(__file__). Use something like this if you want it to continue working for the unfrozen script:
if getattr(sys, 'frozen', False):
dllname = None
elif __file__:
dllname = os.path.dirname(__file__)
ntl = logging.handlers.NTEventLogHandler("Python Logging Test", dllname=dllname)
Related
I know crons run in a different environment than command lines, but I'm using absolute paths everywhere and I don't understand why my script behaves differently. I believe it is somehow related to my cron_supervisor which runs the django "manage.py" within a sub process.
Cron:
0 * * * * /home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py cron_supervisor --command="/home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py envoyer_argent"
This will call the cron_supervisor, and it's call the script, but the script won't be executed as it would if I would run:
/home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py envoyer_argent
Is there something particular to be done for the script to be called properly when running it through another script?
Here is the supervisor, which basically is for error handling and making sure we get warned if something goes wrong within the cron scripts themselves.
import logging
import os
from subprocess import PIPE, Popen
from django.core.management.base import BaseCommand
from command_utils import email_admin_error, isomorphic_logging
from utils.send_slack_message import send_slack_message
CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = CURRENT_DIR + '/../../../'
logging.basicConfig(
level=logging.INFO,
filename=PROJECT_DIR + 'cron-supervisor.log',
format='%(asctime)s %(levelname)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
class Command(BaseCommand):
help = "Control a subprocess"
def add_arguments(self, parser):
parser.add_argument(
'--command',
dest='command',
help="Command to execute",
)
parser.add_argument(
'--mute_on_success',
dest='mute_on_success',
action='store_true',
help="Don't post any massage on success",
)
def handle(self, *args, **options):
try:
isomorphic_logging(logging, "Starting cron supervisor with command \"" + options['command'] + "\"")
if options['command']:
self.command = options['command']
else:
error_message = "Empty required parameter --command"
# log error
isomorphic_logging(logging, error_message, "error")
# send slack message
send_slack_message("Cron Supervisor Error: " + error_message)
# send email to admin
email_admin_error("Cron Supervisor Error", error_message)
raise ValueError(error_message)
if options['mute_on_success']:
self.mute_on_success = True
else:
self.mute_on_success = False
# running process
process = Popen([self.command], stdout=PIPE, stderr=PIPE, shell=True)
output, error = process.communicate()
if output:
isomorphic_logging(logging, "Output from cron:" + output)
# check for any subprocess error
if process.returncode != 0:
error_message = 'Command \"{command}\" - Error \nReturn code: {code}\n```{error}```'.format(
code=process.returncode,
error=error,
command=self.command,
)
self.handle_error(error_message)
else:
message = "Command \"{command}\" ended without error".format(command=self.command)
isomorphic_logging(logging, message)
# post message on slack if process isn't muted_on_success
if not self.mute_on_success:
send_slack_message(message)
except Exception as e:
error_message = 'Command \"{command}\" - Error \n```{error}```'.format(
error=e,
command=self.command,
)
self.handle_error(error_message)
def handle_error(self, error_message):
# log the error in local file
isomorphic_logging(logging, error_message)
# post message in slack
send_slack_message(error_message)
# email admin
email_admin_error("Cron Supervisor Error", error_message)
Example of script not executed properly when being called by the cron, through the cron_supervisor:
# -*- coding: utf-8 -*-
import json
import logging
import os
from django.conf import settings
from django.core.management.base import BaseCommand
from utils.lock import handle_lock
logging.basicConfig(
level=logging.INFO,
filename=os.path.join(settings.BASE_DIR, 'crons.log'),
format='%(asctime)s %(levelname)s: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
class Command(BaseCommand):
help = "Envoi de l'argent en attente"
#handle_lock
def handle(self, *args, **options):
logging.info("some logs that won't be log (not called)")
logging.info("Those logs will be correcly logged")
Additionally, I have another issue with the logging which I don't quite understand either, I specify to store logs within cron-supervisor.log but they don't get stored there, I couldn't figure out why. (but that's not related to my main issue, just doesn't help with debug)
Your cron job can't just run the Python interpreter in the virtualenv; this is completely insufficient. You need to activate the env just like in an interactive environment.
0 * * * * . /home/p1/.virtualenvs/prod/bin/activate; python /home/p1/p1/manage.py cron_supervisor --command="python /home/p1/p1/manage.py envoyer_argent"
This is already complex enough that you might want to create a separate wrapper script containing these commands.
Without proper diagnostics of how your current script doesn't work, it's entirely possible that this fix alone is insufficient. Cron jobs do not only (or particularly) need absoute paths; the main differences compared to interactive shells is that cron jobs run with a different and more spare environment, where e.g. the shell's PATH, various library paths, environment variables etc can be different or missing altogether; and of course, no interactive facilities are available.
The system variables will hopefully be taken care of by your virtualenv; if it's correctly done, activating it will set up all the variables (PATH, PYTHONPATH, etc) your script needs. There could still be things like locale settings which are set up by your shell only when you log in interactively; but again, without details, let's just hope this isn't an issue for you.
The reason some people recommend absolute paths is that this will work regardless of your working directory. But a correctly written script should work fine in any directory; if it matters, the cron job will start in the owner's home directory. If you wanted to point to a relative path from there, this will work fine inside a cron job just as it does outside.
As an aside, you probably should not use subprocess.Popen() if one of the higher-level wrappers from the subprocess module do what you want. Unless compatibility with legacy Python versions is important, you should probably use subprocess.run() ... though running Python as a subprocess of Python is also often a useless oomplication. See also my answer to this related question.
I'm new to Python. This is my first Ansible module in order to delete the SimpleDB domain from ChaosMonkey deletion.
When tested in my local venv with my Mac OS X, it keeps saying
Module unable to decode valid JSON on stdin. Unable to figure out
what parameters were passed.
Here is the code:
#!/usr/bin/python
# Delete SimpleDB Domain
from ansible.module_utils.basic import *
import boto3
def delete_sdb_domain():
fields = dict(
sdb_domain_name=dict(required=True, type='str')
)
module = AnsibleModule(argument_spec=fields)
client = boto3.client('sdb')
response = client.delete_domain(DomainName='module.params['sdb_domain_name']')
module.exit_json(changed = False, meta = response)
def main():
delete_sdb_domain()
if __name__ == '__main__':
main()
And I'm trying to pass in parameters from this file: /tmp/args.json.
and run the following command to make the local test:
$ python ./delete_sdb_domain.py /tmp/args.json
please note I'm using venv test environment on my Mac.
If you find any syntax error in my module, please also point it out.
This is not how you should test your modules.
AnsibleModule expects to have specific JSON as stdin data.
So the closest thing you can try is:
python ./delete_sdb_domain.py < /tmp/args.json
But I bet you have your json file in wrong format (no ANSIBLE_MODULE_ARGS, etc.).
To debug your modules you can use test-module script from Ansible hacking pack:
./hacking/test-module -m delete_sdb_domain.py -a "sdb_domain_name=zzz"
I first posted an answer in this post, but it didn't conform to the forum standards. I hope this time te answer fits the forum standards. This code should be more clear and easy to read.
In Python 3+ I have the following class that I use to build a Windows Service (it does nothing, just writes a log file):
#MyWindowsService.py
import win32serviceutil
import servicemanager
import win32service
import win32event
import sys
import logging
import win32api
class MyWindowsService(win32serviceutil.ServiceFramework):
_svc_name_ = 'ServiceName'
_svc_display_name_ = 'Service Display Name'
_svc_description_ = 'Service Full Description'
logging.basicConfig(
filename = 'c:\\Temp\\{}.log'.format(_svc_name_),
level = logging.DEBUG,
format = '%(levelname)-7.7s # %(asctime)s: %(message)s'
)
def __init__(self, *args):
self.log('Initializing service {}'.format(self._svc_name_))
win32serviceutil.ServiceFramework.__init__(self, *args)
self.stop_event = win32event.CreateEvent(None, 0, 0, None)
def SvcDoRun(self):
self.ReportServiceStatus(win32service.SERVICE_START_PENDING)
try:
self.log('START: Service start')
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.start()
win32event.WaitForSingleObject(self.stop_event, win32event.INFINITE)
except Exception as e:
self.log('Exception: {}'.format(e))
self.SvcStop()
def SvcStop(self):
self.log('STOP: Service stopping...')
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.stop()
win32event.SetEvent(self.stop_event)
self.ReportServiceStatus(win32service.SERVICE_STOPPED)
def log(self, msg):
servicemanager.LogInfoMsg(str(msg)) #system log
logging.info(str(msg)) #text log
def start(self):
self.runflag = True
while self.runflag:
win32api.Sleep((2*1000), True)
self.log('Service alive')
def stop(self):
self.runflag = False
self.log('Stop received')
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(MyWindowsService)
In the script I use a log file to check if it's working properly. I'm running python3.6 (also tried with python3.4) on Windows 7 and I'm experiencing the following problem. When I run python MyWindowsService.py install the prompt says that the service has been installed (but nothing is written in the log file). If I try to start the service, I get Service Error: 1 - More info NET HELPMSG 3547 which doesn't say much about the error. If I run python MyWindowsService.py debug, the program runs just fine (the log file is written), but still I don't have any control over the service: if I open another prompt and try to stop/start the service I still got the same results as stated above.
I also tryed to insert some debug code inside the init function, and when I run python MyWindowsService.py install it seems it doesn't get called. Is it possible?
I've checked for multiple solution and workarounds around the net, but I didn't find anything suitable. What am I missing?
As pointed out by eriksun in the comment to the first post, the problem came from the location of the python script, that was in a drive mapped with an UNC path - I'm working with a virtual machine. Moving the python script in the same drive as the python installation did the job.
To sum it up for future uses, if the service fails to start and you're pretty sure about your code, these are helpful actions to try and solve your issues:
use sc start ServiceName, sc query ServiceName and sc stop ServiceName to get info about the service.
check if your file is in a physical drive or in a UNC-mapped drive. If the latter try to run the script using the UNC path (for example python \\Server\share\python\your-folder\script.py) or move your script in the same drive as the python installation
make sure that "python36.dll", "vcruntime140.dll", and "pywintypes36.dll" are either symlink'd to the directory that has PythonService.exe; or symlink'd to the System32 directory; or that the directories with these DLLs are in the system (not user) Path
Check the system register with command reg query HKLM\System\CurrentControlSet\Services\your_service_name /s to get more information about the script
PLease, feel free to complete, change, modify the last so that it can be usefull for anyone that like me encounder this issue.
EDIT: One more thing... My project was thought to work actually with network folders (and UNC-mapped drive) and it failed when I tried to make it run as service. One very useful (day-saving) resource that I used to make it work is the SysinternalsSuite by Mark Russinovich that I found in this post. Hope this helps.
How can I fetch the title of a screen session from the command line?
I came up with a very small and simple python script with pexpect to do it.
It is handy in multiuser environments where some host is reserved and status is written to screen title by user.
It works for me, feel free to make it better.
In order to fetch specific session title, you need to modify the script and call for correct session.
If you run this through remote connection as local script (through SSH for example), remember to set export TERM=xterm before execution.
try:
import pexpect
import sys
child=pexpect.spawn('screen -x')
child.sendcontrol('a');
child.send('A');
i = child.expect('Set window.*')
child.sendcontrol('c');
child.sendcontrol('a');
child.send('d');
TITLE=str(child.after)
TITLE_P=TITLE.split('7m')
if str(TITLE_P[-1]) == '':
print 'Title not found'
else:
print str(TITLE_P[-1])
except:
print 'Could not check screen Title'
I have a small Python web application using the Cherrypy framework. I am by no means an expert in web servers.
I got Cherrypy working with Apache using mod_python on our Ubuntu server. This time, however, I have to use Windows 2003 and IIS 6.0 to host my site.
The site runs perfectly as a stand alone server - I am just so lost when it comes to getting IIS running. I have spent the past day Googling and blindly trying any and everything to get this running.
I have all the various tools installed that websites have told me to (Python 2.6, CherrpyPy 3, ISAPI-WSGI, PyWin32) and have read all the documentation I can. This blog was the most helpful:
http://whatschrisdoing.com/blog/2008/07/10/turbogears-isapi-wsgi-iis/
But I am still lost as to what I need to run my site. I can't find any thorough examples or how-to's to even start with. I hope someone here can help!
Cheers.
I run CherryPy behind my IIS sites. There are several tricks to get it to work.
When running as the IIS Worker Process identity, you won't have the same permissions as you do when you run the site from your user process. Things will break. In particular, anything that wants to write to the file system will probably not work without some tweaking.
If you're using setuptools, you probably want to install your components with the -Z option (unzips all eggs).
Use win32traceutil to track down problems. Be sure that in your hook script that you're importing win32traceutil. Then, when you're attempting to access the web site, if anything goes wrong, make sure it gets printed to standard out, it'll get logged to the trace utility. Use 'python -m win32traceutil' to see the output from the trace.
It's important to understand the basic process to get an ISAPI application running. I suggest first getting a hello-world WSGI application running under ISAPI_WSGI. Here's an early version of a hook script I used to validate that I was getting CherryPy to work with my web server.
#!python
"""
Things to remember:
easy_install munges permissions on zip eggs.
anything that's installed in a user folder (i.e. setup develop) will probably not work.
There may still exist an issue with static files.
"""
import sys
import os
import isapi_wsgi
# change this to '/myapp' to have the site installed to only a virtual
# directory of the site.
site_root = '/'
if hasattr(sys, "isapidllhandle"):
import win32traceutil
appdir = os.path.dirname(__file__)
egg_cache = os.path.join(appdir, 'egg-tmp')
if not os.path.exists(egg_cache):
os.makedirs(egg_cache)
os.environ['PYTHON_EGG_CACHE'] = egg_cache
os.chdir(appdir)
import cherrypy
import traceback
class Root(object):
#cherrypy.expose
def index(self):
return 'Hai Werld'
def setup_application():
print "starting cherrypy application server"
#app_root = os.path.dirname(__file__)
#sys.path.append(app_root)
app = cherrypy.tree.mount(Root(), site_root)
print "successfully set up the application"
return app
def __ExtensionFactory__():
"The entry point for when the ISAPIDLL is triggered"
try:
# import the wsgi app creator
app = setup_application()
return isapi_wsgi.ISAPISimpleHandler(app)
except:
import traceback
traceback.print_exc()
f = open(os.path.join(appdir, 'critical error.txt'), 'w')
traceback.print_exc(file=f)
f.close()
def install_virtual_dir():
import isapi.install
params = isapi.install.ISAPIParameters()
# Setup the virtual directories - this is a list of directories our
# extension uses - in this case only 1.
# Each extension has a "script map" - this is the mapping of ISAPI
# extensions.
sm = [
isapi.install.ScriptMapParams(Extension="*", Flags=0)
]
vd = isapi.install.VirtualDirParameters(
Server="CherryPy Web Server",
Name=site_root,
Description = "CherryPy Application",
ScriptMaps = sm,
ScriptMapUpdate = "end",
)
params.VirtualDirs = [vd]
isapi.install.HandleCommandLine(params)
if __name__=='__main__':
# If run from the command-line, install ourselves.
install_virtual_dir()
This script does several things. It (a) acts as the installer, installing itself into IIS [install_virtual_dir], (b) contains the entry point when IIS loads the DLL [__ExtensionFactory__], and (c) it creates the CherryPy WSGI instance consumed by the ISAPI handler [setup_application].
If you place this in your \inetpub\cherrypy directory and run it, it will attempt to install itself to the root of your IIS web site named "CherryPy Web Server".
You're also welcome to take a look at my production web site code, which has refactored all of this into different modules.
OK, I got it working. Thanks to Jason and all his help. I needed to call
cherrypy.config.update({
'tools.sessions.on': True
})
return cherrypy.tree.mount(Root(), '/', config=path_to_config)
I had this in the config file under [/] but for some reason it did not like that. Now I can get my web app up and running - then I think I will try and work out why it needs that config update and doesn't like the config file I have...