I have a vanilla python that connects to a sqlite database.
Everything works fine until I try to run it as a daemon. Here is the code I'm using to do that:
def start(self):
if self.lockfile.is_locked():
exit_with_code(7, self.pid_file)
# If we're running in debug, run in the foreground, else daemonise
if self.options['debug']:
try:
self.main()
except KeyboardInterrupt:
pass
finally:
self.close_gracefully()
else:
context = daemon.DaemonContext(
files_preserve = [self.logger.socket(), self.lockfile]
)
context.signal_map = {
signal.SIGTERM: self.close_gracefully
}
with context: self.main()
I can run it in the foreground with python -m starter -debug and everything is fine, my app writes into the database, but when I leave the debug flag off I see the following when I try to write:
no such table: Frontends
I know that the frontends table exists because I've opened the database up. I assume that python is finding the database, because there would be an entirely different error message otherwise.
All my files are owned by vagrant, and ls -l shows the following:
-rwxrwxrwx 1 vagrant vagrant 9216 Nov 9 18:09 development.sqlite
Anyone got any tips?
Update
As requested, here is the code for my db
import os
import sqlite3
class Database(object):
def __init__(self, db_file='/vagrant/my_daemon/db/development.sqlite'):
self.db = sqlite3.connect(db_file)
if os.path.exists(db_file):
print "db exists"
And when I run this it prints "db exists". I instantiate the database in starter.py with a call to Database().
Python daemon closes all open file descriptors (except stdin, stout and stderr) when you daemonise.
I spent ages trying to figure out which files to keep open to prevent my database from being inaccessible, and in the end I found that it's easier to initialise the database inside the daemon context rather than outside. That way I don't need to worry about which files should stay open.
Now everything is working fine.
Related
I wanted to create a desktop launcher for my Python application. The application executes various ssh operations over pexpect with publickey-authentication. The problem is however, when I start my app with the .desktop launcher it doesn't work properly. The ssh connections ask for a password and don't use the publickeys. But it works fine via commandline execution.
The .desktop File looks like this:
[Desktop Entry]
Version=1.0
Name=SSH-Manager
Comment=XYZ
Exec=python /home/userx/SSH-Manager/startup.py
Icon=/home/userx/SSH-Manager/resources/icon.png
Path=/home/userx/repos/SSH-Manager
Terminal=true
Type=Application
Categories=Utility;Application;
StartupNotify=false
The desktop environment is KDE and the desktop user is the same as the commandline user.
Can someone explain why I get such strange behavior with the launcher?
Edit: Example function
def run(self):
self.a_signal.emit("Retrieving Data")
try:
session = pxssh()
session.force_password = False
hostname = self.client
username = "root"
session.login(hostname, username)
session.sendline("ls -a")
session.prompt()
session.logout()
except ExceptionPxssh as e:
print ("pxssh failed: ")
self.error_signal.emit("failed", str(e))
print e
return
self.process_output()
self.finish_signal.emit("done")
As Mirosław Zalewski suspected in the comments, the problem was the ssh-agent was not running for the desktop-environment because ssh-add was initially used in the /etc/sources. Executing ssh-add in the X-users ~./profile therefore solves the problem.
I first posted an answer in this post, but it didn't conform to the forum standards. I hope this time te answer fits the forum standards. This code should be more clear and easy to read.
In Python 3+ I have the following class that I use to build a Windows Service (it does nothing, just writes a log file):
#MyWindowsService.py
import win32serviceutil
import servicemanager
import win32service
import win32event
import sys
import logging
import win32api
class MyWindowsService(win32serviceutil.ServiceFramework):
_svc_name_ = 'ServiceName'
_svc_display_name_ = 'Service Display Name'
_svc_description_ = 'Service Full Description'
logging.basicConfig(
filename = 'c:\\Temp\\{}.log'.format(_svc_name_),
level = logging.DEBUG,
format = '%(levelname)-7.7s # %(asctime)s: %(message)s'
)
def __init__(self, *args):
self.log('Initializing service {}'.format(self._svc_name_))
win32serviceutil.ServiceFramework.__init__(self, *args)
self.stop_event = win32event.CreateEvent(None, 0, 0, None)
def SvcDoRun(self):
self.ReportServiceStatus(win32service.SERVICE_START_PENDING)
try:
self.log('START: Service start')
self.ReportServiceStatus(win32service.SERVICE_RUNNING)
self.start()
win32event.WaitForSingleObject(self.stop_event, win32event.INFINITE)
except Exception as e:
self.log('Exception: {}'.format(e))
self.SvcStop()
def SvcStop(self):
self.log('STOP: Service stopping...')
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
self.stop()
win32event.SetEvent(self.stop_event)
self.ReportServiceStatus(win32service.SERVICE_STOPPED)
def log(self, msg):
servicemanager.LogInfoMsg(str(msg)) #system log
logging.info(str(msg)) #text log
def start(self):
self.runflag = True
while self.runflag:
win32api.Sleep((2*1000), True)
self.log('Service alive')
def stop(self):
self.runflag = False
self.log('Stop received')
if __name__ == '__main__':
win32serviceutil.HandleCommandLine(MyWindowsService)
In the script I use a log file to check if it's working properly. I'm running python3.6 (also tried with python3.4) on Windows 7 and I'm experiencing the following problem. When I run python MyWindowsService.py install the prompt says that the service has been installed (but nothing is written in the log file). If I try to start the service, I get Service Error: 1 - More info NET HELPMSG 3547 which doesn't say much about the error. If I run python MyWindowsService.py debug, the program runs just fine (the log file is written), but still I don't have any control over the service: if I open another prompt and try to stop/start the service I still got the same results as stated above.
I also tryed to insert some debug code inside the init function, and when I run python MyWindowsService.py install it seems it doesn't get called. Is it possible?
I've checked for multiple solution and workarounds around the net, but I didn't find anything suitable. What am I missing?
As pointed out by eriksun in the comment to the first post, the problem came from the location of the python script, that was in a drive mapped with an UNC path - I'm working with a virtual machine. Moving the python script in the same drive as the python installation did the job.
To sum it up for future uses, if the service fails to start and you're pretty sure about your code, these are helpful actions to try and solve your issues:
use sc start ServiceName, sc query ServiceName and sc stop ServiceName to get info about the service.
check if your file is in a physical drive or in a UNC-mapped drive. If the latter try to run the script using the UNC path (for example python \\Server\share\python\your-folder\script.py) or move your script in the same drive as the python installation
make sure that "python36.dll", "vcruntime140.dll", and "pywintypes36.dll" are either symlink'd to the directory that has PythonService.exe; or symlink'd to the System32 directory; or that the directories with these DLLs are in the system (not user) Path
Check the system register with command reg query HKLM\System\CurrentControlSet\Services\your_service_name /s to get more information about the script
PLease, feel free to complete, change, modify the last so that it can be usefull for anyone that like me encounder this issue.
EDIT: One more thing... My project was thought to work actually with network folders (and UNC-mapped drive) and it failed when I tried to make it run as service. One very useful (day-saving) resource that I used to make it work is the SysinternalsSuite by Mark Russinovich that I found in this post. Hope this helps.
How can I fetch the title of a screen session from the command line?
I came up with a very small and simple python script with pexpect to do it.
It is handy in multiuser environments where some host is reserved and status is written to screen title by user.
It works for me, feel free to make it better.
In order to fetch specific session title, you need to modify the script and call for correct session.
If you run this through remote connection as local script (through SSH for example), remember to set export TERM=xterm before execution.
try:
import pexpect
import sys
child=pexpect.spawn('screen -x')
child.sendcontrol('a');
child.send('A');
i = child.expect('Set window.*')
child.sendcontrol('c');
child.sendcontrol('a');
child.send('d');
TITLE=str(child.after)
TITLE_P=TITLE.split('7m')
if str(TITLE_P[-1]) == '':
print 'Title not found'
else:
print str(TITLE_P[-1])
except:
print 'Could not check screen Title'
import logging, logging.handlers
def main():
ntl = logging.handlers.NTEventLogHandler("Python Logging Test")
logger = logging.getLogger("")
logger.setLevel(logging.DEBUG)
logger.addHandler(ntl)
logger.error("This is a '%s' message", "Error")
if __name__ == "__main__":
main()
The Python (2.7.x) script above writes "This is a 'Error' message" to the Windows Event Viewer. When I run it as a script, I get the expected output. If I convert the script to an executable via PyInstaller, I get an entry in the event log but it says something completely different.
The description for Event ID ( 1 ) in Source ( Python Logging Test ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: This is a 'Error' message.
This is the command I use to convert the script into an executable: pyinstaller.py --onefile --noconsole my_script.py though the command line parameters do not appear to have any impact on this behaviour and it will suffice to just call pyinstaller.py my_script.py.
I would appreciate any help in understanding what is going on and how I go about fixing this.
Final solution
I didn't want to go down the resource hacker route, as that is going to be a difficult step to automate. Instead, the approach I took was to grab the win32service.pyd file from c:\Python27\Lib\site-packages\win32 and place it next to my executable. The script was then modified pass the full path to the copy of the win32service.pyd file and this works in both script and exe form. The final script is included below:
import logging, logging.handlers
import os
import sys
def main():
base_dir = os.path.dirname(sys.argv[0])
dllname = os.path.join(base_dir, "win32service.pyd")
ntl = logging.handlers.NTEventLogHandler("Python Logging Test", dllname=dllname)
logger = logging.getLogger("")
logger.setLevel(logging.DEBUG)
logger.addHandler(ntl)
logger.error("This is a '%s' message", "Error")
if __name__ == "__main__":
main()
Usually, the Windows Event Log doesn't store error messages in plain text, but rather message ID references and insertion strings.
Instead of storing a message like Service foo crashed unexpectedly, it stores a message ID which points to a resource string stored in a DLL. In this case, the resource would be something like Service %s crashed unexpectedly and foo would be stored as insertion string. The program which writes the message registers the resource DLL.
The reason for this is localization. DLLs can store lots of different resources (dialog layout, strings, icons…), and one DLL can contain the same resource in many different languages. The operating system automatically chooses the right resources depending on the system locale. Resource DLLs are used by virtually all Microsoft utilities and core utilities.
Side note: Nowadays, the preferred (and cross-platform) way for localization is gettext.
This is used for the message log as well – ideally, you could open a log from an German Windows installation on an English one with all messages in English.
I suspect that the pywin32 implementation skips that mechanism by only having one single message ID (1) which is just something like "%s". It is stored in win32service.pyd and registered by pywin32. This works fine as long as this file exists on the file system, but breaks as soon as it's hidden inside a PyInstaller executable. I guess you have to embed the message ID into your executable directly.
Edit: suspicion confirmed, the message table is indeed stored inside win32service.pyd
Resource Hacker showing the message table http://media.leoluk.de/evlog_rh.png
Try to copy the message table resource from win32service.pyd to your PyInstaller executable (for example using Resource Hacker).
Looking at the logging handler implementation, this might work:
def __init__(self, appname, dllname=None, logtype="Application"):
logging.Handler.__init__(self)
try:
import win32evtlogutil, win32evtlog
self.appname = appname
self._welu = win32evtlogutil
if not dllname:
dllname = os.path.split(self._welu.__file__)
dllname = os.path.split(dllname[0])
dllname = os.path.join(dllname[0], r'win32service.pyd')
You'd have to set dllname to os.path.dirname(__file__). Use something like this if you want it to continue working for the unfrozen script:
if getattr(sys, 'frozen', False):
dllname = None
elif __file__:
dllname = os.path.dirname(__file__)
ntl = logging.handlers.NTEventLogHandler("Python Logging Test", dllname=dllname)
I have Django application which needs to call psql. I do this in a celery thread which looks like this:
#task()
def insert_sqldump_threaded(username, database, file):
host = database.server.db_address
work = subprocess.Popen([settings.PSQL,
"-f%s" % file,
"-d%s" % database.db_name,
"-h%s" % host,
"-U%s" % settings.DB_ADMIN_USER
], env = {'PGPASSFILE': settings.DB_PASSFILE}
)
work.wait()
return work.returncode
On my development server the PGPASSFILE looks like this:
localhost:5432:*:postgres:postgres
which should be fine.
The problem is that all I get when this function gets called is an error from psql:
psql: could not translate host name "localhost" to address: Unknown server error
And now it gets really strange, but when I don't submit the "env" variable, psql seems to recognize the host. At least then it asks for a password.
Any ideas on how to solve this?
I think postgresql needs other environment variables that you clear when you pass env. You can simply change os.environ or make a copy of it beforehand as in the following code:
import os
#task()
def insert_sqldump_threaded(username, database, file):
d = dict(os.environ)
d['PGPASSFILE'] = settings.DB_PASSFILE
host = database.server.db_address
work = subprocess.Popen([settings.PSQL,
"-f%s" % file,
"-d%s" % database.db_name,
"-h%s" % host,
"-U%s" % settings.DB_ADMIN_USER
], env = d
)
work.wait()
return work.returncode
When you don't submit env, it picks up the environment variables from the shell you're running in - see os.environ. It must be depending on one of those for looking up localhost. You'll need to include that in the dictionary. Or just copy in everything from os.environ.