How to parse PostgreSQL configuration file in python? - python

Python has ConfigParser library for parsing mysql configuration files.
https://docs.python.org/2/library/configparser.html
But i need to parse configuration files of PostgreSQL as well. But i couldn't find any parser for them.
In PostgreSQL there is no section or heading, so it can't be done by configparser.
Example:
# The default values of these variables are driven from the -D command-line
# option or PGDATA environment variable, represented here as ConfigDir.
data_directory = '/data/postgresql' # use data in another directory
# (change requires restart)
hba_file = '/etc/postgresql/9.1/main/pg_hba.conf' # host-based authentication file
# (change requires restart)
ident_file = '/etc/postgresql/9.1/main/pg_ident.conf' # ident configuration file
# (change requires restart)
# If external_pid_file is not explicitly set, no extra PID file is written.
external_pid_file = '/var/run/postgresql/9.1-main.pid' # write an extra PID file
# (change requires restart)
I was using the below code for mysql and it's working fine but not for postgresql.conf file.
import ConfigParser
class MysqlParserModule:
def __init__(self):
print 'Mysql Parser Module'
def parse_option(self, conf_file, section, option):
try:
config = ConfigParser.ConfigParser()
config.read(conf_file)
result = config.get(section, option)
print "'%s' value is '%s' under '%s' section" % (option, result, section)
except Exception as ex:
print 'in parse_option exception'
print ex
def test(self):
test = MysqlParserModule()
conf_file = '/root/Desktop/db_config/my_new.cnf'
section = 'client'
option = 'port'
test.parse_option(conf_file, section, option)
if __name__ == '__main__':
test = MysqlParserModule()
test.test()

Related

os.system argument from config file

I made this activity and it works. I need to have config file with USB/VID/PID.
def resetactivity():
os.system(r'"devcon.exe restart "*USB\VID_04E8&PID_3321*"')
I try to do this with config parser. I made config.txt:
[My Section]
usbdev = r'"devcon.exe restart "*USB\VID_04E8&PID_3321*"'
I read my config file in Python:
config = configparser.ConfigParser()
config.read('config.txt')
usbdev = config.get('My Section', 'usbdev')
And when I am trying to use this in os.system command like this:
def resetactivity():
os.system(usbdev)
I get this result:
The filename, directory name, or volume label syntax is incorrect.
'PID_3321*"'' is not recognized as an internal or external command,
operable program or batch file.
Try this code
import configparser
import os
def resetactivity():
config = configparser.ConfigParser()
config.read('config.txt')
usbdev = config.get('My Section', 'usbdev')
print(usbdev)
os.system(usbdev)
if __name__ == "__main__":
resetactivity()
With config.txt formatted as
[My Section]
usbdev = devcon.exe restart "USB\VID_04E8&PID_3321"

update particular option in section in ".ini" file using ConfigParser

I am trying to update a particular option in my config.ini file using ConfigParser.
This is my config.ini file:
[remote-server-details]
MACHINE_NAME =
MACHINE_USERNAME =
MACHINE_PASSWORD = Welcome!23
And this is my step file in Python:
#step('I change the following config options in section "{section_name}" in the {config_file_type} config file')
def set_dag_config_section_values(context, section_name, config_file_type):
parameters = ast.literal_eval(context.text)
context.conf[config_file_type].read(context.dag_config_file[config_file_type])
context.conf[config_file_type].remove_section(section_name)
context.conf[config_file_type].add_section(section_name)
for option, value in parameters.iteritems():
context.conf[config_file_type].set(section_name, option, value)
with open(context.dag_config_file[config_file_type], 'w+') as configfile:
context.conf[config_file_type].write(configfile)
context.conf[config_file_type].read(context.dag_config_file[config_file_type])
readin_parameters = dict(context.conf[config_file_type].items(section_name))
sort_dict(parameters)
sort_dict(readin_parameters)
assert readin_parameters == parameters, 'Value could not be set. %s does not match config contents %s' % (parameters, readin_parameters)
I need to update only MACHINE_NAME and MACHINE_USERNAME, as MACHINE_PASSWORD is already contained in the config file, but I'm getting this error:
INFO - Subtask: remote_server_details['MACHINE_PASSWORD'],
[2018-08-17 17:40:04,723] {{base_task_runner.py:98}} INFO - Subtask: KeyError: 'MACHINE_PASSWORD'

App Engine Unit Testing: ImportError: Start directory is not importable

I'm trying to learn unit testing with Google App Engine by using the exact code they put on the Local Unit Testing for Python page (https://cloud.google.com/appengine/docs/python/tools/localunittesting). I can't figure out this error, though:
ImportError: Start directory is not importable: 'testmem.py'
I'm just using their simple testing framework as testrunner.py and their Datastore and Memcache tests in a file called testmem.py. I call the test from the project root directory as:
<me>$ python testrunner.py ~/google_appengine testmem.py
this matches the usage as far as I can tell: %prog SDK_PATH TEST_PATH.
My file structure is:
__init__.py
app.yaml
testrunner.py
testmem.py
helloworld.py
Can anyone tell me what I'm doing wrong here? Thanks in advance.
Complete error message:
Traceback (most recent call last):
File "testrunner.py", line 30, in <module>
main(SDK_PATH, TEST_PATH)
File "testrunner.py", line 17, in main
suite = unittest.loader.TestLoader().discover(test_path)
File "/usr/lib/python2.7/unittest/loader.py", line 204, in discover
raise ImportError('Start directory is not importable: %r' % start_dir)
ImportError: Start directory is not importable: 'testmem.py'
testrunner.py:
#!/usr/bin/python
import optparse
import sys
import unittest
USAGE = """%prog SDK_PATH TEST_PATH
Run unit tests for App Engine apps.
SDK_PATH Path to the SDK installation
TEST_PATH Path to package containing test modules"""
def main(sdk_path, test_path):
sys.path.insert(0, sdk_path)
import dev_appserver
dev_appserver.fix_sys_path()
suite = unittest.loader.TestLoader().discover(test_path)
unittest.TextTestRunner(verbosity=2).run(suite)
if __name__ == '__main__':
parser = optparse.OptionParser(USAGE)
options, args = parser.parse_args()
if len(args) != 2:
print 'Error: Exactly 2 arguments required.'
parser.print_help()
sys.exit(1)
SDK_PATH = args[0]
TEST_PATH = args[1]
main(SDK_PATH, TEST_PATH)
testmem.py:
import unittest
from google.appengine.api import memcache
from google.appengine.ext import db
from google.appengine.ext import testbed
class TestModel(db.Model):
"""A model class used for testing."""
number = db.IntegerProperty(default=42)
text = db.StringProperty()
class TestEntityGroupRoot(db.Model):
"""Entity group root"""
pass
def GetEntityViaMemcache(entity_key):
"""Get entity from memcache if available, from datastore if not."""
entity = memcache.get(entity_key) # #UndefinedVariable
if entity is not None:
return entity
entity = TestModel.get(entity_key)
if entity is not None:
memcache.set(entity_key, entity) # #UndefinedVariable
return entity
class DemoTestCase(unittest.TestCase):
def setUp(self):
# First, create an instance of the Testbed class.
self.testbed = testbed.Testbed()
# Then activate the testbed, which prepares the service stubs for use.
self.testbed.activate()
# Next, declare which service stubs you want to use.
self.testbed.init_datastore_v3_stub()
self.testbed.init_memcache_stub()
def tearDown(self):
self.testbed.deactivate()
def testInsertEntity(self):
TestModel().put()
self.assertEqual(1, len(TestModel.all().fetch(2)))
def testFilterByNumber(self):
root = TestEntityGroupRoot(key_name="root")
TestModel(parent=root.key()).put()
TestModel(number=17, parent=root.key()).put()
query = TestModel.all().ancestor(root.key()).filter('number =', 42)
results = query.fetch(2)
self.assertEqual(1, len(results))
self.assertEqual(42, results[0].number)
def testGetEntityViaMemcache(self):
entity_key = str(TestModel(number=18).put())
retrieved_entity = GetEntityViaMemcache(entity_key)
self.assertNotEqual(None, retrieved_entity)
self.assertEqual(18, retrieved_entity.number)
if __name__ == '__main__':
unittest.main()
Here, "test path" should be a directory that contains the tests you want to run, not the path to a single module. Try just using . as the directory (assuming you're running it from the top-level project/app directory) and see if that helps.
My team keeps our tests in a separate directory from our source code, so we would use a path to our test directory as that second argument.
Make sure the tests directory is spelled "tests" and not "test". That's the problem I had with this.

Optional parameter not working on schema

I am adding validation using schema for CLI that uses docopt, but I cannot seem to get optional to work. I want to validate that:
the input file exists
valid options are used
if the PATH is added that the directory exists.
Here is app so far
"""DVget
Usage:
DVget [-s] FILE [PATH]
Process a file, return data based on selection
and write results to PATH/output-file
Arguments:
FILE specify input file
PATH specify output directory (default: ./)
Options:
-s returns sections
-p returns name-sets
-m returns modules
"""
import os
from docopt import docopt
from schema import Schema, And, Use, Optional, SchemaError
# START OF SCRIPT
if __name__ == "__main__":
arguments = docopt(__doc__, version="0.1")
#print(arguments)
schema = Schema({
'FILE': [Use(open, error='FILE should be readable')],
Optional('PATH'): And(os.path.exists, error='PATH should exist'),
'-': And(str, lambda s: s in ('s', 'p', 'm'))})
try:
arguments = schema.validate(arguments)
# process(arguments)
except SchemaError as e:
exit(e)
running DVget -s "c:\test.txt" gives me the error message 'PATH should exist' even when using Optional in schema and docopt. Any suggestions?

How to properly install isapi_wsgi on IIS for Python2.7?

I've worked through installing Python as a CGI application on IIS on Windows 7. This is pretty straightforward, but I'd like to use the WSGI stuff, for better flexibility.
I downloaded the archive for isapi_wsgi, unzipped it, and then ran the install as per the instructions, like this:
\python27\python.exe setup.py install
This succeeded:
Then I coded a .py module that had the wsgi glue in it, and tried installing it. This failed like so:
It's a COM Moniker error, and I know that the IIS6-compatible management stuff is based on COM Monikers, which reminded me that there is a pre-req for isapi_wsgi of the IIS6-compatible management stuff. I ran \windows\system32\OptionalFeatures.exe and installed that, then re-ran the .py module and it installed correctly.
C:\dev\wsgi>\Python27\python.exe app1_wsgi.py
Configured Virtual Directory: /wsgi
Installation complete.
Ok, wonderful. Now when I look in the current directory, I see a new DLL named _app1_wsgi.dll, and when I look in IIS Manager I can see a new IIS vdir, and a scriptmap within that vdir for '*', which is mapped to the _app1_wsgi.DLL. All good. But! making a request to http://localhost/wsgi gives me a 500 error.
Through some trial-and-error I see that the .py module that defines my handlers must be in the site-packages directory. I am very surprised by this.
Can I avoid this? Can I simply put the .py module in the same directory as the generated .dll file? Or do I need to deploy all of my python logic to site-packages in order to run it from the WSGI mechanism?
The answer is:
the installation of isapi_wsgi as described in the question, is correct.
with the basic boilerplate of app.py as shown in the example code accompanying isapi_wsgi, the python classes for the web app need to be in the site-packages directory.
it is possible to allow the python source modules to reside in the same directory as with the generated *.dll file, but it requires some special handling in the *wsgi.py file.
a better way to run python on Windows for development purposes is to simply download the Google App Engine and use the builtin dedicated http server. The framework that comes with the GAE SDK handles reloading and allows the .py modules to be placed in particular directories.
If you don't want to download and install the GAE SDK, then you might try the following. Using this code, when a request arrives on isapi_wsgi, the handler looks in the home directory for a py module, and loads it. If the module is already loaded, it checks the file "last modified time" and reloads the module if the last mod time is later than the time from the prior load. It works for simplistic cases but I suppose it will be brittle when there are nested module dependencies.
import sys
import os
import win32file
from win32con import *
# dictionary of [mtime, module] tuple; uses file path as key
loadedPages = {}
def request_handler(env, start_response):
'''Demo app from wsgiref'''
cr = lambda s='': s + '\n'
if hasattr(sys, "isapidllhandle"):
h = None
# get the path of the ISAPI Extension DLL
hDll = getattr(sys, "isapidllhandle", None)
import win32api
dllName = win32api.GetModuleFileName(hDll)
p1 = repr(dllName).split('?\\\\')
p2 = p1[1].split('\\\\')
sep = '\\'
homedir = sep.join(p2[:-1])
# the name of the Python module is in the PATH_INFO
moduleToImport = env['PATH_INFO'].split('/')[1]
pyFile = homedir + sep + moduleToImport + '.py'
fd = None
try:
fd = win32file.CreateFile(pyFile, GENERIC_READ, FILE_SHARE_DELETE, None, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0)
except Exception as exc1:
fd = None
if fd is not None:
# file exists, get mtime
fd.close()
mt = os.path.getmtime(pyFile)
else:
mt = None
if mt is not None:
h = None
if not pyFile in loadedPages:
# need a new import
if homedir not in sys.path:
sys.path.insert(0, homedir)
h = __import__(moduleToImport, globals(), locals(), [])
# remember
loadedPages[pyFile] = [mt, h]
else:
# retrieve handle to module
h = loadedPages[pyFile][1]
if mt != loadedPages[pyFile][0]:
# need to reload the page
reload(h)
loadedPages[pyFile][0] = mt
if h is not None:
if 'handler' in h.__dict__:
for x in h.handler(env, start_response):
yield x
else:
start_response("400 Bad Request", [('Content-Type', 'text/html')])
else:
start_response("404 Not Found", [('Content-Type', 'text/html')])
yield cr()
yield cr("<html><head><title>Module not found</title>" \
"</head><body>")
yield cr("<h3>404 Not Found</h3>")
yield cr("<h3>No handle</h3></body></html>")
else:
start_response("404 Not Found", [('Content-Type', 'text/html')])
yield cr()
yield cr("<html><head><title>Module not found</title>" \
"</head><body>")
yield cr("<h3>404 Not Found</h3>")
yield cr("<h3>That module (" + moduleToImport + ") was not found.</h3></body></html>")
else:
start_response("500 Internal Server Error", [('Content-Type', 'text/html')])
yield cr()
yield cr("<html><head><title>Server Error</title>" \
"</head><body><h1>Server Error - No ISAPI Found</h1></body></html>")
# def test(environ, start_response):
# '''Simple app as per PEP 333'''
# status = '200 OK'
# start_response(status, [('Content-type', 'text/plain')])
# return ['Hello world from isapi!']
import isapi_wsgi
# The entry point(s) for the ISAPI extension.
def __ExtensionFactory__():
return isapi_wsgi.ISAPISimpleHandler(request_handler)
def PostInstall(params, options):
print "The Extension has been installed"
# Handler for our custom 'status' argument.
def status_handler(options, log, arg):
"Query the status of the ISAPI?"
print "Everything seems to be fine..."
if __name__=='__main__':
# This logic gets invoked when the script is run from the command-line.
# In that case, it installs this module as an ISAPI.
#
# The API provided by isapi_wsgi for this is a bit confusing. There
# is an ISAPIParameters object. Within that object there is a
# VirtualDirs property, which itself is a list of
# VirtualDirParameters objects, one per vdir. Each vdir has a set
# of scriptmaps, usually this set of script maps will be a wildcard
# (*) so that all URLs in the vdir will be served through the ISAPI.
#
# To configure a single vdir to serve Python scripts through an
# ISAPI, create a scriptmap, and stuff it into the
# VirtualDirParameters object. Specify the vdir path and other
# things in the VirtualDirParameters object. Stuff that vdp object
# into a sequence and set it into the ISAPIParameters thing, then
# call the vaguely named "HandleCommandLine" function, passing that
# ISAPIParameters thing.
#
# Clear as mud?
#
# Seriously, this thing could be so much simpler, if it had
# reasonable defaults and a reasonable model, but I guess it will
# work as is.
from isapi.install import *
# Setup the virtual directories -
# To serve from root, set Name="/"
sm = [ ScriptMapParams(Extension="*", Flags=0) ]
vdp = VirtualDirParameters(Name="wsgi", # name of vdir/IIS app
Description = "ISAPI-WSGI Demo",
ScriptMaps = sm,
ScriptMapUpdate = "replace"
)
params = ISAPIParameters(PostInstall = PostInstall)
params.VirtualDirs = [vdp]
cah = {"status": status_handler}
# from isapi.install, part of pywin32
HandleCommandLine(params, custom_arg_handlers = cah)
Using this model, requesting http://foo/wsgi/bar will try loading bar.py from the home directory with the WSGI .dll file. If bar.py cannot be found, you get a 404. If bar.py has been updated since the last run, it reloads. If bar cannot be loaded, you get a 500.
bar.py must export a method called handler, publicly. That method must be a generator. like so:
import time
def handler(env, start_response):
start_response("200 OK", [('Content-Type', 'text/html')])
cr = lambda s='': s + '\n'
yield cr("<html><head><title>Hello world!</title></head><body>")
yield cr("<h1>Bargle Bargle Bargle</h1>")
yield cr("<p>From the handler...</p>")
yield cr("<p>(bargle)</p>")
yield cr("<p>The time is now: " + time.asctime() + " </p>")
yield cr("</body></html>")
__all__ = ['handler']
But as I said, I think GAE is probably a better way to develop Python webapps using Windows.
put this on top of your scrip:
import site
site.addsitedir('path/to/your/site-packages')
the same problem you had, was solved with this two lines

Categories