We Keep Coding

Routing all paths to one handler function at Falcon Framework

I want to routes all paths that start with something like “/api” to same handler function.
Such as:
/api/foo
/api/bar
/api/foo/bar
/api/bar/baz/this/that
All should be handled with one function and I should be able to get the full path after /api.
This feature is very handy and I used it often at Node.js Express framework. Now I am looking for ways to accomplish the same thing with Python Falcon framework.
More info can be found here; It defines the feature as "white-listed “global” functionality."
http://expressjs.com/api.html#app.all

Perhaps you are looking for Falcon's sink facility, e.g.:
class Sink(object):
def on_get(self, req, resp):
resp.body = ('\nTwo things awe me most, the starry sky '
'above me and the moral law within me.\n'
'\n'
' ~ Immanuel Kant\n\n')
app = falcon.API()
handler = Sink().on_get
app.add_sink(handler, prefix='/')
This will route all URLs to the sink handler.

If you're looking for a way to handle all requests before they get routed to appropriate resources, I suggest you to look into middleware components.

Resolve Discovery path on App Engine Module

I want to build a client for my cloud endpoints in python like described in the Documentation.
I want to build the api from a Managed VM, so i get the path to the API by calling
modules.get_hostname(module="default")
This works fine for the devserver and i can create the complete path to the discovery endpoint, however on the live system this returns the url to a certain version like:
20150628t110011.default.guestbook.appspot.com
Thus the complete path to the API (default module) would be
https://20150628t110011.default.guestbook.appspot.com/_ah/api/discovery/v1/apis/guestbook/v1/rest?userIp=182.177.0.4"
But there is no discovery document, maybe due to the fact, that the certificate does not match a url that long and the https fails.
Is there a proper way to receive the base url to the default module? like so:
default.guestbook.appspot.com
because that would result in a working discovery endpoint:
https://default.guestbook.appspot.com/_ah/api/discovery/v1/apis/guestbook/v1/rest?userIp=182.177.0.4"
I would like to avoid doing string operations here, because on the local devserver this would not work as the module url resolves to something like localhost:1234.

You probably want to go through the GAE URl routing doc: https://cloud.google.com/appengine/docs/python/modules/routing#routing_via_url
Key points in there:
Google does not issue SSL certificates for double-wildcard domains
hosted at appspot.com, the certificate won't work for https://20150628t110011.default.guestbook.appspot.com
You can get the certificate to work using the -dot- delimiters; in particular the default version of the default
module can be accessed directly at guestbook.appspot.com
The problem gets even more complicated if your app has multiple modules and also if it's mapped to a custom domain.
While trying to address such complications I realized that modules.get_hostname() is nowadays no longer able to perform the original function that its name implies (I guess because of the multiple possible paths for accessing the same entity). Which probably explains why they won't attempt to fix the api to return a proper hostname: (see this Q&A)
But the info it can return (as applicable depending on the invocation arguments and the execution context) is IMHO extremely useful, allowing one to programatically obtain proper hostnames/URLs for all 3 possible app usage contextes: on the development server, on the .appspot.com domain and on custom domains mapping (including with hostname-based mapping):
<instance_id>.<module_version>.<module_name>.<app_name>.(appspot.com|<devserver_hostname>:<port#>)
This would be, for example, my approach for an app not interested in anything below the module name and using hostname-based custom domain dispatch routing - modules mapped to different hostnames):
def get_module_url(self, module_name='default'):
host_name = modules.get_hostname(module=module_name)
if os.environ.get('SERVER_SOFTWARE').startswith('Development'):
return 'http://' + host_name
app_name = app_identity.get_application_id()
split_name = self.request.host.split(':')[0].split('.')
if split_name[-2] == 'appspot':
new_host_name = app_name if module_name == 'default' else module_name + '-dot-' + app_name
else:
# custom hostname-based domain mapping, default module goes to `www`.mydomain.com
new_host_name = 'www' if module_name == 'default' else module_name
if app_name.endswith('-staging'):
# copy of the GAE app for staging purposes on mydomain.com
new_host_name += '-staging'
return '.'.join(['https://' + new_host_name] + split_name[1:])

As per this thread, unfortunately manual conversion is required to convert from the . hostname to -dot-.

In the Pyramid web framework, how do I source sensitive settings into development.ini / production.ini from an external file?

I'd like to keep development.ini and production.ini under version control, but for security reason would not want the sqlalchemy.url connection string to be stored, as this would contain the username and password used for the database connection.
What's the canonical way, in Pyramid, of sourcing this setting from an additional external file?
Edit
In addition to solution using the environment variable, I came up with this solution after asking around on #pyramid:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
# Read db password from config file outside of version control
secret_cfg = ConfigParser()
secret_cfg.read(settings['secrets'])
dbpass = secret_cfg.get("secrets", "dbpass")
settings['sqlalchemy.url'] = settings['connstr'] % (dbpass,)

I looked into this a lot and played with a lot of different approaches. However, Pyramid is so flexible, and the .ini config parser is so minimal in what it does for you, that there doesn't seem to be a de facto answer.
In my scenario, I tried having a production.example.ini in version control at first that got copied on the production server with the details filled in, but this got hairy, as updates to the example didn't get translated to the copy, and so the copy had to be re-created any time a change was made. Also, I started using Heroku, so files not in version control never made it into the deployment.
Then, there's the encrypted config approach. Which, I don't like the paradigm. Imagine a sysadmin being responsible for maintaining the production environment, but he or she is unable to change the location of a database or environment-specific setting without running it back through version control. It's really nice to have the separation between environment and code as much as possible so those changes can be made on the fly without version control revisions.
My ultimate solution was to have some values that looked like this:
[app:main]
sqlalchemy.url = ${SQLALCHEMY_URL}
Then, on the production server, I would set the environment variable SQLALCHEMY_URL to point to the database. This even allowed me to use the same configuration file for staging and production, which is nice.
In my Pyramid init, I just expanded the environment variable value using os.path.expandvars:
sqlalchemy_url = os.path.expandvars(settings.get('sqlalchemy.url'))
engine = create_engine(sqlalchemy_url)
And, if you want to get fancy with it and automatically replace all the environment variables in your settings dictionary, I made this little helper method for my projects:
def expandvars_dict(settings):
"""Expands all environment variables in a settings dictionary."""
return dict((key, os.path.expandvars(value)) for
key, value in settings.iteritems())
Use it like this in your main app entry point:
settings = expandvars_dict(settings)

The whole point of the separate ini files in Pyramid is that you do not have to version control all of them and that they can contain different settings for different scenarios (development/production/testing). Your production.ini almost always should not be in the same VCS as your source code.

I found this way for loading secrets from a extra configuration and from the env.
from pyramid.config import Configurator
from paste.deploy import appconfig
from os import path
__all__ = [ "main" ]
def _load_secrets(global_config, settings):
""" Helper to load secrets from a secrets config and
from env (in that order).
"""
if "drawstack.secrets" in settings:
secrets_config = appconfig('config:' + settings["drawstack.secrets"],
relative_to=path.dirname(global_config['__file__']))
for k, v in secrets_config.items():
if k == "here" or k == "__file__":
continue
settings[k] = v
if "ENV_DB_URL" in global_config:
settings["sqlalchemy.url"] = global_config["ENV_DB_URL"]
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
_load_secrets(global_config, settings)
config = Configurator(settings=settings)
config.include('pyramid_jinja2')
config.include('.models')
config.include('.routes')
config.scan()
return config.make_wsgi_app()
The code above, will load any variables from the value of the config key drawstack.secrets and after that it tries to load DB_URL from the enviornment.
drawstack.secrets can be relative to the original config file OR absolute.

Running scripts within Pyramid framework (ie without a server)

I have a fair bit of experience with PHP frameworks and Python for scripting so am now taking the step to Pyramid.
I'd like to know what is the 'correct' way to run a script in Pyramid. That is, how should I set it up so that it is part of the application and has access to config and thus database but does not run through paster (or whatever WSGI).
As an example, say I have a web application which while a user is offline grabs Facebook updates through a web service. I want to write a script to poll that service and store in the database ready for next login.
How should I do this in terms of:
Adding variables in the ini file
Starting the script correctly
I understand the basics of Python modules and packages; however I don't fully understand Configurator/Paster/package setup, wherein I suspect the answer lies.
Thanks
Update:
Thanks, this seems along the lines of what I am looking for. I note that you have to follow a certain structure (eg have summary and parser attributes set) and that the function called command() will always be run. My test code now looks something like this:
class AwesomeCommand(Command):
max_args = 2
min_args = 2
usage = "NAME"
# These are required
summary = "Say hello!"
group_name = "My Package Name"
# Required:
parser = Command.standard_parser(verbose=True)
def command(self):
# Load the config file/section
config_file, section_name = self.args
# What next?
I'm now stuck as to how to get the settings themselves. For example, in init.py you can do this:
engine = engine_from_config(settings, 'sqlalchemy.')
What do I need to do to transform the config file into the settings?
EDIT: The (simpler) way to do this in Pylons is here:
Run Pylons controller as separate app?

As of Pyramid 1.1, this is handled by the framework:
http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/commandline.html#writing-a-script

paster starts an application given an ini file that describes that application. the "serve" command is a built in command for starting a wsgi application and serving it. BUT, you can write other commands.
from paste.script.command import Command
class AwesomeCommand(Command):
def command(self):
print "the awesome thing it does"
and then register them as entry points in your setup.py.
setup(...
entry_points="""
[paste.app_factory]
.....
[paste.global_paster_command]
myawesome-command = mypackage.path.to.command:AwesomeCommand """)
pyramid adds it's own commands this way like the pshell command.

After going to the pylons discuss list, I came up with an answer. Hope this helps somebody:
#Bring in pyramid application--------------------------------------
import pyramid
from paste.deploy import appconfig
config_file = '/path_to_config_file/configname.ini'
name = 'app_name'
config_name = 'config:%s' % config_file
here_dir = os.getcwd()
conf = appconfig(config_name, name, relative_to=here_dir)
from main_package import main
app = main(conf.global_conf, **conf.local_conf)
#--------------------------------------------------------------------------

you need to make view for that action and then run it using:
paster request development.ini /url_to_your_view

django.core.urlresolvers.resolve incorrect behavior under apache non-root deployment

When a django app is deployed under a non-root apache url (with WsgiScriptAlias /suburl /path_to_django.wsgi) the {%url%} tag and the django.core.urlresolvers.reverse function take into account the SCRIPT_NAME variable and return urls of the form /suburl/path_to_my_view
However, when I use the django.core.urlresolvers.resolve function to resolve those urls it throws an error. That forces me to strip the SCRIPT_NAME of the generated urls before calling resolve. Is this the expected behavior or am I misunderstanding everything?
Regards.

I got the same kind of problem:
SCRIPT_NAME defined in my apache config
a call to django.core.urlresolvers.reverse outside the wsgi didnt prepend the prefix
in any view or resource, a call to the same method prepended the prefix
I managed to have the prefix automatically prepended using the next lines of code:
# in settings.py
from django.core.urlresolvers import set_script_prefix
...
FORCE_SCRIPT_NAME = "your-prefix"
set_script_prefix(FORCE_SCRIPT_NAME)
...
The first line makes sure your wsgi uses your prefix every time. The second one sets the prefix, so that reverse will find it.
Please note that you need to have the same prefix in your apache conf. It's a bit redundant, but the cleaner fix I found.