We Keep Coding

Web2Py Production - making redirections on default index with parameters (maybe with nginx)

I am trying to create redirects using web2py from effectively the default index page (or just the route of the domain/package).
Some keywords (such as 'about', stored in a list) wouldn't redirect. However, all not in that list would redirect.
The desired behaviour is:
https://startbean.com/about -> No redirect
https://startbean.com/myc -> https://startbean.com/company/myc
The default page that is shown at startbean.com is from the package 'default' and is called 'index'. If the redirect was as in the below, it would be easy:
https://startbean.com/default/about -> No redirect
https://startbean.com/default/index/myc -> https://startbean.com/default/company/myc
because the myc is a URL argument. But when it is from the root, Web2Py tries to open a package called 'myc' and then finds no page (index or controller function) so errors.
What is the best way of handling this? I was trying with routes.py, but couldn't figure out a way to do this (am pretty sure it is not supported). I thought about a redirect for the token after / to a page called /default/redirect/<token> which would then decide about the redirect, but there's no way to stop the infinite loop. Another possible solution was a tweak to the nginx config so redirect when there is one token after the /, but again I think this causes a problem with the about.
Maybe there is a catch-all function for controllers that I haven't found? I've gone through the web2py book and found nothing - any ideas very welcome!
Thanks,
Sam

You can use the parameter-based rewrite system. In routes.py include:
routers = dict(
BASE=dict(
default_application='yourapp',
default_controller='default',
default_function='company',
functions=['index', 'company', 'user', 'download', 'call']
),
)
Note, functions should be a list of all functions in the default.py controller.
If you go to /some_token, it will route to /yourapp/default/company/some_token, unless some_token is "yourapp", "default", "company", or any of the functions in the functions list (in which case, it will assume you are actually requesting the "yourapp" app, the "default" controller, or the particular function from the controller).
Note, if you simply go to the root URL (i.e., http://yourdomain.com/), it will route to /yourapp/default/company/, with no URL arg, so you should be prepared for that case.

So I found the solution (sorry for the delay in posting):
In the routes.py at the route of web2py directory, I added a rule in routes_in, so that this was in the my file:
routes_in = (
('/(?!about)$token', '/company/$token'),
)
to manage the default app (removing the application name and the default package name from the URL), I did this (not necessary for the redirects to work):
routers = dict(
BASE = dict(default_application='startbean'),
)
And it all worked :)

In the Pyramid web framework, how do I source sensitive settings into development.ini / production.ini from an external file?

I'd like to keep development.ini and production.ini under version control, but for security reason would not want the sqlalchemy.url connection string to be stored, as this would contain the username and password used for the database connection.
What's the canonical way, in Pyramid, of sourcing this setting from an additional external file?
Edit
In addition to solution using the environment variable, I came up with this solution after asking around on #pyramid:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
# Read db password from config file outside of version control
secret_cfg = ConfigParser()
secret_cfg.read(settings['secrets'])
dbpass = secret_cfg.get("secrets", "dbpass")
settings['sqlalchemy.url'] = settings['connstr'] % (dbpass,)

I looked into this a lot and played with a lot of different approaches. However, Pyramid is so flexible, and the .ini config parser is so minimal in what it does for you, that there doesn't seem to be a de facto answer.
In my scenario, I tried having a production.example.ini in version control at first that got copied on the production server with the details filled in, but this got hairy, as updates to the example didn't get translated to the copy, and so the copy had to be re-created any time a change was made. Also, I started using Heroku, so files not in version control never made it into the deployment.
Then, there's the encrypted config approach. Which, I don't like the paradigm. Imagine a sysadmin being responsible for maintaining the production environment, but he or she is unable to change the location of a database or environment-specific setting without running it back through version control. It's really nice to have the separation between environment and code as much as possible so those changes can be made on the fly without version control revisions.
My ultimate solution was to have some values that looked like this:
[app:main]
sqlalchemy.url = ${SQLALCHEMY_URL}
Then, on the production server, I would set the environment variable SQLALCHEMY_URL to point to the database. This even allowed me to use the same configuration file for staging and production, which is nice.
In my Pyramid init, I just expanded the environment variable value using os.path.expandvars:
sqlalchemy_url = os.path.expandvars(settings.get('sqlalchemy.url'))
engine = create_engine(sqlalchemy_url)
And, if you want to get fancy with it and automatically replace all the environment variables in your settings dictionary, I made this little helper method for my projects:
def expandvars_dict(settings):
"""Expands all environment variables in a settings dictionary."""
return dict((key, os.path.expandvars(value)) for
key, value in settings.iteritems())
Use it like this in your main app entry point:
settings = expandvars_dict(settings)

The whole point of the separate ini files in Pyramid is that you do not have to version control all of them and that they can contain different settings for different scenarios (development/production/testing). Your production.ini almost always should not be in the same VCS as your source code.

I found this way for loading secrets from a extra configuration and from the env.
from pyramid.config import Configurator
from paste.deploy import appconfig
from os import path
__all__ = [ "main" ]
def _load_secrets(global_config, settings):
""" Helper to load secrets from a secrets config and
from env (in that order).
"""
if "drawstack.secrets" in settings:
secrets_config = appconfig('config:' + settings["drawstack.secrets"],
relative_to=path.dirname(global_config['__file__']))
for k, v in secrets_config.items():
if k == "here" or k == "__file__":
continue
settings[k] = v
if "ENV_DB_URL" in global_config:
settings["sqlalchemy.url"] = global_config["ENV_DB_URL"]
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
_load_secrets(global_config, settings)
config = Configurator(settings=settings)
config.include('pyramid_jinja2')
config.include('.models')
config.include('.routes')
config.scan()
return config.make_wsgi_app()
The code above, will load any variables from the value of the config key drawstack.secrets and after that it tries to load DB_URL from the enviornment.
drawstack.secrets can be relative to the original config file OR absolute.

If I make MongoDB my root factory, do I also need to register it with Configurator?

I'm trying to setup Pyramid's Authorization/Authentication feauture using my MongoDB as the root factory. I'm wondering if including these lines (config is Configurator)
db_url = urlparse(eval(settings['mongo_uri']))
conn = pymongo.Connection(host=db_url.hostname,
port=db_url.port)
config.registry.settings['db_conn'] = conn
config.add_subscriber(add_mongo_db, NewRequest)
is redundant? Is this necessary if I've already given config a mongo root factory?

I don't recommend doing it that way. I wrote a pyramid addon to make things easier and cleaner.
Documentation here:
http://packages.python.org/pyramid_mongo/
The following is from a project I'm writing at the moment.
In my ini file (while it may be written in python settings)
mongo.uri = mongodb://localhost/
mongo.db = wife
In my configurator:
config.include('pyramid_mongo')
And in my root_factory:
from pyramid_mongo import get_db
...
...
def root_factory(request):
db = get_db(request)
return Root(db)
get_db can be called from anywhere, you have to pass a request as first argument. You can pass an other argument to query a different database.
Subscribers aren't needed in that case.
Btw, don't worry if it's written in the documentation that it might be risky, the current version of the package has 100% coverage and pass all tests. In the future, this package may integrate some tools in order to simplify traversal with mongodb.

Running scripts within Pyramid framework (ie without a server)

I have a fair bit of experience with PHP frameworks and Python for scripting so am now taking the step to Pyramid.
I'd like to know what is the 'correct' way to run a script in Pyramid. That is, how should I set it up so that it is part of the application and has access to config and thus database but does not run through paster (or whatever WSGI).
As an example, say I have a web application which while a user is offline grabs Facebook updates through a web service. I want to write a script to poll that service and store in the database ready for next login.
How should I do this in terms of:
Adding variables in the ini file
Starting the script correctly
I understand the basics of Python modules and packages; however I don't fully understand Configurator/Paster/package setup, wherein I suspect the answer lies.
Thanks
Update:
Thanks, this seems along the lines of what I am looking for. I note that you have to follow a certain structure (eg have summary and parser attributes set) and that the function called command() will always be run. My test code now looks something like this:
class AwesomeCommand(Command):
max_args = 2
min_args = 2
usage = "NAME"
# These are required
summary = "Say hello!"
group_name = "My Package Name"
# Required:
parser = Command.standard_parser(verbose=True)
def command(self):
# Load the config file/section
config_file, section_name = self.args
# What next?
I'm now stuck as to how to get the settings themselves. For example, in init.py you can do this:
engine = engine_from_config(settings, 'sqlalchemy.')
What do I need to do to transform the config file into the settings?
EDIT: The (simpler) way to do this in Pylons is here:
Run Pylons controller as separate app?

As of Pyramid 1.1, this is handled by the framework:
http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/commandline.html#writing-a-script

paster starts an application given an ini file that describes that application. the "serve" command is a built in command for starting a wsgi application and serving it. BUT, you can write other commands.
from paste.script.command import Command
class AwesomeCommand(Command):
def command(self):
print "the awesome thing it does"
and then register them as entry points in your setup.py.
setup(...
entry_points="""
[paste.app_factory]
.....
[paste.global_paster_command]
myawesome-command = mypackage.path.to.command:AwesomeCommand """)
pyramid adds it's own commands this way like the pshell command.

After going to the pylons discuss list, I came up with an answer. Hope this helps somebody:
#Bring in pyramid application--------------------------------------
import pyramid
from paste.deploy import appconfig
config_file = '/path_to_config_file/configname.ini'
name = 'app_name'
config_name = 'config:%s' % config_file
here_dir = os.getcwd()
conf = appconfig(config_name, name, relative_to=here_dir)
from main_package import main
app = main(conf.global_conf, **conf.local_conf)
#--------------------------------------------------------------------------

you need to make view for that action and then run it using:
paster request development.ini /url_to_your_view

django.core.urlresolvers.resolve incorrect behavior under apache non-root deployment

When a django app is deployed under a non-root apache url (with WsgiScriptAlias /suburl /path_to_django.wsgi) the {%url%} tag and the django.core.urlresolvers.reverse function take into account the SCRIPT_NAME variable and return urls of the form /suburl/path_to_my_view
However, when I use the django.core.urlresolvers.resolve function to resolve those urls it throws an error. That forces me to strip the SCRIPT_NAME of the generated urls before calling resolve. Is this the expected behavior or am I misunderstanding everything?
Regards.

I got the same kind of problem:
SCRIPT_NAME defined in my apache config
a call to django.core.urlresolvers.reverse outside the wsgi didnt prepend the prefix
in any view or resource, a call to the same method prepended the prefix
I managed to have the prefix automatically prepended using the next lines of code:
# in settings.py
from django.core.urlresolvers import set_script_prefix
...
FORCE_SCRIPT_NAME = "your-prefix"
set_script_prefix(FORCE_SCRIPT_NAME)
...
The first line makes sure your wsgi uses your prefix every time. The second one sets the prefix, so that reverse will find it.
Please note that you need to have the same prefix in your apache conf. It's a bit redundant, but the cleaner fix I found.