We Keep Coding

Python 2.7->2.6 problems

I'm a beginner with Python and Django.
I'm setting up a program i've written locally. After almost finishing getting the app to work on the server, i've learned that the server is running python 2.6, while my local system runs 2.7. This is seemingly giving me problems when retrieving paramters from urls.
I'm using a server from Openshift. I don't know much about servers, but my current setup is that I have a local clone of the file, and I work on everything locally, and the push them via git to the server. The server was set up using a predefined quick setup from inside the Openshift interface.
I'm using the following urlpattern, which works just fine locally on my computer.
url(r'^website/(?P<url>[:\w/\-.]+)$', 'page'),
However, on my server version i'm running into some problems. The following url, returns two different urls to the view, depending on whether i'm on the server or running local.
#when using this url
website/http://example.com
#local view called page, retrives this argument
http://example.com
#server version retrieves almost the same, but with one / in the beginner less.
http:/example.com
It seems to me that a backslash is being chopped off somewhere. How can I change it to parse the argument with both backslashes?
# the receiving view
def page(request, url):
p = Page.objects.get(url=url)
domain = p.website.url
return render_to_response('seo/page.html', {'domain': domain, 'page': p}, context_instance=RequestContext(request))
The local version is returning the desired page just fine. The server version returns this:
DoesNotExist at /website/http:/coverme.dk/collections/iphone-sleeves-covers
I noticed that one of the backslashes in http:// was missing here, and assumed the error was based on it being sent to the view incorrectly.
I've just tested with an url that does not exist in the database on the local version, and it displays the error message correctly.
I've also double checked that the object for url='http://coverme.dk/collections/iphone-sleeves-covers' actually exists. I've also checked with several others.
I've experimented with messing around with the input url, and it seems to working just fine, except when I use double, triple of more backslashes. All backslashes succesively after the first are ignored in the url.
/website/http://////coverme.dk////collections/iphone-sleeves-covers
#gives the same as
/website/http:/coverme.dk/collections/iphone-sleeves-covers.
Any kind of help is much appreciated. A link to some documentation that could help me out would be greatly appreciated as well.
EDIT: Updating django solved this issue.

From a comment by the author of the question:
Using /website/http%3A%2F%2Fcoverme.dk%2Fcollections%2Fiphone-sleeves-covers as the url returns: The requested URL /website/coverme.dk/collections/iphone-sleeves-covers was not found on this server. Django version on the server is 1.4 and the one being used locally is 1.5.1. I've still to understand why i'm seeing different results locally and on the server, but i'm starting to think i should just switch to an url pattern that doesn't use //?
Updating Django solved the issue for me

Subprocess doesn't wait and makes PhantomJS crash

I have two scripts running, one on port :80 and one on port :81. Because some of our users are having issues with stuff happening on the server with port :81, I'm trying to implement a workaround like this;
Old way of doing it, which works fine for most users:
AngularJS app makes request to example.com:81/getpdf/1
Flask server generates PNG and PDF files using PhantomJS and ImageMagick using two separate subprocess.Popen calls and the .wait() method
Using Flask's send_file(), the PDF gets sent back to the user and starts downloading
My workaround for this issue:
AngularJS makes request to example.com/getpdf/1
Flask server (:80) makes a new GET request, r = requests.get(url_with_port_81), faking the old AngularJS request to create the PNG/PDF
Instead of using send_file(), I now return the path of the generated PDF
I return send_file(r.text)
Now, using my workaround, the subprocesses I run to create the PNG/PDFs somehow crash. I have to sudo pkill python, and only when I do so, I'm getting a PNG with no data in the folder on my server.
Basically, PhantomJS has run but hasn't loaded any data (only html/css, but no important stuff that needs to come from the Flask server) and crashes. How is this even possible? I'm just faking the request the browser makes using requests.get, or am I not aware of something here?
I thought subprocess.Popen is non-blocking, so my requests for data could still be answered to fill the PNG/PDFs?

I finally found the reason my subprocess kept crashing.
Apparently, it's a bug in Python < 2.7.3, described here: http://bugs.python.org/issue12786
I had to use 'close_fds=True' in my Popen call and all was fixed. Thanks for your effort either way, #Mark Hildreth!

How can a CGI server based on CGIHTTPRequestHandler require that a script start its response with headers that include a `content-type`?

Later note: the issues in the original posting below have been largely resolved.
Here's the background: For an introductory comp sci course, students develop html and server-side Python 2.7 scripts using a server provided by the instructors. That server is based on CGIHTTPRequestHandler, like the one at pointlessprogramming. When the students' html and scripts seem correct, they port those files to a remote, slow Apache server. Why support two servers? Well, the initial development using a local server has the benefit of reducing network issues and dependency on the remote, weak machine that is running Apache. Eventually porting to the Apache-running machine has the benefit of publishing their results for others to see.
For the local development to be most useful, the local server should closely resemble the Apache server. Currently there is an important difference: Apache requires that a script start its response with headers that include a content-type; if the script fails to provide such a header, Apache sends the client a 500 error ("Internal Server Error"), which too generic to help the students, who cannot use the server logs. CGIHTTPRequestHandler imposes no similar requirement. So it is common for a student to write header-free scripts that work with the local server, but get the baffling 500 error after copying files to the Apache server. It would be helpful to have a version of the local server that checks for a content-type header and gives a good error if there is none.
I seek advice about creating such a server. I am new to Python and to writing servers. Here are the issues that occur to me, but any helpful advice would be appreciated.
Is a content-type header required by the CGI standard? If so, other people might benefit from an answer to the main question here. Also, if so, I despair of finding a way to disable Apache's requirement. Maybe the relevant part of the CGI RFC is section 6.3.1 (CGI Response, Content-Type): "If an entity body is returned, the script MUST supply a Content-Type field in the response."
To make a local server that checks for the content-type header, perhaps I should sub-class CGIHTTPServer.CGIHTTPRequestHandler, to override run_cgi() with a version that issues an error for a missing header. I am looking at CGIHTTPServer.py __version__ = "0.4", which was installed with Python 2.7.3. But run_cgi() does a lot of processing, so it is a little unappealing to copy all its code, just to add a couple calls to a header-checking routine. Is there a better way?
If the answer to (2) is something like "No, overriding run_cgi() is recommended," I anticipate writing a version that invokes the desired script, then checks the script's output for headers before that output is sent to the client. There are apparently two places in the existing run_cgi() where the script is invoked:
3a. When run_cgi() is executed on a non-Unix system, the script is executed using Python's subprocess module. As a result, the standard output from the script will be available as an in-memory string, which I can presumably check for headers before the call to self.wfile.write. Does this sound right?
3b. But when run_cgi() is executed on a *nix system, the script is executed by a forked process. I think the child's stdout will write directly to self.wfile (I'm a little hazy on this), so I see no opportunity for the code in run_cgi() to check the output. Ugh. Any suggestions?
If analyzing the script's output is recommended, is email.parser the standard way to recognize whether there is a content-type header? Is another standard module recommended instead?
Is there a more appropriate forum for asking the main question ("How can a CGI server based on CGIHTTPRequestHandler require...")? It seems odd to ask if there is a better forum for asking programming questions than Stack Overflow, but I guess anything is possible.
Thanks for any help.

How to Disable Django / mod_WSGI Page Caching

I have Django running in Apache via mod_wsgi. I believe Django is caching my pages server-side, which is causing some of the functionality to not work correctly.
I have a countdown timer that works by getting the current server time, determining the remaining countdown time, and outputting that number to the HTML template. A javascript countdown timer then takes over and runs the countdown for the user.
The problem arises when the user refreshes the page, or navigates to a different page with the countdown timer. The timer appears to jump around to different times sporadically, usually going back to the same time over and over again on each refresh.
Using HTTPFox, the page is not being loaded from my browser cache, so it looks like either Django or Apache is caching the page. Is there any way to disable this functionality? I'm not going to have enough traffic to worry about caching the script output. Or am I completely wrong about why this is happening?
[Edit] From the posts below, it looks like caching is disabled in Django, which means it must be happening elsewhere, perhaps in Apache?
[Edit] I have a more thorough description of what is happening: For the first 7 (or so) requests made to the server, the pages are rendered by the script and returned, although each of those 7 pages seems to be cached as it shows up later. On the 8th request, the server serves up the first page. On the 9th request, it serves up the second page, and so on in a cycle. This lasts until I restart apache, when the process starts over again.
[Edit] I have configured mod_wsgi to run only one process at a time, which causes the timer to reset to the same value in every case. Interestingly though, there's another component on my page that displays a random image on each request, using order('?'), and that does refresh with different images each time, which would indicate the caching is happening in Django and not in Apache.
[Edit] In light of the previous edit, I went back and reviewed the relevant views.py file, finding that the countdown start variable was being set globally in the module, outside of the view functions. Moving that setting inside the view functions resolved the problem. So it turned out not to be a caching issue after all. Thanks everyone for your help on this.

From my experience with mod_wsgi in Apache, it is highly unlikely that they are causing caching. A couple of things to try:
It is possible that you have some proxy server between your computer and the web server that is appropriately or inappropriately caching pages. Sometimes ISPs run proxy servers to reduce bandwidth outside their network. Can you please provide the HTTP headers for a page that is getting cached (Firebug can give these to you). Headers that I would specifically be interested in include Cache-Control, Expires, Last-Modified, and ETag.
Can you post your MIDDLEWARE_CLASSES from your settings.py file. It possible that you have a Middleware that performs caching for you.
Can you grep your code for the following items "load cache", "django.core.cache", and "cache_page". A *grep -R "search" ** will work.
Does the settings.py (or anything it imports like "from localsettings import *") include CACHE_BACKEND?
What happens when you restart apache? (e.g. sudo services apache restart). If a restart clears the issue, then it might be apache doing caching (it is possible that this could also clear out a locmen Django cache backend)

Did you specifically setup Django caching? From the docs it seems you would clearly know if Django was caching as it requires work beforehand to get it working. Specifically, you need to define where the cached files are saved.
http://docs.djangoproject.com/en/dev/topics/cache/

Are you using a multiprocess configuration for Apache/mod_wsgi? If you are, that will account for why different responses can have a different value for the timer as likely that when timer is initialised will be different for each process handling requests. Thus why it can jump around.
Have a read of:
http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
Work out in what mode or configuration you are running Apache/mod_wsgi and perhaps post what that configuration is. Without knowing, there are too many unknowns.

I just came across this:
Support for Automatic Reloading To help deployment tools you can
activate support for automatic reloading. Whenever something changes
the .wsgi file, mod_wsgi will reload all the daemon processes for us.
For that, just add the following directive to your Directory section:
WSGIScriptReloading On

How to improve Trac's performance

I have noticed that my particular instance of Trac is not running quickly and has big lags. This is at the very onset of a project, so not much is in Trac (except for plugins and code loaded into SVN).
Setup Info: This is via a SELinux system hosted by WebFaction. It is behind Apache, and connections are over SSL. Currently the .htpasswd file is what I use to control access.
Are there any recommend ways to improve the performance of Trac?

It's hard to say without knowing more about your setup, but one easy win is to make sure that Trac is running in something like mod_python, which keeps the Python runtime in memory. Otherwise, every HTTP request will cause Python to run, import all the modules, and then finally handle the request. Using mod_python (or FastCGI, whichever you prefer) will eliminate that loading and skip straight to the good stuff.
Also, as your Trac database grows and you get more people using the site, you'll probably outgrow the default SQLite database. At that point, you should think about migrating the database to PostgreSQL or MySQL, because they'll be able to handle concurrent requests much faster.

We've had the best luck with FastCGI. Another critical factor was to only use https for authentication but use http for all other traffic -- I was really surprised how much that made a difference.

I have noticed that if
select disctinct name from wiki
takes more than 5 seconds (for example due to a million rows in this table - this is a true story (We had a script that filled it)), browsing wiki pages becomes very slow and takes over 2*t*n, where t is time of execution of the quoted query (>5s of course), and n is a number of tracwiki links present on the viewed page.
This is due to trac having a (hardcoded) 5s cache expire for this query. It is used by trac to tell what the colour should the link be. We re-hardcoded the value to 30s (We need that many pages, so every 30s someone has to wait 6-7s).
It may not be what caused Your problem, but it may be. Good luck on speeding up Your Trac instance.

Serving the chrome files statically with and expires-header could help too. See the end of this page.