Twill : requests.exceptions.MissingSchema - python

I'm working on a simple web scraper for a page which requires users to be logged in to see its content.
from twill.commands import *
go("https://website.com/user")
fv("1","edit-name","NICKNAME")
fv("1","edit-pass","NICKNAME")
submit('0')
That is my current code. When running it, I get the following error
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '/user': No schema supplied. Perhaps you meant http:///user?
What am I doing wrong?

Related

CherryPy redirect to root on missing parameters error

I have a web server using CherryPy. I would like for users to be redirected to the root page if they try and access a page which requires parameters to be entered. If a user were to try and directly access:
localhost:8080/page
they will get an error message which looks something like:
404 Not Found
Missing parameters: parameter1,parameter2,...
Instead of this, I would like the user to be redirected to:
localhost:8080/
I have tried to change cherrypy.config such that whenever a 404 occurs, it checks the error message and sees if the issue is missing parameters. If this is not the issue, I would like the default 404 page to be shown. My code looks like this:
cherrypy.config.update({'error_page.404': error_page_404})
with the function error_page_404:
def error_page_404(status, message, traceback, version):
if message.startswith('Missing parameters'):
raise cherrypy.HTTPRedirect('/')
else:
raise cherrypy.HTTPError(status, message)
but when I try to access
localhost:8080/page,
I get the following error:
404 Not Found
Missing parameters: parameter1,parameter2
In addition, the custom error page failed:
cherrypy._cperror.HTTPRedirect: (['http://127.0.0.1:8080/'], 303)
Any thoughts?
Thank you!
I've managed to make a workaround by returning HTML code that redirects to the root:
def error_page_404(status, message, traceback, version):
if message.startswith('Missing parameters'):
return """<meta http-equiv="Refresh" content="0; url='/'" />"""
else:
return f'<h2>404 Not Found</h2>\n' \
f'<p>{message}</p>'
While this works for my purposes, I can't help but feel there must be a more native way of doing this in cherrypy

Display a specific template when Django raises an exception

In my django app, I have multiple places where I raise a specific custom exception DeserializationError. My goal is to show/redirect to a pretty page to show the user this error, including the error message, when this error is raised. Basically, a page that says something like
Something went wrong. Please contact webmaster#email.com.
Error: DeserializationError. Message: SomeModel, "somemodel", does not exist.
Would this even be possible? I've been trying to search for a solution but haven't been able to find anything yet.
Most likely such errors will return HTTP 500 server error.
In django you can write your own custom view to handle such cases and return your own page with the html that you like.
The 500 (server error) view explains writing server error views. There are more types of errors handled as explained on same page.
An option for handling HTTP 500 errors, add this to your Settings file,
handler500 = 'mysite.views.my_custom_error_view'
and in the view, you can render the "error page" using
HttpResponseNotFound('<h1>Page not found</h1>')
the server_error() view will be overridden by handler500.

Python requests causing error on certain urls

For some reason when I try to get and process the following url with python-requests I receive an error causing my program to fail. Other similar urls seems to work fine
import requests
test = requests.get('http://t.co/Ilvvq1cKjK')
print test.url, test.status_code
What could be causing this URL to fail instead of just producing a 404 status code?
The requests library has an exception hierarchy as listed here
So wrap your GET request in a try/except block:
import requests
try:
test = requests.get('http://t.co/Ilvvq1cKjK')
print test.url, test.status_code
except requests.exceptions.ConnectionError as e:
print e.request.url, "*connection failed*"
That way you end up with similar behaviour to what you're doing now (so you get the redirected url), but cater for not being able to connect rather than print the status code.

Python mongolab REST api

I am trying to access the mongolab REST api through python. Is the correct way to do this via pythons urllib2? I have tried the following:
import urllib2
p = urllib2.urlopen("https://api.mongolab.com/api/1/databases/mydb/collections/mycollection?apiKey=XXXXXXXXXXXXXXXX")
But this gives me an error:
urllib2.URLError: <urlopen error unknown url type: https>
What is the correct way of doing this? After connecting, how do I go on to POST a document to my collection? If someone could post up a code example, I would be very grateful. Thanks all for the help!
EDIT:
I've recompiled python with ssl support. How do I POST insert a document to a collection using mongolab REST API? Here is the code I have atm:
import urllib
import urllib2
url = "https://api.mongolab.com/api/1/databases/mydb/collections/mycollection?apiKey=XXXXXXXXXXXXXXXX"
data = {"x" : "1"}
request = urllib2.Request(url, data)
p = urllib2.urlopen(request)
Now, when I run this, I get the error
urllib2.HTTPError: HTTP Error 415: Unsupported Media Type
How do I insert documents using HTTP POST? Thanks!
That error is raised if you version of python does not include ssl support. What version are you using? Did you compile it yourself?
That said, when you get a version including ssl, using requests is a lot easier than urllib2, especially when POSTing data.

How to get the content of "Google Dictionary" by python script

Thank to the script, I've logged in google successfully.
But I replaced the value of "gv_home_page_url" with http:// www.google.com.tw/dictionary/wordlist?hl=zh-TW, the error occured.
The message is "
urllib2.HTTPError: HTTP Error 500: Internal Server Error"
Any idea will be appreciated, thanks.
That's not a valid URL. If you try to enter it in your bar, you'll get redirected to Google search results. Using urllib2, you'll get an error.
See here for another way to get data from Google Dictionary:
http://googlesystem.blogspot.com/2009/12/on-googles-unofficial-dictionary-api.html
Does Google require an SSL url? You're using "http://" but the example script uses "https://"

Categories