We Keep Coding

flask not providing results to ajax call

I am making ajax call to flask function to get data using a token as following
#app.route("/questgen/results", methods = ['POST', 'GET'])
def fetch_results():
token = request.form.get('token')
print(token)
conn = sqlite3.connect("database.db" , timeout=20)
cur= conn.cursor()
select_query = '''SELECT processed, output_text
FROM results
WHERE token = token
'''
cur.execute(select_query)
records = cur.fetchall()
processed = ""
html = ""
for row in records:
processed = row[0]
html = row[1]
conn.close()
data = {'processed': processed, 'html' : html}
return redirect(url_for('questgen', data = data))
ajax call is as following
$.ajax({
type: "POST",
url: "/questgen/results",
data: { token: token },
datatype: "json",
success: function (data) {
if (data.processed == 1) {
$('#divresults').html(data.html);
$('#divresults').show();
hideMyModal();
clearInterval(saveInterval);
}
}
});
the complete is a bit lengthy it can be found in this gist the problem is that it returns
TypeError: The view function did not return a valid response. The
function either returned None or ended without a return statement.
even though I have tried same flask function as python function by using return data on the same database and it works. I have even tried to get token as function parameter but still it's not working. Can someone help with what am I doing wrong here? Thank you

The jQuery ajax call does not handle redirects automatically. You'll just get a response with a 301 or 302 status code. If you really need to have it redirect, you'll need to check for a 302 status return and make the call again with the changed data. It would be better if you could just do the redirection internally by calling the other function.

Try jsonify to return data
from flask import Flask, jsonify, request #import this
And then use this to return data
return jsonify({"res": data})
In ajax you will get your data in res
console.log(data.res) // your data
console.log(data.res.processed) // your if condition
Also check whether you need to parse response body or not

Why does my Flask API endpoint get a CORS error, but other similarly programmed endpoints do not?

Problem
I have a Flask API with two different kinds of routes. I am making requests to my API endpoints from a JavaScript/D3.js application. I need help troubleshooting a CORS error.
Endpoint format
One kind of endpoint follows the format http://127.0.0.1:5000/api/v0.1.0/infrastructure/primary_type/secondary_type
And the other kind follows the format http://127.0.0.1:5000/api/v0.1.0/infrastructure/primary_type/
I am getting a CORS error for the one following the format http://127.0.0.1:5000/api/v0.1.0/infrastructure/primary_type/
CORS Error
Access to fetch at 'http://127.0.0.1:5000/api/v0.1.0/infrastructure/primary_type' from origin 'http://127.0.0.1:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
Endpoint Code
Working Endpoint
#bp.route("<primary_type>/<secondary_type>")
def get_infrastructure(primary_type, secondary_type):
infrastructure = ''
if primary_type == 'null':
infrastructure = shotgun_api.db.infrastructure.find({'properties.type.secondary': secondary_type}, projection = {'_id': False})
else:
infrastructure = shotgun_api.db.infrastructure.find({"$and": [{'properties.type.primary': primary_type},{'properties.type.secondary': secondary_type}]}, projection = {"_id": False})
response = jsonify([resource for resource in infrastructure])
response.headers.add("Access-Control-Allow-Origin", "*")
return response
Endpoint triggering errors
#bp.route('<primary_type>/')
def get_by_primary_type(primary_type):
infrastructure = shotgun_api.db.infrastructure.find({'properties.type.primary': primary_type}, projection = {'_id': False})
response = jsonify([resource for resource in infrastructure])
response.headers.add("Access-Control-Allow-Origin", "*")
return response
Expected behavior
I would expect that using the same response.headers.add... code would provide me with a 200 OK status but it only works for the more complex endpoint for some reason.
Client Side JavaScript/D3.js
for (let i = 0, num_draw_props = lyr.draw_props.length; i < num_draw_props; ++i) {
start_loading_layer();
Promise.all(lyr.draw_props[i].src.map(x => lyr.draw_props[i].d3_fetch(x)))
.then(function(files) {
lyr.context.restore();
lyr.context.save();
return files;
}).then(files => {
transform_layer(lyr.context, transform);
return files
}).then(files => {
console.time('draw_layer')
lyr.draw_props[i].draw_layer(lyr.context, files);
console.timeEnd('draw_layer')
});
}
Where d3_fetch is equal to d3.json(). When src is 'http://127.0.0.1:5000/api/v0.1.0/infrastructure/mines/coal' everything works fine, but when src is http://127.0.0.1:5000/api/v0.1.0/infrastructure/railroads I get the previously mentioned CORS error.
What I've tried
I have tried using Flask-CORS but was unable to get it accessible to my application despite installing it with pip. I would prefer to solve this without using Flask-CORS. I am otherwise at a loss for what to do here and any advice is appreciated.

Cant get AngularJS's $http.post data in Django

I know its a very basic question but after wasting my whole day I am asking this. I am just sending data using following AngularJS code to Django:
$http.post('/data/creation',
{
html: 'a'
}).
success(function(data, status, headers, config) {
// this callback will be called asynchronously
// when the response is available
console.log(data);
console.log(status);
console.log(headers);
console.log(config);
}).
error(function(data, status, headers, config) {
// called asynchronously if an error occurs
// or server returns response with an error status.
console.log(status);
console.log(data);
});
and in django:
#csrf_exempt
def snippets_post(request):
html = False
css = False
js = False
JSONdata = False
response = "You're looking at the results of question %s."
if request.method == 'POST':
try:
JSONdata = request.POST.get('data', False) # it was [] in actual
except:
JSONdata = 'ERROR'
return HttpResponse(JSONdata)
I am getting False as response, "by replacing data to html in POST.get result is same". I don't know whats going wrong here. Can any one help me here on this?
Thanks

Actually, when we send data from AngularJs using $http's POST, it sends the data with the content-type = "application/json" to the server. And Django doesn't understand that format. And so you can't get the sent data.
Solution is to change the content-type header by using following config:
app.config(['$httpProvider', function ($httpProvider) {
$httpProvider.defaults.headers.post['Content-Type'] = 'application/x-www-form-urlencoded';
}]);

Login to site using CSRF token using Go/Python

I want to backup automatically web content from a site which requires login. I try to login by simulating a POST request. But I get the error:
csrf token: CSRF attack detected
Here are some extracts from the code I use:
func postLoginForm(csrfToken string) {
values := make(url.Values)
values.Set("signin[username]", "myusername")
values.Set("signin[password]", "mypassword")
values.Set("signin[_csrf_token]", csrfToken)
resp, err := http.PostForm("https://spwebservicebm.reaktor.no/admin/nb", values)
dumpHTTPResponse(resp) // show response to STDOUT
}
The csrf token I get by fetching the login page and scanning it for a hidden input field named signin[_csrf_token]. The important part of the code for doing that is the following:
// Finds input field named signin[_csrf_token] and returns value as csrfToken
func handleNode(n *html.Node) (csrfToken string, found bool) {
if n.Type == html.ElementNode && n.Data == "input" {
m := make(map[string]string)
for _, attr := range n.Attr {
m[attr.Key] = attr.Val
}
if m["name"] == "signin[_csrf_token]" {
return m["value"], true
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
if csrfToken, found = handleNode(c); found {
return
}
}
return "", false
}
I don't need to be using Go, that is just because I am most familiar with that. Using python could be a solution as well, but I did not have any more luck with that.

The issue is that Go 1.2 does not automatically use a cookie jar for its HTTP requests. The first request is to get the CSRF token from the login page. The second request is to POST a login using that CSRF token. But since no session cookie is attached to the HTTP header on the second request the server does not know that it is the same program trying to login. Thus the server thinks it is a CSRF attempt (that you picked the CSRF token from somewhere else and tried to reuse it).
So to get login page and extract CSRF token, we first create our own client object. Otherwise we have nowhere to attach the cookie jar. http.PostForm does give access to cookie jar:
client = &http.Client{}
Create a cookie Jar described in authenticated http client requests from golang . This was easier to setup and debug than the official: http://golang.org/pkg/net/http/cookiejar/ Cookie Jar
jar := &myjar{}
jar.jar = make(map[string] []*http.Cookie)
client.Jar = jar
resp, err := client.Get("https://spwebservicebm.reaktor.no/admin")
doc, err := html.Parse(resp.Body)
Then to login, we reuse the client object with the cookie jar attached to it:
values := make(url.Values)
values.Set("signin[username]", "myusername")
values.Set("signin[password]", "mypassword")
values.Set("signin[_csrf_token]", csrfToken)
resp, err := client.PostForm("https://spwebservicebm.reaktor.no/admin/login", values)
You'll notice that the code is almost identical to the one in the question except we use client.PostForm instead of http.PostForm.
Thanks to dommage and answer to authenticated http client requests from golang for getting me on right track.

You can scrape the token using beautifulsoup and store it, either in the header or send it to the server. You can do something like this:
from requests import session
from bs4 import BeautifulSoup
def authent(self):
ld('trying to get token...')
r = self.session.get(BASE_URL, headers=FF_USER_AGENT)
soup = BeautifulSoup(r.content)
elgg_token = soup.select('input[name="__elgg_token"]')[0]["value"]
elg_ts = soup.select('input[name="__elgg_ts"]')[0]["value"]
payload["__elgg_token"] = elgg_token # I sent it to the server...
payload["__elgg_ts"] = elg_ts
r = self.session.post(LOGIN_URL, data=payload, headers=FF_USER_AGENT)
if r.url != DASHBOARD_URL:
raise AuthentError("Error")

Python Scrapy - mimetype based filter to avoid non-text file downloads

I have a running scrapy project, but it is being bandwidth intensive because it tries to download a lot of binary files (zip, tar, mp3, ..etc).
I think the best solution is to filter the requests based on the mimetype (Content-Type:) HTTP header. I looked at the scrapy code and found this setting:
DOWNLOADER_HTTPCLIENTFACTORY = 'scrapy.core.downloader.webclient.ScrapyHTTPClientFactory'
I changed it to:
DOWNLOADER_HTTPCLIENTFACTORY = 'myproject.webclients.ScrapyHTTPClientFactory'
And played a little with the ScrapyHTTPPageGetter, here is the edits highlighted:
class ScrapyHTTPPageGetter(HTTPClient):
# this is my edit
def handleEndHeaders(self):
if 'Content-Type' in self.headers.keys():
mimetype = str(self.headers['Content-Type'])
# Actually I need only the html, but just in
# case I've preserved all the text
if mimetype.find('text/') > -1:
# Good, this page is needed
self.factory.gotHeaders(self.headers)
else:
self.factory.noPage(Exception('Incorrect Content-Type'))
I feel this is wrong, I need more scrapy friendly way to cancel/drop request right after determining that it's unwanted mimetype. Instead of waiting for the whole data to be downloaded.
Edit:
I'm asking specifically for this part self.factory.noPage(Exception('Incorrect Content-Type')) is that the correct way to cancel a request.
Update 1:
My current setup have crashed the Scrapy server, so please don't try to use the same code above to solve the problem.
Update 2:
I have setup an Apache-based website for testing using the following structure:
/var/www/scrapper-test/Zend -> /var/www/scrapper-test/Zend.zip (symlink)
/var/www/scrapper-test/Zend.zip
I have noticed that Scrapy discards the ones with the .zip extension, but scraps the one without .zip even though it's just a symbolic link to it.

I built this Middleware to exclude any response type that isn't in a whitelist of regular expressions:
from scrapy.http.response.html import HtmlResponse
from scrapy.exceptions import IgnoreRequest
from scrapy import log
import re
class FilterResponses(object):
"""Limit the HTTP response types that Scrapy dowloads."""
#staticmethod
def is_valid_response(type_whitelist, content_type_header):
for type_regex in type_whitelist:
if re.search(type_regex, content_type_header):
return True
return False
def process_response(self, request, response, spider):
"""
Only allow HTTP response types that that match the given list of
filtering regexs
"""
# each spider must define the variable response_type_whitelist as an
# iterable of regular expressions. ex. (r'text', )
type_whitelist = getattr(spider, "response_type_whitelist", None)
content_type_header = response.headers.get('content-type', None)
if not type_whitelist:
return response
elif not content_type_header:
log.msg("no content type header: {}".format(response.url), level=log.DEBUG, spider=spider)
raise IgnoreRequest()
elif self.is_valid_response(type_whitelist, content_type_header):
log.msg("valid response {}".format(response.url), level=log.DEBUG, spider=spider)
return response
else:
msg = "Ignoring request {}, content-type was not in whitelist".format(response.url)
log.msg(msg, level=log.DEBUG, spider=spider)
raise IgnoreRequest()
To use it, add it to settings.py:
DOWNLOADER_MIDDLEWARES = {
'[project_name].middlewares.FilterResponses': 999,
}

May be it is to late. You can use the Accept header to filter the data that you are looking for.

The solution is to setup a Node.js proxy and configure Scrapy to use it through http_proxy environment variable.
What the proxy should do is:
Take HTTP requests from Scrapy and sends it to the server being crawled. Then it gives back the response from to Scrapy i.e. intercept all HTTP traffic.
For binary files (based on a heuristic you implement) it sends 403 Forbidden error to Scrapy and immediate closes the request/response. This helps to save time, traffic and Scrapy won't crash.
Sample Proxy Code
That actually works!
http.createServer(function(clientReq, clientRes) {
var options = {
host: clientReq.headers['host'],
port: 80,
path: clientReq.url,
method: clientReq.method,
headers: clientReq.headers
};
var fullUrl = clientReq.headers['host'] + clientReq.url;
var proxyReq = http.request(options, function(proxyRes) {
var contentType = proxyRes.headers['content-type'] || '';
if (!contentType.startsWith('text/')) {
proxyRes.destroy();
var httpForbidden = 403;
clientRes.writeHead(httpForbidden);
clientRes.write('Binary download is disabled.');
clientRes.end();
}
clientRes.writeHead(proxyRes.statusCode, proxyRes.headers);
proxyRes.pipe(clientRes);
});
proxyReq.on('error', function(e) {
console.log('problem with clientReq: ' + e.message);
});
proxyReq.end();
}).listen(8080);