I am working on a site using Flask that is pulling data from an API, processing it into JSON, and then dynamically loading it throughout the site as needed. I am having an issue with matching the URL appropriately while at the same time efficiently loading what data I need.
Here is my main file:
import requests
from flask import Flask, render_template
app = Flask(__name__)
url = 'https://omgvamp-hearthstone-v1.p.mashape.com/cards'
myHeaders={
"X-Mashape-Key": 'key-here',
"Accept": "application/json"
}
r = requests.get(url, headers=myHeaders)
cards = r.json()
badSets = ['Promo', 'Hall of Fame', 'Tavern Brawl', 'Hero Skins', ' Missions', 'Credits', 'System', 'Debug']
#app.route('/')
def index():
return render_template('index.html', cards=cards, badSets=badSets)
#app.route('/<setName>', methods=['GET'])
def set(setName):
return render_template('set.html', cards=cards, setName=setName, badSets=badSets)
#app.route('/<setName>/<cardName>', methods=['GET'])
def individualCard(setName, cardName):
return render_template('card.html', cards=cards, setName=setName, cardName=cardName, badSets=badSets)
First, I'd prefer to not have to pass each html page all of my data. Is there some way to pull only what is needed and ignore the rest?
Second, I would like to pass a dictionary object i.e.
if card['name'] == card[setName][--index--][cardName]:
pass card object to html page here
How could this be achieved, and is it possible without having to use a for-loop through all of my data?
I'm happy to load my html pages if needed.
Assuming the basic structure of your parsed json data looks like this (a dictionary with lists of dictionaries):
{
"Basic": [{"cardId": 4711, ...}, {"cardId": 4712, ...}, ...],
"Fancy": [...],
...
}
You could rewrite:
#app.route('/<setName>', methods=['GET'])
def set(setName):
cardset = cards.get(setName)
return render_template('set.html', cardset=cardset)
This extracts the card set we are looking for by the dictionary key, according to the assumed data structure above.
Then in the template, instead of cardName, pass the cardId and rewrite the other route:
#app.route('/<setName>/<cardId>', methods=['GET'])
def individualCard(setName, cardId):
cardset = cards.get(setName)
matches = [x for x in cardset if x['cardId'] == cardId]
card = matches[0]
return render_template('card.html', card=card)
This uses list comprehension to extract a list of matches (everything that has the cardId we are looking for) from our selected cardset, and should be a list with a single element. We return the first element to the template for rendering.
Now this obviously does not do any error checking, for example the dictionary key we passed might be wrong and not be found, the cardId might not be found, or more interestingly there might be more than one result for the cardId we passed.
But this would be the general idea on how to approach this.
Also note I've left out badSets for clarity in the example, I'm assuming this are card sets that are forbidden, or for testing purposes.
For this case, you'd want to check the dictionary key first before looking up the set, and show an error page or something maybe:
#app.route('/<setName>', methods=['GET'])
def set(setName):
if setName in badSets:
return render_template('error.html')
cardset = cards.get(setName)
return render_template('set.html', cardset=cardset)
Disclaimer: This is coming purely from memory and it is late, so there might be an error here or there...
Related
Question
I have created a dynamic route as /update/<randomString> in my Flask app.py file, where randomString is a randomly generated string with the length of 50. However if I search for /update/1 I am able to view the same dynamic route /update/<randomString> without any error! Can anyone explain why is it so?
See what I've tried so far:
#app.route('/')
def index():
randomString = ''.join(secrets.choice(string.ascii_uppercase+string.digits+string.ascii_lowercase) for k in range (50))
session['randomString'] = str(randomString)
return render_template('index.html')
#app.route('/update/<randomString>')
def update(randomString):
if 'randomString' in session:
randomString = session['randomString']
return render_template('update.html')
else:
return 'error...'
Link of the dynamic page at update.html page, where random string is passed with the help of session(defined at index.html page).
Dynamic page
Edit: I am also able to view dynamic route when I click on the link defined above and my URL section shows that long randomString. Problem is: I can access the same route when I search for http://127.0.0.1:5000/update/1
Screenshot one
Screenshot two
While storing the random string, the key you use is randomString. So you are storing the random string in a dict like
session['randomString'] = '1234567890'
Then when you access the session in the /update route you are just checking if session has a key named randomString. You should also check if session['randomString'] == '1234567890' and render the page only if the random string in session is the same as you created in the / path. You can replace the if with
if 'randomString' in session and session['randomString'] == randomString :
I'm trying to generate a filter for SQL alchemy DB. I'm able to run the query correctly and get the desired result however I need to get the requests correctly and format it accordingly.
I'm hitting the url like:
http://localhost:3000/books?filter=price,eq,37&filter=code,eq,50
When I get the args they come out as:
ImmutableMultiDict([('filter', 'price,eq,37'), ('filter', 'code,eq,50')])
But what I really want is:
{"filter": ["price,eq,37", "code,eq,50"]}
Getting the request like:
#app.route('/books', methods=['GET'])
def books() -> str:
args = request.args
print(args)
Any help would be greatly appreciated.
You can pass flat=False parameter to the request.args.to_dict() method for getting multiple query values for a single key.
Example:
#app.route('/')
def index():
query_params = request.args.to_dict(flat=False)
# print(query_params)
return query_params
Kindly do upvote the solution, If you find it helpful.
Method lists() returns an iterable of (key, values):
dict(request.args.lists()) # returns {"filter": ["price,eq,37", "code,eq,50"]}
Method getlist() returns all values of a key:
request.args.getlist('filter') # returns ["price,eq,37", "code,eq,50"]
I am trying to write a dynamic page in Python with Flask on my pythonanywhere.com free hosting. I have the following code, hoping I could write to the resp variable to make my pages.
#app.route('/xdcc-search/search.html')
def search_app():
try:
with open('templates/xdcc-search/search.html', 'r') as dynamic:
dynamic.read()
except:
pass
dynamic.replace("<file>","MY_FILENAME.tar")
resp = make_response(render_template(dynamic), 200)
no_cache(resp)
return resp
I get an error stating dynamic is referenced before assignment. Is there a way to edit the template after render_template(filename) retreives and assembles the page?
When you do this:
with open('templates/xdcc-search/search.html', 'r') as dynamic:
dynamic.read()
...you are reading in the contents of the file, but you are throwing them away -- read() is a function that reads the contents of the file and returns them.
Fixing your code so that it actually does what you are trying to do gives this:
#app.route('/xdcc-search/search.html')
def search_app():
try:
with open('templates/xdcc-search/search.html', 'r') as dynamic:
contents = dynamic.read()
except:
pass
contents.replace("<file>","MY_FILENAME.tar")
resp = make_response(render_template(contents), 200)
no_cache(resp)
return resp
...but that is still wrong; render_template takes as its parameter the name of the file that contains the template, not its contents. So what you need to do to get this working would be to replace that render_template with render_template_string.
#app.route('/xdcc-search/search.html')
def search_app():
try:
with open('templates/xdcc-search/search.html', 'r') as dynamic:
contents = dynamic.read()
except:
pass
contents.replace("<file>","MY_FILENAME.tar")
resp = make_response(render_template_string(contents), 200)
no_cache(resp)
return resp
But this is still not using Flask templates the way they are meant to be used. The point of templates is that they should contain things in curly brackets to specify what changes should be made. Replacing a static string inside one with an explicit call to replace bypasses that and does a more primitive version of the same thing.
What you really should be doing is changing your template so that instead of having <file> inside, it, it has {{ file }}, and then you can replace all of that messy view code with this:
#app.route('/xdcc-search/search.html')
def search_app():
resp = make_response(render_template("xdcc-search/search.html", file="MY_FILENAME.tar"), 200)
no_cache(resp)
return resp
Finally, I'm not sure that you need that no_cache, as view functions are not cached by default. Also, the default status code on a response is 200. So probably all you need is this:
#app.route('/xdcc-search/search.html')
def search_app():
return render_template("xdcc-search/search.html", file="MY_FILENAME.tar")
I created a web crawler that, given a base_url, will spider out and find all possible endpoints. While I am able to get all the endpoints, I need a way to figure out how I got there in the first place -- a 'url stack-trace' persay or breadcrumbs of url's leading to each endpoint.
I first start by finding all url's given a base url. Since the sublinks I'm looking for are within a json, I thought the best way to do this would be using a variation of a recurisve dictionary example I found here: http://www.saltycrane.com/blog/2011/10/some-more-python-recursion-examples/:
import requests
import pytest
import time
BASE_URL = "https://www.my-website.com/"
def get_leaf_nodes_list(base_url):
"""
:base_url: The starting point to crawl
:return: List of all possible endpoints
"""
class Namespace(object):
# A wrapper function is used to create a Namespace instance to hold the ns.results variable
pass
ns = Namespace()
ns.results = []
r = requests.get(BASE_URL)
time.sleep(0.5) # so we don't cause a DDOS?
data = r.json()
def dict_crawler(data):
# Retrieve all nodes from nested dict
if isinstance(data, dict):
for item in data.values():
dict_crawler(item)
elif isinstance(data, list) or isinstance(data, tuple):
for item in data:
dict_crawler(item)
else:
if type(data) is unicode:
if "http" in data: # If http in value, keep going
# If data is not already in ns.results, don't append it
if str(data) not in ns.results:
ns.results.append(data)
sub_r = requests.get(data)
time.sleep(0.5) # so we don't cause a DDOS?
sub_r_data = sub_r.json()
dict_crawler(sub_r_data)
dict_crawler(data)
return ns.results
To reiterate, the get_leaf_nodes_list does a get request and looks for any values within the json for a url (if 'http' string is in the value for each key) to recursively do more get requests until there's no url's left.
So to reiterate here are the questions I have:
How do I get a linear history of all the url's I hit to get to each endpoint?
Corollary to that, how would I store this history? As the leaf nodes list grows, my process gets expontentially slower and I am wondering if there's a better data type out there to store this information or a more efficient process to the code above.
I have a Pyramid web service, and code samples are as follows:
View declaration:
#view_config(route_name="services/Prices/GetByTicker/")
def GET(request):
ticker = request.GET('ticker')
startDate = request.GET('startDate')
endDate = request.GET('endDate')
period = request.GET('period')
Routing:
config.add_route('services/Prices/GetByTicker/', 'services/Prices/GetByTicker/{ticker}/{startDate}/{endDate}/{period}')
Now I know this is all screwed up but I don't know what the convention is for Pyramid. At the moment this works inasmuch as the request gets routed to the view successfully, but then I get a "Dictionary object not callable" exception.
The URL looks horrible:
#root/services/Prices/GetByTicker/ticker=APPL/startDate=19981212/endDate=20121231/period=d
Ideally I would like to be able to use a URL something like:
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
Any Pyramid bods out there willing to take five minutes to explain what I'm doing wrong?
from you sample code, i think you use the URL Dispatch
so it should be like this
config.add_route('services/Prices/GetByTicker/', 'services/Prices/GetByTicker/')
then the URL like:
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
will match it
--edit--
you don't have to use a name like "services/Prices/GetByTicker" for route_name,and you can get the GET params use request.params['key']
View declaration:
#view_config(route_name="services_Prices_GetByTicker")
def services_Prices_GetByTicker(request):
ticker = request.params['ticker']
startDate = request.params['startDate']
endDate = request.params['endDate']
period = request.params['period']
Routing:
config.add_route('services_Prices_GetByTicker', 'services/Prices/GetByTicker/')
The query string is turned into the request.GET dictionary. You are using parenthesis to call the dictionary instead of accessing items via the brackets. For a url such as
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
request.GET['ticker'] # -> 'APPL' or an exception if not available
request.GET.get('ticker') # -> 'APPL' or None if not available
request.GET.get('ticker', 'foo') # -> 'APPL' or 'foo' if not available
request.GET.getall('ticker') # -> ['APPL'] or [] if not available
The last option is useful if you expect ticker to be supplied multiple times.
request.params is a combination of request.GET and request.POST where the latter is a dictionary representing the request's body in a form upload.
Anyway, the answer is that request.GET('ticker') syntactically is not one of the options I mentioned, stop doing it. :-)