The situation is the following: I have a Flask web server, and I need to develop a new API endpoint for it (POST/GET), which would update/read a server status.
My POST API requires an input like this:
{
"name": <name_of_the_server>,
"serverUrl" <url_of_the_server>,
"available": true/false
}
I want to save this information in Redis for a week. I save the whole JSON from the API request as a dictionary value in Redis. I dump and load it with pickle. The key is the "name" paramater.
I also have a GET API endpoint, in which I can specify the name of the server, and get the status information about it. Hence, Python checks in Redis for the key, containing the name of the server (given by the user through the API), and returns the value (the dictionary with all of the information that was saved previously).
This is all good, and works as intended. However, through the GET API, I should also be able to search for server status with the server URL (as well as the server name).
So far my solution is like this:
// Redis is a class, I previously instantiated
redis.set(server_name, server_data, ex=time_to_live)
redis.set(server_url, server_data, ex=time_to_live)
I set two keys, containing the exact same information. Through the GET endpoint, this gives me the chance to check server status by using either the server name or the server URL, as they return the same value. Also, they have the same expiry date, and are always updated together.
This works, however I feel this is more of a "cheat" solution, so I would want to ask for an advice if I should keep it that way, or if there is a better way to achieve the same functionality. Thank you in advance!
Related
Does a Session object maintain the same TCP connection with a client?
In the code below, a request from the client is submitted to a handler, the handler creates a sessions object why does session["count"] on an object give a dictionary?
A response is then given back to the client, upon another request is the code re-executed?
So that another session object is created?
How does the session store the previous count information if it did not return a cookie to the client?
from appengine_utilities import sessions
class SubmitHandler(webapp.RequestHandler):
def get(self):
session = sessions.Session()
if "count" in session:
session["count"]=session["count"]+1
else:
session["count"]=1
template_values={'message':"You have clicked:"+str(session["count"])}
# render the page using the template engine
path = os.path.join(os.path.dirname(__file__),'index.html')
self.response.out.write(template.render(path,template_values))
You made several questions so let's go one by one:
Sessions are not related to TCP connections. A TCP connection is maintained when both client and server agreed upon that using the HTTP Header keep-alive. (Quoted from Pablo Santa Cruz in this answer).
Looking at the module session.py in line 1010 under __getitem__ definition I've found the following TODO: It's broke here, but I'm not sure why, it's returning a model object. Could be something along these lines, I haven't debug it myself.
From appengine_utilities documentation sessions are stored in Datastore and Memcache or kept entirely as cookies. The first option also involves sending a token to the client to identify it in subsequent requests. Choosing one or another depends on your actual settings or the default ones if you haven't configured your own. Default settings are defined to use the Datastore option.
About code re-execution you could check that yourself adding some logging code to count how many times is the function executed.
Something important, I have noticed that this library had it's latest update on 2nd of January 2016, so it has gone unmaintained for 4 years. It would be best if you change to an up to date library, for example the webapp2 session module. Furthermore, Python 2 is sunsetting by this year (1st January 2020) so you might consider switching to python 3 instead.
PD: I found the exact code you posted under this website. In case you took it from there consider next time to include a reference/citation to it's origin.
I simply want to receive notifications from dropbox that a change has been made. I am currently following this tutorial:
https://www.dropbox.com/developers/reference/webhooks#tutorial
The GET method is done, verification is good.
However, when trying to mimic their implementation of POST, I am struggling because of a few things:
I have no idea what redis_url means in the def_process function of the tutorial.
I can't actually verify if anything is really being sent from dropbox.
Also any advice on how I can debug? I can't print anything from my program since it has to be ran on a site rather than an IDE.
Redis is a key-value store; it's just a way to cache your data throughout your application.
For example, access token that is received after oauth callback is stored:
redis_client.hset('tokens', uid, access_token)
only to be used later in process_user:
token = redis_client.hget('tokens', uid)
(code from https://github.com/dropbox/mdwebhook/blob/master/app.py as suggested by their documentation: https://www.dropbox.com/developers/reference/webhooks#webhooks)
The same goes for per-user delta cursors that are also stored.
However there are plenty of resources how to install Redis, for example:
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-redis
In this case your redis_url would be something like:
"redis://localhost:6379/"
There are also hosted solutions, e.g. http://redistogo.com/
Possible workaround would be to use database for such purpose.
As for debugging, you could use logging facility for Python, it's thread safe and capable of writing output to file stream, it should provide you with plenty information if properly used.
More info here:
https://docs.python.org/2/howto/logging.html
I'm working on an application that will have to use multiple external APIs for information and after processing the data, will output the result to a client. The client uses a web interface to query, once query is send to server, server process send requests to different API providers and after joining the responses from those APIs then return response to client.
All responses are in JSON.
current approach:
import requests
def get_results(city, country, query, type, position):
#get list of apis with authentication code for this query
apis = get_list_of_apis(type, position)
results = [ ]
for api in apis:
result = requests.get(api)
#parse json
#combine result in uniform format to display
return results
Server uses Django to generate response.
Problem with this approach
(i) This may generate huge amounts of data even though client is not interested in all.
(ii) JSON response has to be parsed based on different API specs.
How to do this efficiently?
Note: Queries are being done to serve job listings.
Most APIs of this nature allow for some sort of "paging". You should code your requests to only draw a single page from each provider. You can then consolidate the several pages locally into a single stream.
If we assume you have 3 providers, and page size is fixed at 10, you will get 30 responses. Assuming you only show 10 listings to the client, you will have to discard and re-query 20 listings. A better idea might be to locally cache the query results for a short time (say 15 minutes to an hour) so that you don't have to requery the upstream providers each time your user advances a page in the consolidated list.
As far as the different parsing required for different providers, you will have to handle that internally. Create different classes for each. The list of providers is fixed, and small, so you can code a table of which provider-url gets which class behavior.
Shameless plug but I wrote a post on how I did exactly this in Durango REST framework here.
I highly recommend using Django REST framework, it makes everything so much easier
Basically, the model on your APIs end is extremely simple and simply contains information on what external API is used and the ID for that API resource. A GenericProvider class then provides an abstract interface to perform CRUD operations on the external source. This GenericProvider uses other providers that you create and determines what provider to use via the provider field on the model. All of the data returned by the GenericProvider is then serialised as usual.
Hope this helps!
Here is my goal: I would like to be able to report various metrics to zabbix so that we can display the graphs on a web page.
These metrics include:
latency per soap service submission
various query results from one or more databases.
What things do I need to write and/or expose? Or is the zabbix server going to go and get it from an exposed service somewhere?
I've been advised that a script that returns a single value will work, but I'm wondering if that's the right way.
I can offer 2 suggestions to get the metrics into Zabbix:
Use the zabbix_sender binary to feed the data from your script directly to the Zabbix server. This allows your script to call on it's own interval and set all the parameters needed. You really only need to know the location to the zabbix_sender binary. Inside the Zabbix server interface, you would create items with the type of Zabbix trapper. This is the item type which receives values send from the zabbix_sender. You make up the key name and it has to match.
The second way you could do this is to specify a key name and script/binary inside the zabbix_agentd.conf file. Every time the Zabbix server requests this item the script would be called and the data from the script recorded. This allows you to set the intervals in the Zabbix item configuration rather than forcing you to run your script on its own intervals. However, you would need to add this extra bit of information to your zabbix_agentd.conf file for every host.
There may be other ways to do this directly from Python (zabbix_sender bindings for Python maybe?). But these are the 2 ways I have used before which work well. This isn't really Python specific. But you should be able to use zabbix_sender in your Python scripting. Hope this information helps!
Update: I also remembered that Zabbix was working on/has a API (JSON/RPC style). But the documentation site is down at the moment and I am not sure if the API is for submitting item data or not. Here is the Wiki on the API: http://www.zabbix.com/wiki/doc/api
And a project for Python API: https://github.com/gescheit/scripts/tree/master/zabbix/
There seems to be little documentation on the API as it is new as of Zabbix version 1.8
Actually there is a python binding for zabbix_sender. http://pypi.python.org/pypi/zbxsend
I have two Membase clients - a Django website and a small python test file. I observed that the data saved by the python file can't be seen by the Django website, it returns None. However, If I save the same data (same keys and values) in the Django website, the site will be able to read the keys/values then.
This is the same code I use in both clients:
import memcache
mc = memcache.Client(['127.0.0.1:11211'])
ids = ["100000314125574", "100000335352069"]
users = mc.get_multi(ids)
I am using the python-memcache client library.
I have been trying to solve this for hours. Please share your ideas. It is a real mystery to me why Django can't read the data saved from another client.
A good troubleshooting step is just to telnet manually to 127.0.0.1 on port 11211 and do a 'get ' for that key. If you get something back then at least you know the Membase server is functioning properly and you can focus on the Django side of things...where I'm much less knowledgeable.