I'm writing a script that uses the requests library to obtain API data from a website requiring authentication. From my understanding and testing, the site doesn't seem to use auth-saving cookies, so authentication is required with each request, such as this:
page = requests.get('http://a_url.com', auth=(uname, pword))
This is fine for what I need except, that I have specific tasks split up into separate functions that handle different data in different ways. The only commonality is that they all require the username and password (obtained at the command line), so I've ended up with just short of every function taking "uname" and "pword" as their first parameters, followed by any other necessary params:
def get_all_projects(uname, pword):
# function
def get_single_project(uname, pword, project_name):
# function
def get_select_projects(uname, pword, projects):
for project in projects:
get_single_project(uname, pword, project)
# Etc...
Aside from turning the script into a class, is there any other way to deliver the credentials to the functions that need them without having to parameterize them? Requests.Session() won't work, since as stated above I need to include the auth with each call to get. I feel like a decorator would fit the situation well, but I don't understand them well enough yet to be able to confirm that belief.
As #barny pointed out, I missed the fact that you can indeed pass authentication to a requests.Session() via the following (from the docs and barny's comment):
s = requests.Session()
s.auth = (uname, pword)
so my url request from the op can now just be:
page = s.get('http://a_url.com')
with the auth automatically included as long as the session is active.
Along with that, my functions can now be condensed to just using the session object as a param instead of the username and password each time:
def get_all_projects(s):
# function
def get_single_project(s, project_name):
# function
def get_select_projects(s, projects):
for project in projects:
get_single_project(s, project)
# Etc...
Related
i am trying build a github info in python, i want the script grab this things:
"stargazers_count": 2, << i need the number updated
"watchers_count": 2, << i need the number updated
"forks": 1, << i need the number updated
ex: https://api.github.com/repos/toddmotto/angular-1-5-components-app
import requests
r = requests.get('https://api.github.com/repos/toddmotto/angular-1-5-components-app')
print(r.json())
i need the results be like
stargazers_count: number
watchers_count: number
forks: number
Extract the required value using the key.
Ex:
import requests
r = requests.get('https://api.github.com/repos/toddmotto/angular-1-5-components-app').json()
print(r["stargazers_count"])
print(r["watchers_count"])
print(r["forks"])
Output:
573
573
200
r.json() provides you a python dict.
import requests
r = requests.get('https://api.github.com/repos/toddmotto/angular-1-5-components-app')
resp = r.json()
print(r['stargazers_count']) #573
print(r['watchers_count']) #573
print(r['forks']) #200
You can just put the json keys in a list and then iterate over the list picking out the values you need.
from requests import get
keys = [
"stargazers_count",
"watchers_count",
"forks"
]
response = get('https://api.github.com/repos/toddmotto/angular-1-5-components-app').json()
for key in keys:
print('{key}:\t{value}'.format(
key=key,
value=response[key]
)
)
use response.keys() if you need all keys
# prints out all keys
for key in response.keys():
print('{key}:\t{value}'.format(
key=key,
value=response[key]
)
)
Using the Requests Library in Python
First things first, let’s introduce you to Requests.
What is the Requests Resource?
Requests is an Apache2 Licensed HTTP library, written in Python. It is designed to be used by humans to interact with the language. This means you don’t have to manually add query strings to URLs, or form-encode your POST data. Don’t worry if that made no sense to you. It will in due time.
What can Requests do?
Requests will allow you to send HTTP/1.1 requests using Python. With it, you can add content like headers, form data, multipart files, and parameters via simple Python libraries. It also allows you to access the response data of Python in the same way.
In programming, a library is a collection or pre-configured selection of routines, functions, and operations that a program can use. These elements are often referred to as modules and stored in object format.
Libraries are important because you load a module and take advantage of everything it offers without explicitly linking to every program that relies on them. They are truly standalone, so you can build your own programs with them and yet they remain separate from other programs.
Think of modules as a sort of code template.
To reiterate, Requests is a Python library.
Importing the Requests Module
To work with the Requests library in Python, you must import the appropriate module. You can do this simply by adding the following code at the beginning of your script:
import requests
Of course, to do any of this – installing the library included – you need to download the necessary package first and have it accessible to the interpreter.
Making a Request
When you ping a website or portal for information this is called making a request. That is exactly what the Requests library has been designed to do.
To get a webpage you would do something like the following:
r = requests.get('https://api.github.com/repos/toddmotto/angular-1-5-components-app')
finalResult = r.json()
print "stargazers_count",finalresult["stargazers_count"]
Creating a function that will take care for different combinations of requested attributes.
Taking care of failures as well.
import requests
import pprint
def get_github_activity_attributes(url, attributes=[]):
""" Do http call and return the full response as dict or a subset of the response """
r = requests.get(url)
if r.status_code == 200:
if not attributes:
# The caller asked to return all attributes
return True, r.json()
else:
# The caller asked for subset of the attributes (assuming first level attributes only)
data = r.json()
return True, {attr: data[attr] for attr in attributes}
else:
return False, 'Request failed with status code {}.'.format(r.status_code)
ok, all_attributes = get_github_activity_attributes('https://api.github.com/repos/toddmotto/angular-1-5-components-app')
if ok:
print('All attributes below')
pprint.pprint(all_attributes)
ok, few_attributes = get_github_activity_attributes('https://api.github.com/repos/toddmotto/angular-1-5-components-app',
['forks', 'watchers_count'])
if ok:
print('Few attributes below')
pprint.pprint(few_attributes)
need the results be like
stargazers_count: number
watchers_count: number
forks: number
The simplest technique is to use PyGithub
from getpass import getpass
from github import Github
gh = Github(
login_or_token="mpenning",
password=getpass("Github password for mpenning: ")
)
repo = gh.get_repo("mpenning/ciscoconfparse")
print("stargazers_count:", repo.stargazers_count)
# There is a bug in `watchers_count`. It returns number of stargazers.
print("watchers_count:", repo.watchers_count)
print("forks:", repo.forks_count)
stdout:
% python simple_pygithub_demo.py
Github password for mpenning:
stargazers_count: 647
watchers_count: 647
forks 195
I am using the third party API for Parse called ParsePy.
https://github.com/dgrtwo/ParsePy
I am able to login, but I am trouble retrieving query for the current user.
For instance, I want to retrieve the username for the current user or a specific column, but seems to experiencing issues.
I really do not know where to start, and quite frankly I am skeptical of using the API due to stability issues.
Any help or guidance would be greatly appreciated
from parse_rest.connection import register
from parse_rest.datatypes import Object
from parse_rest.user import User
from parse_rest.connection import SessionToken
application_id = ""
rest_api_key = ""
# the application_id for the app. Obtain
# Register takes three arguments, two required and one being optional
# Master is an optional argument. Since this is a desktop-based application, the master key should NOT be included
register(application_id, rest_api_key, master_key=None)
u = User.login('123', '123')
not sure what to do next
I'm using the gdata Python library to do batched deletes of contacts, and I just get the "If-Match or If-None-Match header or entry etag attribute required" error.
I think the problem started when I had to enable the Contacts API in the console (which until a few days ago wasn't required? *).
EDIT:
It's actually failing for both updating and deleting operations. Batched insert works fine.
Tried specifying the If-Match header, but it's still failing:
custom_headers = atom.client.CustomHeaders(**{'If-Match': '*'})
request_feed = gdata.contacts.data.ContactsFeed()
request_feed.AddDelete(entry=contact, batch_id_string='delete')
response_feed = self.gd_client.ExecuteBatch(
request_feed,
'https://www.google.com/m8/feeds/contacts/default/full/batch',
custom_headers=custom_headers
)
Also created a ticket on the project page, but I doubt it will get any attention there.
EDIT 2:
Using the Batch method with force=True (which just adds the If-Match: * header) is the same result.
response_feed = self.gd_client.Batch(
request_feed,
uri='https://www.google.com/m8/feeds/contacts/default/full/batch',
force=True
)
* Can someone verify this? I never had to enable it in the console before and my app was able to use the Contacts API without problem, and I believe it wasn't even available before. I was surprised to see it yesterday.
Copying answer from the Google code ticket.
Basically, you need to patch the client's Post method to modify the request feed slightly. Here's one way to do it without directly modifying the library source:
def patched_post(client, entry, uri, auth_token=None, converter=None, desired_class=None, **kwargs):
if converter is None and desired_class is None:
desired_class = entry.__class__
http_request = atom.http_core.HttpRequest()
entry_string = entry.to_string(gdata.client.get_xml_version(client.api_version))
entry_string = entry_string.replace('ns1', 'gd') # where the magic happens
http_request.add_body_part(
entry_string,
'application/atom+xml')
return client.request(method='POST', uri=uri, auth_token=auth_token,
http_request=http_request, converter=converter,
desired_class=desired_class, **kwargs)
# when it comes time to do a batched delete/update,
# instead of calling client.ExecuteBatch, instead directly call patched_post
patched_post(client_instance, entry_feed, 'https://www.google.com/m8/feeds/contacts/default/full/batch')
The ticket referenced in the original post has some updated information and a temporary work around that allows batch deletes to succeed. So far it's working for me!
http://code.google.com/p/gdata-python-client/issues/detail?id=700
You can also specify the etag attribute to get around it. This works in the batch request payload:
<entry gd:etag="*" >
<batch:id>delete</batch:id>
<batch:operation type="delete"/>
<id> urlAsId </id>
</entry>
I'm using Pyramid with Cornice to create an API for a Backbone.js application to consume. My current code is working perfectly for GET and POST requests, but it is returning 404 errors when it receives PUT requests. I believe that this is because Backbone sends them as http://example.com/api/clients/ID, where ID is the id number of the object in question.
My Cornice setup code is:
clients = Service(name='clients', path='/api/clients', description="Clients")
#clients.get()
def get_clients(request):
...
#clients.post()
def create_client(request):
...
#clients.put()
def update_client(request):
...
It seems that Cornice only registers the path /api/clients and not /api/clients/{id}. How can I make it match both?
The documentation gives an example of a service that has both an individual path (/users/{id}) and an object path (/users). Would this work for you ?
#resource(collection_path='/users', path='/users/{id}')
A quick glance at the code for the resource decorator shows that it mainly creates two Service : one for the object and one for the collection. Your problem can probably be solved by adding another Service :
client = Service(name='client', path='/api/clients/{id}', description="Client")
I am experimenting with facebook and trying to create an event, via the Graph API. I am using django and the python-facebook-sdk from github. I can successfully post to my wall pull friends etc.
I am using django-social-auth for facebook login stuff and have settings.py for permissions:
FACEBOOK_EXTENDED_PERMISSIONS = ['publish_stream','create_event','rsvp_event']
In the graph api explorer on facebook my request works so I know what parameters to use and, well, I am using them.
Here is my python code:
def new_event(self):
event = {}
event['name'] = name
event['privacy'] = 'OPEN'
event['start_time'] = '2011-11-04T14:42Z'
event['end_time'] = '2011-11-05T14:46Z'
self.graph.put_object("me", "events", args=None, post_args=event)
The code that is calling the facebook api is roughly: (also the access_token is added to the post_args which then is converted to post_data and urlencoded.
file = urllib.urlopen("https://graph.facebook.com/me/events?" +
urllib.urlencode(args), post_data)
The error I am getting is:
Exception Value: (#100) Invalid parameter
I am trying to figure out what is wrong, but am also curios of how to figure out overall what is wrong so I can debug this in the future. it seems to be too generic of an error because I don't know what is wrong.
Not really sure how post_args works but this call did the trick
graph.put_object("me","events",start_time="2013-11-04T14:42Z", privacy="OPEN", end_time="2013-11-05T14:46Z", name="Test Event")
The invalid parameter most likely is pointing to how you are feeding the parameters as post_args. I don't think the SDK was ever designed to feed it like this. I could be mistaken as I'm not really sure what post_args would be doing.
Another way based on how put_object is setup with **data it would be
graph.put_object("me","events", **event)