I am working on a Braintree subscription search in order to retrieve customer IDs and subscription prices tied to these IDs. In my code I am following the suggestions from this post.
Here is an excerpt of my code:
gateway = braintree.BraintreeGateway(
braintree.Configuration(
environment=braintree.Environment.Production,
merchant_id= 'our_merchant_id',
public_key='our_public_key',
private_key='our_private_key'
)
)
subscriptions = gateway.subscription.search(
braintree.SubscriptionSearch.status.in_list(
braintree.Subscription.Status.Active,
braintree.Subscription.Status.PastDue,
braintree.Subscription.Status.Pending
)
)
result = {}
for subscription in subscriptions.items:
payment_method = gateway.payment_method.find(subscription.payment_method_token)
result[payment_method.customer_id] = subscription.price
"""do something with result """
This approach works fine in the BT sandbox and on small queries around 100 records. However, whenever I try to query for more than about 120 subscriptions the BT server responds consistently with a 504 error. I would like to query for around 5000 subscriptions at a time in production. Any suggestions?
Traceback:
Traceback (most recent call last):
File "BT_subscrAmount.py", line 22, in <module>
for subscription in subscriptions.items:
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/resource_collection.py", line 38, in items
for item in self.__method(self.__query, batch):
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/subscription_gateway.py", line 79, in __fetch
response = self.config.http().post(self.config.base_merchant_path() +
"/subscriptions/advanced_search", {"search": criteria})
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/util/http.py", line 56, in post
return self.__http_do("POST", path, Http.ContentType.Xml, params)
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/util/http.py", line 86, in __http_do
Http.raise_exception_from_status(status)
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/util/http.py", line 49, in raise_exception_from_status
raise UnexpectedError("Unexpected HTTP_RESPONSE " + str(status))
braintree.exceptions.unexpected_error.UnexpectedError: Unexpected
HTTP_RESPONSE 504
With the help from an amazing BT support engineer, we were finally able to figure out a solution to our problem.
The explanation:
When BT returns search results, they gather all associated data and return a large (serialized) response object to the client. In our case some of the subscriptions had a large number of associated transactions, which makes the process of fetching, serializing and returning them to the client slow. When making a request, the BT gateway returns pages in groups of 50 with a default timeout of 60 seconds. Any group with several large response objects will likely time out and produce a 504 error.
The solution:
To avoid this delay, instead of using the items method, we can use the id method from the ResourceCollections class and return just the ids of the objects. With the individual subscription id, we can fetch the individual subscription and only the attributes we need, like so:
subscriptions = gateway.subscription.search(
braintree.SubscriptionSearch.status.in_list(
braintree.Subscription.Status.Active
)
)
subscription_ids = subscriptions.ids
def find_customer_id(id):
subscription = gateway.subscription.find(id)
payment_method = gateway.payment_method.find(subscription.payment_method_token)
return (payment_method.customer_id, subscription.price)
Hope this will help someone else!
Split your queries into small batches (5000+ subscriptions to become 50 calls of 100 subscriptions) and then aggregate as you get the responses back. Many APIs have rate-limits and response limits hard-coded.
Related
I'm trying to retrieve the most recent 3200 tweets from a user with the Tweepy v2 API. I authenticate fine and find the user with:
client = tweepy.Client(bearer_token=bearer_token,....
user = client.get_user(username=screen_name)
I can't seem to use a Cursor to get the tweets like this:
tweets = tweepy.Cursor(client.get_users_tweets,
id=user.data.id,
count=200).items(3200)
The error is "tweepy.errors.TweepyException: This method does not perform pagination". The Tweepy docs say that get_users_tweets does support pagination, though.
ETA traceback:
Traceback (most recent call last):
File "/Users/sean/projects/misc/./tweet-dump.py", line 71, in <module>
get_all_tweets("Daily_Epsilon")
File "/Users/sean/projects/misc/./tweet-dump.py", line 23, in get_all_tweets
tweets = tweepy.Cursor(client.get_users_tweets,
File "/usr/local/lib/python3.9/site-packages/tweepy/cursor.py", line 40, in __init__
raise TweepyException('This method does not perform pagination')
tweepy.errors.TweepyException: This method does not perform pagination
The tweepy.Cursor handles pagination for the Twitter API V1.1.
Since you are using Twitter API V2, you have to use a tweepy.Paginator (see here).
So your code should be something like that:
# Please note that the count argument is called max_results in this method
paginator = tweepy.Paginator(client.get_users_tweets, id=user.data.id, max_results=200)
for tweet in paginator.flatten(limit=3200):
print(tweet)
Boto provides access to most Amazon MWS APIs, but not for GetLowestPricedOffersForSKU. I tried to hack one, but it generates an Invalid MarketplaceId error.
Boto has code for a very similarly structured API -- GetLowestOfferListingsForSKU:
#requires(['MarketplaceId', 'SellerSKUList'])
#structured_lists('SellerSKUList.SellerSKU')
#api_action('Products', 20, 5, 'GetLowestOfferListingsForSKU')
def get_lowest_offer_listings_for_sku(self, request, response, **kw):
"""Returns the lowest price offer listings for a specific
product by item condition and SellerSKUs.
"""
return self._post_request(request, kw, response)
So I modified the #api_action to change the MWS Operation to GetLowestPricedOffersForSKU:
### MINE ###
#requires(['MarketplaceId', 'SellerSKUList'])
#structured_lists('SellerSKUList.SellerSKU')
#api_action('Products', 20, 5, 'GetLowestPricedOffersForSKU')
def get_lowest_priced_offers_for_sku(self, request, response, **kw):
return self._post_request(request, kw, response)
I call this method as follows:
conn = connection.MWSConnection(
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
Merchant=ACCOUNT_ID
)
response = conn.get_lowest_priced_offers_for_sku(
MarketplaceId=marketplace_id, SellerSKUList=sku_list, ItemCondition=condition
)
When I call get_lowest_priced_offers_for_sku, I get an Invalid MarketplaceId error. If the only change I make is to call get_lowest_offer_listings_for_sku -- leaving every variable identical -- the code works find and return a valid response object. This works just fine:
response = conn.get_lowest_offer_listings_for_sku(
MarketplaceId=marketplace_id, SellerSKUList=sku_list, ItemCondition=condition
)
What do I need to do to access the Amazon MWS GetLowestPricedOffersForSKU via boto?
Not sure and neither I am python programmer, but In PHP AmazonMWS API I use code below where I use setMarketplaceId()
$request = new MarketplaceWebServiceProducts_Model_GetLowestPricedOffersForSKURequest();
$request->setSellerId($this->seller_id);
$request->setMarketplaceId($this->marketplace_id);
$request->setItemCondition("New");
Boto2 doesnt support GetLowestPricedOffersForSKU as you can see from the documentation http://docs.pythonboto.org/en/latest/ref/mws.html
I am having trouble with the getting the correct response for a Python Query on Google App Engine.
Here is the LOG from GAE, it successfully prints out the object that matches the query parameters, but now I am trying to send that objects data as JSON back. I am creating a rudimentary user auth system that queries for a USER object using email and password, and if it exists than it returns all the data back.
How can I break down this query to return the data that was found?
E 00:21:06.352 Encountered unexpected error from ProtoRPC method implementation: AttributeError ('list' object has no attribute 'email')
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0/protorpc/wsgi/service.py", line 181, in protorpc_service_app
response = method(instance, request)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/endpoints-1.0/endpoints/api_config.py", line 1332, in invoke_remote
return remote_method(service_instance, request)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0/protorpc/remote.py", line 412, in invoke_remote_method
response = method(service_instance, request)
File "/base/data/home/apps/s~caramel-theory-800/1.381271940209425490/photoswap_api.py", line 34, in user_auth
return UserCreateResponseMessage(email=query.email, password=query.password, username=query.username,
AttributeError: 'list' object has no attribute 'email'
E 00:21:06.360 [User(key=Key('User', 5086441721823232), email=u'pop', password=u'top', username=u'yop')]
Here is the ENDPOINT API
I believe the issue is within the last line of code where it trys to return the results of the query (UserAuthResponseMessage)..
#endpoints.method(UserAuthRequestMessage, UserAuthResponseMessage,
path='user', http_method='GET',
name='user.auth')
def user_auth(self, request):
# create some type of query to check for email address, and than check to see if passwords match
query = User.query(User.email == request.email, User.password == request.password).fetch()
print query
# return the info from the server
return UserCreateResponseMessage(email=query[0].email, password=query[0].password, username=query[0].username,
id=query[0].key.id())
APPLICATION = endpoints.api_server([PhotoswapAPI], restricted=False)
You problem is your are performing a query and expecting to reference the email property of the query which of course it doesn't have one.
If you expect only one result from the query then you should be using get rather than index addressing.
The other problem is the user_auth method included doesn't match your stack trace. So there is a problem here.
With the help of Tim Hoffman this is the solution that I was able to come up with.
I was returning the wrong ResponseMessage, and I was better off using .get instead of .fetch as there should only be 1 result returned.
#endpoints.api(name='photoswap', version='v1')
class PhotoswapAPI(remote.Service):
#endpoints.method(UserCreateRequestMessage, UserCreateResponseMessage,
path='user', http_method='POST',
name='user.create')
def user_create(self, request):
entity = User(email=request.email, password=request.password, username=request.username)
entity.put()
return UserCreateResponseMessage(email=entity.email, password=entity.password, username=entity.username,
id=entity.key.id())
#endpoints.method(UserAuthRequestMessage, UserAuthResponseMessage,
path='user', http_method='GET',
name='user.auth')
def user_auth(self, request):
# create some type of query to check for email address, and than check to see if passwords match
query = User.query(User.email == request.email, User.password == request.password).get()
print query
# return the info from the server
return UserAuthResponseMessage(email=query.email, password=query.password, username=query.username, id=query.key.id())
APPLICATION = endpoints.api_server([PhotoswapAPI], restricted=False)
I have a working python program that is fetching a large volume of data via SOAP using suds. The web service is implemented with a paging function such that I can grab nnn rows with each fetch call and grab the next nnn with subsequent calls. If I authenticate to the HTTP server with code like the following
client = suds.client.Client(url=url, location=location, username=username, password=password, timeout=timeout)
everything works great. If, however, I use the following
t = suds.transport.https.HttpAuthenticated(username=username, password=password)
t.handler = urllib2.HTTPBasicAuthHandler(t.pm)
t.urlopener = urllib2.build_opener(t.handler)
client = suds.client.Client(url=url, location=location, timeout=timeout, transport=t)
it works for exactly 6 iterations. That is if I specify a fetch limit of 10 rows per fetch, I get back 60 rows. On the seventh fetch, I receive
File "build/bdist.linux-i686/egg/suds/client.py", line 542, in __call__
File "build/bdist.linux-i686/egg/suds/client.py", line 602, in invoke
File "build/bdist.linux-i686/egg/suds/client.py", line 649, in send
File "build/bdist.linux-i686/egg/suds/client.py", line 698, in failed
AttributeError: 'NoneType' object has no attribute 'read'
Does anyone have any suggestions as to what might be causing this. It is definitely this change that is causing the problem. I can swap authentication styles back and forth and it is completely reproducible.
I am running python 2.6.6 with suds 0.4.
Thanks
The problem seems to be that an urllib2.HTTPError is being raised from a lower level, and its fp attribute is None:
Line 81 in suds.transport.http:
except u2.HTTPError, e:
if e.code in (202,204):
result = None
else:
raise TransportError(e.msg, e.code, e.fp)
That exception eventually gets passed to line 698 in suds.client where that error.fp.read() line blows up:
def failed(self, binding, error):
status, reason = (error.httpcode, tostr(error))
reply = error.fp.read()
I'd suggest monkey-patching the suds.SoapClient class to get the HTTP error code and message. Add these lines before you construct your suds.Client, then run it to see what HTTP error the 7th fetch raises:
class TestClient(suds.client.SoapClient):
def failed(self, binding, error):
print error.httpcode, error.args
suds.client.SoapClient = TestClient
I'm trying to use the Yahoo Social Python SDK to get a users contacts through oAuth. This is for a webapp running on App Engine. SO, I have everything set up to run through the oAuth dance, exchanging consumer keys and verifiers and all that jazz. I store the token and can reuse it to retrieve a users contacts until the token the expires an hour later. So, is there anyone out there who has used the Python SDK and can tell me what is wrong with this simple code:
import yahoo.application
CONSUMER_KEY = '####'
CONSUMER_SECRET = '##'
APPLICATION_ID = '##'
CALLBACK_URL = '##'
oauthapp = yahoo.application.OAuthApplication(CONSUMER_KEY, CONSUMER_SECRET, APPLICATION_ID, CALLBACK_URL)
oauthapp.token = yahoo.oauth.AccessToken.from_string(access_token) #access_token is legit string pulled from datastore
oauthapp.token = oauthapp.refresh_access_token(oauthapp.token)
contacts = oauthapp.getContacts()
Running this throws the following error:
'oauth_token'<br>
Traceback (most recent call last):<br>
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 513, in __call__<br>
handler.post(*groups)<br>
File "/base/data/home/apps/testproj/2.345270664321958961/scripteditor.py", line 1249, in post<br>
oauthapp.token = oauthapp.refresh_access_token(oauthapp.token)<br>
File "/base/data/home/apps/testproj/2.345270664321958961/yahoo/application.py", line 90, in refresh_access_token<br>
self.token = self.client.fetch_access_token(request)<br>
File "/base/data/home/apps/testproj/2.345270664321958961/yahoo/oauth.py", line 165, in fetch_access_token<br>
return AccessToken.from_string(self.connection.getresponse().read().strip())<br>
File "/base/data/home/apps/testproj/2.345270664321958961/yahoo/oauth.py", line 130, in from_string<br>
key = params['oauth_token'][0]<br>
KeyError: 'oauth_token'<br>
Basically, if I comment out the line with refresh_access_token, and the token has not expired, this code works and I get the users contacts. But with refresh_acces_token, it fails at that line. Can anyone give a hand?
Looks like something wrong with passing params. Try to debug oauth_token variable.
Solved. For reasons I can't understand, the above code now just works. It might have been a problem on yahoo's end, but I really can't be sure. It's been running fine for two weeks.