We Keep Coding

Braintree subscription search

I am working on a Braintree subscription search in order to retrieve customer IDs and subscription prices tied to these IDs. In my code I am following the suggestions from this post.
Here is an excerpt of my code:
gateway = braintree.BraintreeGateway(
braintree.Configuration(
environment=braintree.Environment.Production,
merchant_id= 'our_merchant_id',
public_key='our_public_key',
private_key='our_private_key'
)
)
subscriptions = gateway.subscription.search(
braintree.SubscriptionSearch.status.in_list(
braintree.Subscription.Status.Active,
braintree.Subscription.Status.PastDue,
braintree.Subscription.Status.Pending
)
)
result = {}
for subscription in subscriptions.items:
payment_method = gateway.payment_method.find(subscription.payment_method_token)
result[payment_method.customer_id] = subscription.price
"""do something with result """
This approach works fine in the BT sandbox and on small queries around 100 records. However, whenever I try to query for more than about 120 subscriptions the BT server responds consistently with a 504 error. I would like to query for around 5000 subscriptions at a time in production. Any suggestions?
Traceback:
Traceback (most recent call last):
File "BT_subscrAmount.py", line 22, in <module>
for subscription in subscriptions.items:
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/resource_collection.py", line 38, in items
for item in self.__method(self.__query, batch):
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/subscription_gateway.py", line 79, in __fetch
response = self.config.http().post(self.config.base_merchant_path() +
"/subscriptions/advanced_search", {"search": criteria})
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/util/http.py", line 56, in post
return self.__http_do("POST", path, Http.ContentType.Xml, params)
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/util/http.py", line 86, in __http_do
Http.raise_exception_from_status(status)
File "/home/karen/miniconda3/lib/python3.6/site-
packages/braintree/util/http.py", line 49, in raise_exception_from_status
raise UnexpectedError("Unexpected HTTP_RESPONSE " + str(status))
braintree.exceptions.unexpected_error.UnexpectedError: Unexpected
HTTP_RESPONSE 504

With the help from an amazing BT support engineer, we were finally able to figure out a solution to our problem.
The explanation:
When BT returns search results, they gather all associated data and return a large (serialized) response object to the client. In our case some of the subscriptions had a large number of associated transactions, which makes the process of fetching, serializing and returning them to the client slow. When making a request, the BT gateway returns pages in groups of 50 with a default timeout of 60 seconds. Any group with several large response objects will likely time out and produce a 504 error.
The solution:
To avoid this delay, instead of using the items method, we can use the id method from the ResourceCollections class and return just the ids of the objects. With the individual subscription id, we can fetch the individual subscription and only the attributes we need, like so:
subscriptions = gateway.subscription.search(
braintree.SubscriptionSearch.status.in_list(
braintree.Subscription.Status.Active
)
)
subscription_ids = subscriptions.ids
def find_customer_id(id):
subscription = gateway.subscription.find(id)
payment_method = gateway.payment_method.find(subscription.payment_method_token)
return (payment_method.customer_id, subscription.price)
Hope this will help someone else!

Split your queries into small batches (5000+ subscriptions to become 50 calls of 100 subscriptions) and then aggregate as you get the responses back. Many APIs have rate-limits and response limits hard-coded.

"IncompleteRead" Error when retrieving Twitter Data using Python

While running this program to retrieve Twitter data using Python 2.7.8 :
#imports
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
#setting up the keys
consumer_key = '…………...'
consumer_secret = '………...'
access_token = '…………...'
access_secret = '……………..'
class TweetListener(StreamListener):
# A listener handles tweets are the received from the stream.
#This is a basic listener that just prints received tweets to standard output
def on_data(self, data):
print (data)
return True
def on_error(self, status):
print (status)
#printing all the tweets to the standard output
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
stream = Stream(auth, TweetListener())
t = u"سوريا"
stream.filter(track=[t])
after running this program for 5 hours i got this Error message:
Traceback (most recent call last):
File "/Users/Mona/Desktop/twitter.py", line 32, in <module>
stream.filter(track=[t])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 316, in filter
self._start(async)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 237, in _start
self._run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 173, in _run
self._read_loop(resp)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 225, in _read_loop
next_status_obj = resp.read( int(delimited_string) )
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 543, in read
return self._read_chunked(amt)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 612, in _read_chunked
value.append(self._safe_read(chunk_left))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 660, in _safe_read
raise IncompleteRead(''.join(s), amt)
IncompleteRead: IncompleteRead(0 bytes read, 976 more expected)
>>>
Actually i don't know what to do with this problem !!!

You should check to see if you're failing to process tweets quickly enough using the stall_warnings parameter.
stream.filter(track=[t], stall_warnings=True)
These messages are handled by Tweepy (check out implementation here) and will inform you if you're falling behind. Falling behind means that you're unable to process tweets as quickly as the Twitter API is sending them to you. From the Twitter docs:
Setting this parameter to the string true will cause periodic messages to be delivered if the client is in danger of being disconnected. These messages are only sent when the client is falling behind, and will occur at a maximum rate of about once every 5 minutes.
In theory, you should receive a disconnect message from the API in this situation. However, that is not always the case:
The streaming API will attempt to deliver a message indicating why a stream was closed. Note that if the disconnect was due to network issues or a client reading too slowly, it is possible that this message will not be received.
The IncompleteRead could also be due to a temporary network issue and may never happen again. If it happens reproducibly after about 5 hours though, falling behind is a pretty good bet.

I've just had this problem. The other answer is factually correct, in that it's almost certainly:
Your program isn't keeping up with the stream
you get a stall warning if that's the case.
In my case, I was reading the tweets into postgres for later analysis, across a fairly dense geographic area, as well as keywords (London, in fact, and about 100 keywords). It's quite possible that, even though you're just printing it, your local machine is doing a bunch of other things, and system processes get priority, so the tweets will back up until Twitter disconnects you. (This is typically manifests as an apparent memory leak - the program increases in size until it gets killed, or twitter disconnects - whichever is first.)
The thing that made sense here was to push off the processing to a queue. So, I used a redis and django-rq solution - it took about 3 hours to implement on dev and then my production server, including researching, installing, rejigging existing code, being stupid about my installation, testing, and misspelling things as I went.
Install redis on your machine
Start the redis server
Install Django-RQ (or just Install RQ if you're working solely in python)
Now, in your django directory (where appropriate - ymmv for straight python applications) run:
python manage.py rqworker &
You now have a queue! You can add jobs to that like by changing your handler like this:
(At top of file)
import django_rq
Then in your handler section:
def on_data(self, data):
django_rq.enqueue(print, data)
return True
As an aside - if you're interested in stuff emanating from Syria, rather than just mentioning Syria, then you could add to the filter like this:
stream.filter(track=[t], locations=[35.6626, 32.7930, 42.4302, 37.2182]
That's a very rough geobox centred on Syria, but which will pick up bits of Iraq/Turkey around the edges. Since this is an optional extra, it's worth pointing this out:
Bounding boxes do not act as filters for other filter parameters. For
example track=twitter&locations=-122.75,36.8,-121.75,37.8 would match
any tweets containing the term Twitter (even non-geo tweets) OR coming
from the San Francisco area.
From this answer, which helped me, and the twitter docs.
Edit: I see from your subsequent posts that you're still going down the road of using Twitter API, so hopefully you got this sorted anyway, but hopefully this will be useful for someone else! :)

This worked for me.
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
while True:
try:
stream.filter(track=['python', 'java'], stall_warnings=True)
except (ProtocolError, AttributeError):
continue

A solution is restarting the stream immediately after catching exception.
# imports
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
# setting up the keys
consumer_key = "XXXXX"
consumer_secret = "XXXXX"
access_token = "XXXXXX"
access_secret = "XXXXX"
# printing all the tweets to the standard output
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
class TweetListener(StreamListener):
# A listener handles tweets are the received from the stream.
# This is a basic listener that just prints received tweets to standard output
def on_data(self, data):
print(data)
return True
def on_exception(self, exception):
print('exception', exception)
start_stream()
def on_error(self, status):
print(status)
def start_stream():
stream = Stream(auth, TweetListener())
t = u"سوريا"
stream.filter(track=[t])
start_stream()

For me the back end application to which the URL is pointing is directly returning the string
I changed it to
return Response(response=original_message, status=200, content_type='application/text')
in the start I just returned text like
return original_message
I think this answer works only for my case

Send NULL message in PyZMQ

I'm trying to translate the following example HTTP server based on 0MQ written in C seen here: Hintjens' blog into Python.
def test_http_socket():
ctx = zmq.Context()
sock = ctx.socket(zmq.ROUTER)
sock.setsockopt(zmq.ROUTER_RAW, 1)
sock.bind("tcp://*:8080")
while True:
id_bytes = sock.recv()
print("id=",id_bytes)
request = sock.recv()
print("request=",request)
if b'/close' in request:
return
sock.send(id_bytes, zmq.SNDMORE)
sock.send(b"""HTTP/1.0 200 OK
Content-Type: text/plain
Hello, world !""")
sock.send(zmq.NULL)
The problem is on the next line, where I try to translate the C expression
zmq_send (router, NULL, 0, 0);
In Python I have the following stack trace:
Traceback (most recent call last):
File "<pyshell#96>", line 1, in <module>
test_http_socket()
File "<pyshell#95>", line 18, in test_http_socket
sock.send(zmq.NULL)
File "socket.pyx", line 565, in zmq.backend.cython.socket.Socket.send (zmq\backend\cython\socket.c:5104)
File "socket.pyx", line 612, in zmq.backend.cython.socket.Socket.send (zmq\backend\cython\socket.c:4868)
File "socket.pyx", line 168, in zmq.backend.cython.socket._send_copy (zmq\backend\cython\socket.c:1914)
File "buffers.pxd", line 200, in buffers.asbuffer_r (zmq\backend\cython\socket.c:6833)
File "buffers.pxd", line 151, in buffers.asbuffer (zmq\backend\cython\socket.c:6270)
TypeError: 0 does not provide a buffer interface.
In fact, sock.send can only be used to send buffers or messages.
Is there a way to use 0MQ's zmq_send in another manner from Python, to send a NULL frame ?
By the way, a comment in Github entry shows that it may not close the connection to the client even in C.
How can I ask the remote client to close its connection (in Python) ?
I'm using libzmq 4.0.1 and PyZMQ 14.0.1 over Python 3.3.2 on Windows 32bits.

First point: zmq.NULL is the ZMQ_NULL constant, for use in zeromq's security mechanism. For example:
socket.mechanism = zmq.NULL # or zmq.PLAIN or zmq.CURVE
It is not the NULL special C constant.
To send an empty message, simply send an empty bytestring:
socket.send(b'')
The second point is that you need to send the empty frame as a separate message, which you are not doing.
Here is a working example:
def http_hello_world():
ctx = zmq.Context()
sock = ctx.socket(zmq.ROUTER)
sock.router_raw = True
sock.bind("tcp://*:8080")
while True:
id_bytes, request = sock.recv_multipart()
print("id: %r" % id_bytes)
print("request:", request.decode('utf8'))
if b'/close' in request:
return
# send the body of the response
sock.send_multipart([
id_bytes,
b"""HTTP/1.0 200 OK
Content-Type: text/plain
Hello, world!
"""
])
# send an empty message to finish the response
sock.send_multipart([
id_bytes,
b''
])
By the way, a comment in Github entry shows that it may not close the connection to the client even in C. How can I ask the remote client to close its connection (in Python) ?
I think, as long as you send the empty frame to terminate the message, connections should be closed.

googlevoice will not programmatically login (Python)

I am getting the following error traceback when my program tries to login to GoogleVoice (from googlevoice import Voice, util) to send SMS message.
File "C:\Users\ble1usb\Dropbox\Git\ers-dataanalyzzer\MainFrame.py", line 38, in call_mainframe
ah.compare_status() # compares current status with historical status. alerts alarm team if necessary
File "C:\Users\ble1usb\Dropbox\Git\ers-dataanalyzzer\alarm_handler.py", line 61, in compare_status
self.megaphone = megaphone.MegaPhone() # Am I going to have problems putting this here? I am getting relentless login fails due to the shitty googlevoice login
File "C:\Users\ble1usb\Dropbox\Git\ers-dataanalyzzer\megaphone.py", line 18, in __init__
self.voice.login(bl_google_credentials[0], bl_google_credentials[1])
File "C:\Python27\lib\site-packages\googlevoice\voice.py", line 70, in login
galx = re.search(r"name=\"GALX\"\s+value=\"(.+)\"", content).group(1)
AttributeError: 'NoneType' object has no attribute 'group'
My program has been running successfully for the last several weeks. Every once in a while, the above error would get thrown and the error handling would just try again. Now, it has not had a successful login in several hundred tries.
One issue that I feel may be important is that the program was logging in every ten (10) minutes regardless of whether or not an SMS was sent (rare case, every few days at most).
In the above traceback, You can see that I moved the function call that performed the GoogleVoice login to within the loop only in case its needed. It is still having problems because there is an alarm notification that needs to be sent out.
I tried logging in and out of my Google account, no avail.
Here's a clue --> the following code was copied out of the file identified in the last line of the traceback (source of error):
def login(self, email=None, passwd=None):
"""
Login to the service using your Google Voice account
Credentials will be prompted for if not given as args or in the ``~/.gvoice`` config file
"""
if hasattr(self, '_special') and getattr(self, '_special'):
return self
if email is None:
email = config.email
if email is None:
email = input('Email address: ')
if passwd is None:
passwd = config.password
if passwd is None:
from getpass import getpass
passwd = getpass()
content = self.__do_page('login').read()
# holy hackjob
galx = re.search(r"name=\"GALX\"\s+value=\"(.+)\"", content).group(1)
self.__do_page('login', {'Email': email, 'Passwd': passwd, 'GALX': galx})
del email, passwd
try:
assert self.special
except (AssertionError, AttributeError):
raise LoginError
return self
We should note that
galx = re.search(r"name=\"GALX\"\s+value=\"(.+)\"", content).group(1)
is the source of the error and that (this is important) the line immediately above it says
# holy hackjob

I have been able to correct the issue. It appears that Google did make a modification:
The patch is here: http://pastebin.com/bxvNjj00
Or you can simply modify voice.py and change the offending line that begins with galx = to this:
galx = re.search(r"name=\"GALX\" type=\"hidden\"\n *value=\"(.+)\"", content).group(1)

I'm not very well versed in regex, so I rewrote mine to dice up the string. In case the new modification breaks in the future, a try except statement can be used like this:
try:
galx = re.search(r"name=\"GALX\" type=\"hidden\"\n *value=\"(.+)\"", content).group(1)
except:
galx = ''.join(e for e in content if e.isalnum()) # Remove special characters (leaving only letters & numbers)
galx = galx[galx.index("GALX"):] # Grab everything from GALX forward
galx = galx[:galx.index("input")] # Truncate at input (first word after GALX value)
galx = galx[galx.index("value")+5:] # Extract GALX value

Difficulties with sending mail from GAE

I'm trying to send an email from within Google App Engine.
Whenever I try to send, I get a "Missing subject error".
When I run the code on the development server, it seems to work fine - the console output looks right and I'm taken to the page that I was expecting. But when I upload it and run it, I get:
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 636, in __call__
handler.post(*groups)
File "/base/data/home/apps/spam-tool/1.349401522260793315/spam-tool.py", line 34, in post
body=cgi.escape(self.request.get('content')))
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 297, in send_mail
message.send(make_sync_call)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 799, in send
raise ERROR_MAP[e.application_error](e.error_detail)
BadRequestError: Missing subject
But there's definitely a subject.
The code I'm using is:
> message = mail.EmailMessage(sender="admin_address#gmail.com>",
subject="test")
message.subject=self.request.get('content')
message.to = addr_to_send_to
message.body = self.request.get('content')
message.send()
(Yes, the subject is set twice... I've tried only setting it in one place or the other, and neither worked.)
Thanks in advance.

Try this:
def email_member(request):
to_addr = request.POST.get('email_to')
if not mail.is_email_valid(to_addr):
# Return an error message...
pass
else:
# note we can also send html email, e.g. html='<html><head></head><body>Hello world</body><html>'
mail.send_mail(
sender='admin#yourdomain.com',
to=to_addr,
subject=request.POST.get('email_title'),
body=request.POST.get('email_body'))
I suggest you also send your email within the taskqueue, so you could setup something like this, you could pass over a unique parameter so that you are less likely to be hacked.
def mail_member_information(email_to, email_title, email_body):
taskqueue.add(
url = 'email_message',
params =
{
'email_to': email_to,
'email_title': email_title,
'email_body': email_body,
'valid_guid': 'ae9e34ca-a2c5-476e-b8be-72d807e3dc6b'
}
)
If you just want to send to the administrators then use this:
mail.send_mail_to_admins(sender='admin#yourdomain.com',
subject=request.POST.get('email_title'),
body=request.POST.get('email_body'),
html=request.POST.get('email_body'))

I don't know if it is what is throwing you off... but sender should be like this:
sender = '<admin_address#gmail.com>',
you are missing the '<' for the sender. Sometimes it throws an error downstream of where the actual error is.
Also, make sure the admin_address#gmail.com is a Google Account that has been added as a developer for that app.

I think you can send like this, make sure that sender is a admin of the app
from google.appengine.api import mail
mail.send_mail(sender="something#gmail.com",
to=email_valid.email,
subject="Please validate your email",
body=body_text,
html=htmlbody_text)