We Keep Coding

tornado.httpclient AsyncHTTPClient() python3

I have a strange problem and i hope somebody encountered with it.
I'm working with TelegramAPI and i want to POST file using
multipart/form-data. File size 32K
data = {'photo': open('test.jpg', 'rb').read()}
Using simple requests python lib i have no problem:
res = requests.post(url, files=data)
BUT
When i try to use
http_client = httpclient.AsyncHTTPClient()
http_client.fetch(url, method='POST', body=urllib.parse.urlencode(data))
With the same picture
I got an error
tornado.httpclient.HTTPError: HTTP 413: Request Entity Too Large
I don't know why? requests works fine, but not AsyncHTTPClient, help me please

Please check out this demo code. You will see there an example on how to upload files.

The body argument in Tornado's HTTP client is similar to the data argument in requests. The files argument is something else entirely: it encodes the file using the multipart encoding. Which one you want to use depends on what format the server is expecting.
In this case the server is expecting multipart encoding, not URL encoding. Tornado does not have built-in support for generating multipart encoding, but as Vitalie said in the other answer, this example code shows how to do it.

Kinesis python client drops any message with "\x" escape?

I'm using boto3 (version 1.4.4) to talk to Amazon's Kinesis API:
import boto3
kinesis = boto3.client('kinesis')
# write a record with data '\x08' to the test stream
response = kinesis.put_record(StreamName='test', Data=b'\x08', PartitionKey='foobar')
print(response['ResponseMetadata']['HTTPStatusCode']) # 200
# now read from the test stream
shard_it = kinesis.get_shard_iterator(StreamName="test", ShardId='shardId-000000000000', ShardIteratorType="LATEST")["ShardIterator"]
response = kinesis.get_records(ShardIterator=shard_it, Limit=10)
print(response['ResponseMetadata']['HTTPStatusCode']) # 200
print(response['Records']) # []
When I test it with any data without the \x escape I'm able to get back the record as expected. Amazon boto3's doc says that "The data blob can be any type of data; for example, a segment from a log file, geographic/location data, website clickstream data, and so on." then why is the message with \x escaped characters dropped? Am I expected to '\x08'.encode('string_escape') before sending the data to kinesis?
If you are interested, I have characters like \x08 in the message data because I'm trying to write a serialized protocol buffer message to a Kinesis stream.

Okay so I finally figured it out. The reason why it wasn't working was because my botocore was on version 1.4.62. I only realized it because another script that ran fine on my colleague's machine was throwing exceptions on mine. We had the same boto3 version but different botocore versions. After I pip install botocore==1.5.26 both the other script and my kinesis put_record started working.
tldr: botocore 1.4.62 is horribly broken in many ways so upgrade NOW. I can't believe how much of my life is wasted by outdated broken libraries. I wonder if Amazon dev can unpublish broken versions of the client?

BigQuery Upload Encoding Error (Python 3)

I am trying to upload some data into my BigQuery tables using HTTP post (httplib2). My existing upload Python code has always worked until I moved to Python 3. Now I get encoding errors on the body of the request for the same data I was uploading successfully before using Python 2.7.x.
The error happens when the HTTP client library tries to encode the body using the default HTTP encoding of ISO-8859-1. It fails stating that it couldn't encode characters at position xxxx-yyyy.
I have heard of a Python 3 Unicode bug but don't know if it's related or not. What should I do to get my upload working with Python 3 and the bigquery API (v2)?

Python Request - result length limited?

I'm downloading JSON file in Python3 with Requests:
data = requests.get(addr).text
But some characters at end sometimes got cutted. For example: only 4570 of 4630 characters was in string.What is reason of this behaviour? Spamming F5 in browser does not reproduce problem, only in script.
EDIT:
Ok, I see that sometimes server responding HTTP304 (Not changed) and missing Content-Length header. Any solution possible to read all without this header?

Simulating behaviour of a browser while reading gz files from a web server

I am trying to read a gz file from a webserver through a python script. The size of the source file is 60Mb or more. I dont want to wait for the whole file to be read to decompress and read the contents. Rather I want to decompress information as and when I receive a few bytes.
I tried doing that, but I get turned off by errors like "CRC check failed ". I am using gzip module , as the server returns the content-encoding as "gzip". I have also tried my luck with zlib, but no fruit.
I have seen Mozilla Firefox or Google Chrome doing the above without any issues . I watched the HTTP headers , and I see that the content is not received all at once, but the browser is able to show decompressed partial data as and when it receives it. How do they do it ? Any help is appreciated.

use zlib.decompressobj with the wbits parameter 31. Then deobj.decompress() will allow you to decompress gzip input a piece at a time.
The 31 does not mean 31 bits. It is really 15 + 16, where 15 represents the maximum size of the sliding window of 2^15 bytes, and the 16 is an option to request gzip format decoding. Without adding 16, the zlib format will be decoded, which will reject gzip input.