We Keep Coding

F12 Network response garbled characters

This is my first time to climb such data, I do not know why his data does not show up, using python's chardet library does not work. Is there someone who can help me? Thank you!
I tried to crawl through the data, but the data seems to be garbled

This isn't text. Inspecting the response headers of some of those responses shows
content-type: application/x-protobuf
That's a binary format that can't be parsed without the message specification. It's used as the message format in gRPC.
You can check the Protocol Buffer Basics: Python tutorial at the Protobuf site to understand how it works.

Given a URL, how to encode a file's contents as base64 with Python / Django?

I am building a Django-based website, and am having trouble figuring out a decent way to email some larger PDFs and such to my users.
The files in question never touch our servers; they're handled on a CDN. So, my starting point is with the unique URLs for the files, not with the files themselves. It would be nice to find a solution that doesn't involve saving the files locally.
In order for me to be able to send the email in the way I want (with the PDF/DOCX/whatever attached to it), I need to be able to encode the attachment as a base-64 string.
I would prefer not to save the file to our server; I would also prefer not to read a response object in chunks and write it plainly to a file on our server, then encode that file.
That said, given a direct url to a file is there a way to stream the response and encode it in base64 as it comes in?
I have been reading about Django's StreamingHttpResponse and FileWrapper and feel like I am close, but I'm not able to put it together just yet.
Edit: the snippet below is working for now, but I'm worried about memory usage - how well would something like this scale?
import base64
req = requests.get('url')
encoded = base64.b64encode(req.content)

Thanks to beetea I am comfortable implementing the simple:
import base64
req = requests.get('url')
encoded = base64.b64encode(req.content)
As the solution to this issue.

Dump JSON from string in unknown character encoding

I'm trying to dump HTML from websites into JSON, and I need a way to handle the different character encodings.
I've read that if it isn't utf-8, it's probably ISO-8859-1, so what I'm doing now is:
for possible_encoding in ["utf-8", "ISO-8859-1"]:
try:
# post_dict contains, among other things, website html retrieved
# with urllib2
json = simplejson.dumps(post_dict, encoding=possible_encoding)
break
except UnicodeDecodeError:
pass
if json is None:
raise UnicodeDecodeError
This will of course fail if I come across any other encodings, so I'm wondering if there is a way to solve this problem in the general case.
The reason I'm trying to serialize the HTML in the first place is because I need to send it in a POST request to our NodeJS server. So, if someone has a different solution that allows me to do that (maybe without serializing to JSON at all), I'd be happy to hear that as well.

You should know the character encoding regardless of media type you use to send POST request (unless you want to send binary blobs). To get the character encoding of your html content, see
A good way to get the charset/encoding of an HTTP response in Python
.
To send post_dict as json, make sure all strings in it are Unicode (just convert html to Unicode as soon as you receive it) and don't use the encoding parameter for json.dumps() call. The parameter won't help you anyway if different web-sites (where you get your html strings) use different encodings.

Obtaining MD5 hash value from online image / movie using Python

I'm currently looking to put together a quick script using Python 2.x to try and obtain the MD5 hash value of a number of images and movies on specific websites. I have noted on the w3.org website that the HTTP/1.1 protocol does offer an option within the content field to access the MD5 value but I'm wondering if this has to be set by the website admin? My script is as below:-
import httplib
c = httplib.HTTPConnection("www.samplesite.com")
c.request("HEAD", "/sampleimage.jpg")
r = c.getresponse()
res = r.getheaders()
print res
I have a feeling I need to edit 'HEAD' or possibly r.getheaders but I'm just not sure what to replace them with.
Any suggestions? As said, I'm just looking to point at an image and to then capture the MD5 hash value of the said image / movie. Ideally I don't want to have to download the image / movie to save bandwidth hence why I'm trying to do it this way.
Thanks in advance

Yes, it's rare that servers will actually respond to requests with an MD5 header. You can check for that, but in most cases, you'll actually need to download the video or image, unfortunately.
(At least hashlib is simple!)

Torrent Tracker info hash GET Request- Python

I'm trying to connect to a torrent tracker to receive a list of peers to play bit torrent with, however I am having trouble forming the proper GET request.
As far as I understand, I must obtain the 20 byte SHA1 hash of the bencoded 'info' section from the .torrent file. I use the following code:
h = hashlib.new('sha1')
h.update(bencode.bencode(meta_dict['info']))
info_hash = h.digest()
This is where I am stuck. I can not figure out how to create the proper url-encoded info_hash to stick into a URL string as a parameter.
I believe it involves some combination of urllib.urlencode and urllib.quote, however my attempts have not worked so far.

well a bit late but might help someone.
Using module requests encodes the url by it's self. First you need to create a dictionary with the parameters (info_hash, peer_id etc). Then you only have to do a get request
response = requests.get(tracker_url, params=params)

I think that urllib.quote_plus() is all you need.