We Keep Coding

How to decode a post url string containing plus sign

I have a encoded URL string (click to visit the page)
http://epub.sipo.gov.cn/patentoutline.action?strWhere=OPD%3D%272019.02.15%27+and+PA%3D%27%25%E5%8D%8E%E4%B8%BA%25%27
obtain by chrome inspect. I try to write a requests post function to get the page, the best I could figure out is the following, however, it does not work properly. The troubling part seems to be the plus sign. (If there is no and clause, "OPD='2019.02.15'" or "PA='%华为%'" works fine.)
import requests
url = 'http://epub.sipo.gov.cn/patentoutline.action'
params = {'strWhere': r"OPD='2019.02.15' and PA='%华为%'"} # cannot find results
# params = {'strWhere': r"OPD='2019.02.15'"} # works
# params = {'strWhere': r"PA='%华为%'"} # works
r = requests.post(url, data=params)
print(r.content.decode())

replace in the url the spaces with %20
you can use a function before send it
str.replace(old, new[, max])
for example
params = {'strWhere': r"OPD='2019.02.15' and PA='%华为%'"}
params['strWhere'] = params['strWhere'].replace(' ', '%20')

python %s is discarding \n while sending via POST method

I am working on an email sending project in python
when i pass %s in post method with a variable its discarding all new lines and sending it to email so im getting all stuffed emails can somebody please help
post method :
r = requests.post('https://maker.ifttt.com/trigger/Custom Newsfeed/with/key/nYmoaoROeu6Mf2SrBOgUg?value1=%s' % (s))
so if s contains a news with headline and body, in email im getting it all concatenated together

If on python3, try
'value1={}'.format(s)
Or if on 3.6, try
f'value1={s}'

Use format() to construct that url. Also, learn about using named placeholders using format(), the example provided below makes use of them.
s = s.replace('\n', '%0A') # Replaces \n with urlencoded \n (%0A)
url = 'https://maker.ifttt.com/trigger/Custom Newsfeed/with/key/nYmoaoROeu6Mf2SrBOgUg?value1={value1}'.format(value1=s)
r = requests.post(url)
You could also use urlencode.
>>> import urllib
>>> f = { 'value1' : s}
>>> urllib.urlencode(f)
'value1=cool+event'

Python Requests URL with Unicode Parameters

I'm currently trying to hit the google tts url, http://translate.google.com/translate_tts with japanese characters and phrases in python using the requests library.
Here is an example:
http://translate.google.com/translate_tts?tl=ja&q=ひとつ
However, when I try to use the python requests library to download the mp3 that the endpoint returns, the resulting mp3 is blank. I have verified that I can hit this URL in requests using non-unicode characters (via romanji) and have gotten correct responses back.
Here is a part of the code I am using to make the request
langs = {'japanese': 'ja',
'english': 'en'}
def get_sound_file_for_text(text, download=False, lang='japanese'):
r = StringIO()
glang = langs[lang]
text = text.replace('*', '')
text = text.replace('/', '')
text = text.replace('x', '')
url = 'http://translate.google.com/translate_tts'
if download:
result = requests.get(url, params={'tl': glang, 'q': text})
r.write(result.content)
r.seek(0)
return r
else:
return url
Also, if I print textor url within this snippet, the kana/kanji is rendered correctly in my console.
Edit:
If I attempt to encode the unicode and quote it as such, I still get the same response.
# -*- coding: utf-8 -*-
from StringIO import StringIO
import urllib
import requests
__author__ = 'jacob'
langs = {'japanese': 'ja',
'english': 'en'}
def get_sound_file_for_text(text, download=False, lang='japanese'):
r = StringIO()
glang = langs[lang]
text = text.replace('*', '')
text = text.replace('/', '')
text = text.replace('x', '')
text = urllib.quote(text.encode('utf-8'))
url = 'http://translate.google.com/translate_tts?tl=%(glang)s&q=%(text)s' % locals()
print url
if download:
result = requests.get(url)
r.write(result.content)
r.seek(0)
return r
else:
return url
Which returns this:
http://translate.google.com/translate_tts?tl=ja&q=%E3%81%B2%E3%81%A8%E3%81%A4
Which seems like it should work, but doesn't.
Edit 2:
If I attempt to use urlllb/urllib2, I get a 403 error.
Edit 3:
So, it seems that this problem/behavior is simply limited to this endpoint. If I try the following URL, a different endpoint.
http://www.kanjidamage.com/kanji/13-un-%E4%B8%8D
From within requests and my browser, I get the same response (they match). If I even try ascii characters to the server, like this url.
http://translate.google.com/translate_tts?tl=ja&q=sayonara
I get the same response as well (they match again). But if I attempt to send unicode characters to this URL, I get a correct audio file on my browser, but not from requests, which sends an audio file, but with no sound.
http://translate.google.com/translate_tts?tl=ja&q=%E3%81%B2%E3%81%A8%E3%81%A4
So, it seems like this behavior is limited to the Google TTL URL?

The user agent can be part of the problem, however, it is not in this case. The translate_tts service rejects (with HTTP 403) some user agents, e.g. any that begin with Python, curl, wget, and possibly others. That is why you are seeing a HTTP 403 response when using urllib2.urlopen() - it sets the user agent to Python-urllib/2.7 (the version might vary).
You found that setting the user agent to Mozilla/5.0 fixed the problem, but that might work because the API might assume a particular encoding based on the user agent.
What you actually should do is to explicitly specify the URL character encoding with the ie field. Your URL request should look like this:
http://translate.google.com/translate_tts?ie=UTF-8&tl=ja&q=%E3%81%B2%E3%81%A8%E3%81%A4
Note the ie=UTF-8 which explicitly sets the URL character encoding. The spec does state that UTF-8 is the default, but doesn't seem entirely true, so you should always set ie in your requests.
The API supports kanji, hiragana, and katakana (possibly others?). These URLs all produce "nihongo", although the audio produced for hiragana input has a slightly different inflection to the others.
import requests
one = u'\u3072\u3068\u3064'
kanji = u'\u65e5\u672c\u8a9e'
hiragana = u'\u306b\u307b\u3093\u3054'
katakana = u'\u30cb\u30db\u30f3\u30b4'
url = 'http://translate.google.com/translate_tts'
for text in one, kanji, hiragana, katakana:
r = requests.get(url, params={'ie': 'UTF-8', 'tl': 'ja', 'q': text})
print u"{} -> {}".format(text, r.url)
open(u'/tmp/{}.mp3'.format(text), 'wb').write(r.content)

I made this little method before to help me with UTF-8 encoding. I was having issues printing cyrllic and CJK languages to csvs and this did the trick.
def assist(unicode_string):
utf8 = unicode_string.encode('utf-8')
read = utf8.decode('string_escape')
return read ## UTF-8 encoded string
Also, make sure you have these two lines at the beginning of your .py.
#!/usr/bin/python
# -*- coding: utf-8 -*-
The first line is just a good python habit, it specifies which compiler to use on the .py (really only useful if you have more than one version of python loaded on your machine). The second line specifies the encoding of the python file. A slightly longer answer for this is given here.

Setting the User-Agent to Mozilla/5.0 fixes this issue.
from StringIO import StringIO
import urllib
import requests
__author__ = 'jacob'
langs = {'japanese': 'ja',
'english': 'en'}
def get_sound_file_for_text(text, download=False, lang='japanese'):
r = StringIO()
glang = langs[lang]
text = text.replace('*', '')
text = text.replace('/', '')
text = text.replace('x', '')
url = 'http://translate.google.com/translate_tts'
if download:
result = requests.get(url, params={'tl': glang, 'q': text}, headers={'User-Agent': 'Mozilla/5.0'})
r.write(result.content)
r.seek(0)
return r
else:
return url

i can't take the "?" character in the URL of django url

I try to code to django a redirect fonction. I provide the url, and i want to redirect to the provided URL.
in my urls.py:
urlpatterns = patterns('',
url(r'^redirect/(?P<name>.*)$', redirect),
# ...
)
when i test the function with a standard link (ex: google.com), it works perfectly.
when i test the function with a link that containt a "?" character, only the part before the "?" is taken into account.
Example :
"GET /redirect/http://www.polyvore.com/lords_liverpool_hey_jude_tee/thing?id=53713291 HTTP/1.1" 302 0
name = http://www.polyvore.com/lords_liverpool_hey_jude_tee/thing
the ?id=53713291 is not taken into account....
i though that .* means all kind character, wrong?
Do you know what happens ? and how to take the entiere url what ever the character it contains?
thank you very much for your help.

You seems to don't understand how URL works. Everything after the ? is parsed as arguments for your current view. If you print data in your request.GET dict, you'll find something like:
{'id': 53713291}
The only way to fix that is to urlencode your URL in the argument and decode it before the redirection.
>>> import urllib
>>> urllib.quote_plus("http://www.polyvore.com/lords_liverpool_hey_jude_tee/thig?id=53713291")
'http%3A%2F%2Fwww.polyvore.com%2Flords_liverpool_hey_jude_tee%2Fthing%3Fid%3D5313291'
# You should make your URL with this value, for example:
# /redirect/http%3A%2F%2Fwww.polyvore.com%2Flords_liverpool_hey_jude_tee%2Fthing%3Fid%3D5313291
# And in your view, use unquote_plus before the redirection:
>>> urllib.unquote_plus('http%3A%2F%2Fwww.polyvore.com%2Flords_liverpool_hey_jude_tee%2Fthing%3Fid%3D5313291')
'http://www.polyvore.com/lords_liverpool_hey_jude_tee/thing?id=5313291'
More information about Query String on Wikipedia.

You're passing in a regex, so you need to escape the special characters (ie, \?)
But if you're trying to pass in querystring parameters you'll want to handle that differently: https://stackoverflow.com/a/3711911

Getting an empty file when served by django

I have the following code for managing file download through django.
def serve_file(request, id):
file = models.X.objects.get(id=id).file #FileField
file.open('rb')
wrapper = FileWrapper(file)
mt = mimetypes.guess_type(file.name)[0]
response = HttpResponse(wrapper, content_type=mt)
import unicodedata, os.path
filename = unicodedata.normalize('NFKD', os.path.basename(file.name)).encode("utf8",'ignore')
filename = filename.replace(' ', '-') #Avoid browser to ignore any char after the space
response['Content-Length'] = file.size
response['Content-Disposition'] = 'attachment; filename={0}'.format(filename)
#print response
return response
Unfortunately, my browser get an empty file when downloading.
The printed response seems correct:
Content-Length: 3906
Content-Type: text/plain
Content-Disposition: attachment; filename=toto.txt
blah blah ....
I have similar code running ok. I don't see what can be the problem. Any idea?
PS: I have tested the solution proposed here and get the same behavior
Update:
Replacing wrapper = FileWrapper(file) by wrapper = file.read() seems to fix the problem
Update: If I comment the print response, I get similar issue:. the file is empty. Only difference: FF detects a 20bytes size. (the file is bigger than this)

File object is an interable, and a generator. It can be read only once before being exausted. Then you have to make a new one, of use a method to start at the begining of the object again (e.g: seek()).
read() returns a string, which can be read multiple times without any problem, this is why it solves your issue.
So just make sure that if you use a file like object, you don't read it twice in a row. E.G: don't print it, then returns it.

From django documentation:
FieldFile.open(mode='rb') Behaves like the standard Python open()
method and opens the file associated with this instance in the mode
specified by mode.
If it works like pythons open then it should return a file-object, and should be used like this:
f = file.open('rb')
wrapper = FileWrapper(f)