We Keep Coding

Python: How to only URL Encode a specific URL Parameter?

I have some big URLs that contain a lot of URL parameters.
For my specific case, I need to URL Encode the content of one specific URL Parameter (q) when the content after the "q=" starts with a slash ("/")
Example URL:
https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=/"TEST"/"TEST"
How can I only URL encode that last part of the URL which is within the "q" parameter?
The output of this example should be:
https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=%2F%22TEST%22%2F%22TEST%22%20
I already tried some different things with urllib.parse but it doesnt work the way I want it.
Thanks for your help!

split the string on the &q=/ part and only encode the last string
from urllib import parse
url = 'https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=/"TEST"/"TEST"'
encoded = parse.quote_plus(url.split("&q=/")[1])
encoded_url = f"{url.split('&q=/')[0]}&q=/{encoded}"
print(encoded_url)
output
https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=%2F%22TEST%22%2F%22TEST%22
Note that there's a difference between this and the requested output, but you have an url encoded space (%20) at the end
EDIT
Comment shows a different need for the encoding, so the code needs to change a bit. The code below only encodes the part after &q=. Basically, first split the url and the parameters, then iterate through the parameters to find the q= parameter, and encode that part. Do some f-string and join magic and you get an url that has the q parameter encoded. Note that this might have issues if an & is present in the part that needs to be encoded.
url = 'https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=/"TEST"/"TEST"&utm_source=test1&cpc=123&gclid=abc123'
# the first parameter is always delimited by a ?
baseurl, parameters = url.split("?")
newparameters = []
for parameter in parameters.split("&"):
# check if the parameter is the part that needs to be encoded
if parameter.startswith("q="):
# encode the parameter
newparameters.append(f"q={parse.quote_plus(parameter[2:])}")
else:
# otherwise add the parameter unencoded
newparameters.append(parameter)
# string magic to create the encoded url
encoded_url = f"{baseurl}?{'&'.join(newparameters)}"
print(encoded_url)
output
https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=%2F%22TEST%22%2F%22TEST%22&utm_source=test1&cpc=123&gclid=abc123
EDIT 2
Trying to solve the edge case where there's a & character in the string to be encoded, as this messes up the string.split("&").
I tried using urllib.parse.parse_qs() but this has the same issue with the & character. Docs for reference.
This question is a nice example of how edge cases can mess up simple logic and make it overly complicated.
The RFC3986 also didn't specify any limitations on the name of the query string, otherwise that could've been used to narrow down possible errors even more.
updated code
from urllib import parse
url = 'https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=/"TEST"/&"TE&eeST"&utm_source=test1&cpc=123&gclid=abc123'
# the first parameter is always delimited by a ?
baseurl, parameters = url.split("?")
# addition to handle & in the querystring.
# it reduces errors, but it can still mess up if there's a = in the part to be encoded.
split_parameters = []
for index, parameter in enumerate(parameters.split("&")):
if "=" not in parameter:
# add this part to the previous entry in split_parameters
split_parameters[-1] += f"&{parameter}"
else:
split_parameters.append(parameter)
newparameters = []
for parameter in split_parameters:
# check if the parameter is the part that needs to be encoded
if parameter.startswith("q="):
# encode the parameter
newparameters.append(f"q={parse.quote_plus(parameter[2:])}")
else:
# otherwise add the parameter unencoded
newparameters.append(parameter)
# string magic to create the encoded url
encoded_url = f"{baseurl}?{'&'.join(newparameters)}"
print(encoded_url)
output
https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=%2F%22TEST%22%2F%26%22TE%26eeST%22&utm_source=test1&cpc=123&gclid=abc123

#EdoAkse has a good answer, and should get the credit for the answer.
But the purist in me would do the same thing slightly differently, because
(1) I don't like doing the same function on the same data twice (for efficiency), and
(2) I like the logical symmetry of using the join function to reverse a split.
My code would look more like this:
from urllib import parse
url = 'https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=/"TEST"/"TEST"'
splitter = "&q=/"
unencoded,encoded = url.split(splitter)
encoded_url = splitter.join(unencoded,parse.quote_plus(encoded))
print(encoded_url)
Edit: I couldn't resist posting my edited answer based on the commentary. You can see the virtual identical code developed independently. This must be the right approach then, I guess.
from urllib import parse
url = 'https://www.exmple.com/test?test1=abc&test2=abc&test3=abc&q=/"TEST"/"TEST"'
base_url,arglist = url.split("?",1)
args = arglist.split("&")
new_args = []
for arg in args:
if arg.lower().startswith("q="):
new_args.append(arg[:2]+parse.quote_plus(arg[2:]))
else:
new_args.append(arg)
encoded_url = "?".join([base_url,"&".join(new_args)])
print(encoded_url)

Parse a URL and replace variables if present

I have URI's specified in an xls file. I want to read that xls file, get the URI from there, parse it, and replace variables (if present) with the corresponding values, and then make an API call to that URI
For example:
These are a few URI's in the xls sheet:
https://api.something.com/v1/me
https://api.something.com/v1/{user_id}/account
(Where user_id is a variable, that has to be replaces by an appropriate value.) is there an easy way to parse the URI and check if there's a variable present there, if yes, get the value of the variable and form a new string with the value and then use the URI to make an API call. Else use the URI as is.

Field names can be discovered using stdlib string.Formatter:
>>> s = "https://api.something.com/v1/{user_id}/account"
>>> from string import Formatter
>>> parsed = Formatter().parse(s)
>>> field_names = []
>>> for literal_text, field_name, format_spec, conversion in parsed:
... if field_name is not None:
... field_names.append(field_name)
...
>>> field_names
['user_id']

Fortunately, Python has a built-in mechanism for handling this!
>>> 'https://api.something.com/v1/{user_id}/account'.format(user_id='my_id', unused_variable='xyzzy')
'https://api.something.com/v1/my_id/account'

What is the correct way to convert a url query into a list? [duplicate]

My problem is that with the given code:
from flask import Flask, request
app = Flask(__name__)
#app.route("/")
def hello():
return str(request.values.get("param", "None"))
app.run(debug=True)
and I visit:
http://localhost:5000/?param=a&param=bbb
I should expect an output of ['a', 'bbb'] except Flask seems to only accept the first param and ignore the rest.
Is this a limitation of Flask? Or is it by design?

You can use getlist, which is similar to Django's getList but for some reason isn't mentioned in the Flask documentation:
return str(request.args.getlist('param'))
The result is:
[u'a', u'bbb']
Use request.args if the param is in the query string (as in the question), request.form if the values come from multiple form inputs with the same name. request.values combines both, but should normally be avoided for the more specific collection.

If you used $('form').serialize() in jQuery to encode your form data, you can use request.form['name'] to get data, but note that when multiple input elements' names are the same, request.form['name'] will only get the first matched item. So I checked the form object from the Flask API, and I found this. Then I checked the MultiDict object, and I found the function getlist('name').
If there are multiple inputs with the same name, try this method: request.form.getlist('name')

Another option is to use a flat json structure with request.args. Because sometimes you simply do not know the parameter beforehand and you cannot use .getlist().
arguments = request.args.to_dict(flat=False)
# Get a specific parameter
param = arguments.get('param')
print(param)
# Get all the parameters that have more than one value
for field, values in arguments.items():
if len(values) > 1:
print(values)

One more way is that you can use one key and one value that holds multiple values and then in the server you can split and do what ever you want .Hope this Helps somewone
http://localhost/api/products/filters?Manufacturer=Dell|HP|HUAWEI|Lenovo
OR
http://localhost/api/products/filters?Manufacturer=Dell_HP_HUAWEI_Lenovo
OR
http://localhost/api/products/filters?Manufacturer=Dell__HP__HUAWEI__Lenovo

Python Requests - add text at the beginning of query string

When sending data through python-requests a GET request, I have a need to specifically add something at the beginning of the query string. I have tried passing the data in through dicts and json strings with no luck.
The request as it appears when produced by requests:
/apply/.../explain?%7B%22......
The request as it appears when produced by their interactive API documentation (Swagger):
/apply/.../explain?record=%7B%22....
Where the key-value pairs of my data follow the excerpt above.
Ultimately, I think the missing piece is the record= that gets produced by their documentation. It is the only piece that is different from what is produced by Requests.
At the moment I've got it set up something like this:
import requests
s = requests.Session()
s.auth = requests.auth.HTTPBasicAuth(username,password)
s.verify = certificate_path
# with data below being a dictionary of the values I need to pass.
r = s.get(url,data=data)
I am trying to include an image of the documentation below, but don't yet have enough reputation to do so:
apply/model/explain documentation

'GET' requests don't have data, that's for 'POST' and friends.
You can send the query string arguments using params kwarg instead:
>>> params = {'record': '{"'}
>>> response = requests.get('http://www.example.com/explain', params=params)
>>> response.request.url
'http://www.example.com/explain?record=%7B%22'

From the comments i felt the need to explain this.
http://example.com/sth?key=value&anotherkey=anothervalue
Let's assume you have a url like the above in order to call with python requests you only have to write
response = requests.get('http://example.com/sth', params={
'key':'value',
'anotherkey':'anothervalue'
})
Have in mind that if your value or your keys have any special character in them they will be escaped thats the reason for the %7B%2 part of url in your question.

How to retrieve GET vars in python bottle app

I'm trying to make a simple REST api using the Python bottle app.
I'm facing a problem in retrieving the GET variables from the request global object.
Any suggestions how to retrieve this from the GET request?

They are stored in the request.query object.
http://bottlepy.org/docs/dev/tutorial.html#query-variables
It looks like you can also access them by treating the request.query attribute like a dictionary:
request.query['city']
So dict(request.query) would create a dictionary of all the query parameters.
As #mklauber notes, this will not work for multi-byte characters. It looks like the best method is:
my_dict = request.query.decode()
or:
dict(request.query.decode())
to have a dict instead of a <bottle.FormsDict object at 0x000000000391B...> object.

If you want them all:
from urllib.parse import parse_qs
dict = parse_qs(request.query_string)
If you want one:
one = request.GET.get('one', '').strip()

Can you try this please:
For this example : http://localhost:8080/command?param_name=param_value
In your code:
param_value = request.query.param_name

from the docs
name = request.cookies.name
# is a shortcut for:
name = request.cookies.getunicode('name') # encoding='utf-8' (default)
# which basically does this:
try:
name = request.cookies.get('name', '').decode('utf-8')
except UnicodeError:
name = u''
So you might prefer using attribute accessor (request.query.variable_name) than request.query.get('variable_name')
Another point is you can use request.params.variable_name which works both for GET and POST methods, than having to swich request.query.variable_name or request.forms.variable_name depending GET/POST.