We Keep Coding

Postman API request in Python

I am trying to web-scrape data from https://www.mcmaster.com. They have provided me with a .pfx file and a passphrase. When making a GET request on Postman using their .json file, I input my website login/password and upload the .pfx certificate with its passphrase and everything works fine. Now I am trying to do this same thing but in Python, but am a bit unsure.
Here is my current Python code, I am unsure where I would put the website email/password login and how to successfully do a GET request.
import requests_pkcs12
from requests_pkcs12 import get
r = get('https://api.mcmaster.com/v1/login', pkcs12_filename='Schallert.pfx', pkcs12_password='mcmasterAPI#1901')
response = requests_pkcs12.get(r)
print(response.text)
Here is how I have it setup in Postman (Website email/pw login)
.PFX Certificate page

Postman has a built in feature where it will convert requests into code. You can do it like so:
On the far right click the Code Snippet Button (</>)
Once you are on that page, there is two available python options
Then all you need to do is copy the code into a Python file and add all your customizations (Should be already optimized)
One thing I’ll warn you about though is the URL. Postman doesn’t add
http:// or https:// to the URL, which means Python will throw a No
Scheme Supplied error.
Available Packages for Auto Conversion:
Requests
http.client
Meaning you will have to use a different package instead of requests_pkcs12

After a quick web search, it looks like you need to create a temporary certificate as a .pem file, which is then passed to the request.
from contextlib import contextmanager
from pathlib import Path
from tempfile import NamedTemporaryFile
import requests
from cryptography.hazmat.primitives.serialization import Encoding, PrivateFormat, NoEncryption
from cryptography.hazmat.primitives.serialization.pkcs12 import load_key_and_certificates
#contextmanager
def pfx_to_pem(pfx_path, pfx_password):
pfx = Path(pfx_path).read_bytes()
private_key, main_cert, add_certs = load_key_and_certificates(pfx, pfx_password.encode('utf-8'), None)
with NamedTemporaryFile(suffix='.pem') as t_pem:
with open(t_pem.name, 'wb') as pem_file:
pem_file.write(private_key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()))
pem_file.write(main_cert.public_bytes(Encoding.PEM))
for ca in add_certs:
pem_file.write(ca.public_bytes(Encoding.PEM))
yield t_pem.name
with pfx_to_pem('your pfx file path', 'your passphrase') as cert:
requests.get(url, cert=cert, data=payload)

The package requests_pkcs12 is a wrapper written above the requests package. So, all the parameters that accept requests will accept by the requests_pkcs12.
Here is the source code proof for that. https://github.com/m-click/requests_pkcs12/blob/master/requests_pkcs12.py#L156
Also, from your screenshots, I understood that you are using POST not GET.
import json
from requests_pkcs12 import post
url = "https://api.mcmaster.com/v1/login"
payload = {'Username': 'yourusername',
'Password': 'yourpassword'}
resp = post(url, pkcs12_filename='Schallert.pfx', pkcs12_password='mcmasterAPI#1901', data=json.dumps(payload))
print(resp)
Footnote: I hope the password mcmasterAPI#1901 it's a fake one. if not please don't share any credentials in the platform.

How do I properly format an Azure Devops Pipeline API JSON request containing a variable in Python?

I am trying to figure out how to properly format a JSON request passing a variable via Python in Azure Devops. It seems that the code does not escape the double quotes in the dictionary. When typing in the same variable via GUI, it works as expected.
Python Code
from requests.auth import HTTPBasicAuth
# Api function
def auth_restapi():
auth = HTTPBasicAuth(username, password)
return auth
url = "https://[ORG].visualstudio.com/[PROJECT]/_apis/build/builds?api-version=6.0"
build_version = "1.2.3"
data = {"templateParameters": {"build_version": build_version}, "definition": {"id": 4}}
requests.post(url, auth=auth_restapi(), json=data)
When running this script, the logs show:
build_version: "1.2.3"
When entering the version manually in the GUI, the logs show the expected result:
build_version: 1.2.3
Is there any way to remove the quotes from the Python request variable?

After lots of troubleshooting I seemed to have resolved the problem. There were two issues:
The URL should be the pipeline url and not the build url.
https://dev.azure.com/[ORG]/[PROJECT]/_apis/pipelines/4/runs?api-version=6.1-preview.1
The body should contain the word variables.
data = {"variables": {"build_version": {"value": build_version}}}

Pagination: Invalid query parameter

I'm testing Amadeus' API following Amadeus Github tuto to get familiar. I didn't know about pagination and it seems difficult to handle.
from amadeus import Client, ResponseError
from amadeus import Location
from json_encoder import json
def getFirst():
response = amadeus.reference_data.locations.get(
keyword='LON',
subType=Location.ANY
)
return (response)
amadeus = Client(
client_id=REPLACE_BY_YOUR_API_KEY,
client_secret=REPLACE_BY_YOUR_API_SECRET
)
try:
first = getFirst()
next = amadeus.next(first)
print(next.data)
except ResponseError as error:
print(error)
first call works but with next (pagination) Im trying different calls with the same [400]
[page] Invalid query parameter error.
This is the JSON from get in proposal
Also I found this Amadeus pagination calls where si used the same kind of calls with success
...Im wondering how can I use properly this function from Amadeus library.
Thanks for reading & help!

Thanks for reporting this bug. It has been fixed, you can now install the version 2.0.1 of the Python SDK to get the fix:
pip install amadeus
Root cause:
APIs supporting pagination use a square bracket parameter to index the page and
the offset. The format was not correctly encoded by the urlencode method from
urllib when building the URL parameter list.
For the following example:
{'longitude': 0.1278, 'latitude': 51.5074, 'page': {'offset': 10}}
It was encoded as:
longitude=0.1278&latitude=51.5074&page={offset : 10}
Instead of:
longitude=0.1278&latitude=51.5074&page[offset]=10
The new private method _urlencode parses and prepares the dictionary before the
actual call to urlencode is done.

Indexing to elasticsearch 6.1 with python script

I am trying to index to elasticsearch using a python 2.7 script as follows:
from __future__ import print_function
import urllib, urllib2
#FORMDATA is a json format string that has been taken from a file
ELASTIC_URL = 'http://1.2.3.9:9200/indexname/entry/
req = urllib2.Request(ELASTIC_URL)
req.add_header('contentType', 'application/x-www-form-urlencoded')
response = urllib2.urlopen(req, FORMDATA, timeout=4).read()
print(response)
I keep getting the error HTTP Error 406: Not Acceptable: HTTPError
i have also tried formatting the data with urllib.quote(FORMDATA) and get the same error. The data is not a dictionary it is a string that when converted to json is multi dimensional.
I think this is something to do with the fact the req header needs to specify the contentType to be the correct format but i'm struggling to workout what this is. I managed to do this import on elasticsearch 5.x but now on 3.x it doesn't seem to be working.
Any ideas??

Almost all elasticsearch API calls use Content-Type: application/json in the headers - this should be what you need here.
Also be aware that if you are submitting data, this will need to be in the form of a POST (or a PUT if generating your own id), not a GET request: https://www.elastic.co/guide/en/elasticsearch/guide/current/index-doc.html

making a simple GET/POST with url Encoding python

i have a custom url of the form
http://somekey:somemorekey#host.com/getthisfile.json
i tried all the way but getting errors :
method 1 :
from httplib2 import Http
ipdb> from urllib import urlencode
h=Http()
ipdb> resp, content = h.request("3b8138fedf8:1d697a75c7e50#abc.myshopify.com/admin/shop.json")
error :
No help on =Http()
Got this method from here
method 2 :
import urllib
urllib.urlopen(url).read()
Error :
*** IOError: [Errno url error] unknown url type: '3b8108519e5378'
I guess something wrong with the encoding ..
i tried ...
ipdb> url.encode('idna')
*** UnicodeError: label empty or too long
Is there any way to make this Complex url get call easy .

You are using a PDB-based debugger instead of a interactive Python prompt. h is a command in PDB. Use ! to prevent PDB from trying to interpret the line as a command:
!h = Http()
urllib requires that you pass it a fully qualified URL; your URL is lacking a scheme:
urllib.urlopen('http://' + url).read()
Your URL does not appear to use any international characters in the domain name, so you do not need to use IDNA encoding.
You may want to look into the 3rd-party requests library; it makes interacting with HTTP servers that much easier and straightforward:
import requests
r = requests.get('http://abc.myshopify.com/admin/shop.json', auth=("3b8138fedf8", "1d697a75c7e50"))
data = r.json() # interpret the response as JSON data.

The current de facto HTTP library for Python is Requests.
import requests
response = requests.get(
"http://abc.myshopify.com/admin/shop.json",
auth=("3b8138fedf8", "1d697a75c7e50")
)
response.raise_for_status() # Raise an exception if HTTP error occurs
print response.content # Do something with the content.