We Keep Coding

Python API script

I am making a python script using API of a free test automation website called TestProject.
Link to their API: https://api.testproject.io/docs/v2/
Basically what i want to do is grab pdf of reports of all tests and save them somewhere.
But to make the GET request to do that i first need projectID and jobID which i already wrote functions getting them and saving them in the array.
But now i have a problem where its looping through both lists and not using correct projectID and jobID and its throwing errors because it does not exist.
So what i need is something to check if jobID is in projectID so that way i can make a GET request to get all the executionID's to get the PDF of the report.
I am kinda new to programming so i would love any help i can get. If anyone has any better solutions please feel free to let me know.
My script:
import requests
import json
import csv
from datetime import datetime
from jsonpath_ng import jsonpath, parse
API_key = 'api_key'
headers = {'Authorization':'{}'.format(API_key)}
list_projectId = []
list_jobId = []
list_executionId = []
ParseData_projectId = parse('$..id')
ParseData_jobId = parse('$..id')
ParseData_executionId = parse('$..id')
def parsing (response,ParseData,list_data):
# parses data and appends it to the list
Data = json.loads(response)
Parsaj = ParseData
Podatki = Parsaj.find(Data)
for i in range(0, len(Podatki)):
vrednost = Podatki[i].value
list_data.append(vrednost)
def projectId():
# gets all projectId's and saves them in list_projectId
url = 'https://api.testproject.io/v2/projects?_start=0'
response = requests.get(url,headers=headers)
response_json = response.json()
converted = json.dumps(response_json)
parsing(converted,ParseData_projectId,list_projectId)
def jobId():
# gets all jobId's and saves them in list_jobId
for i in range(0, len(list_projectId)):
id = list_projectId[i]
url = 'https://api.testproject.io/v2/projects/{}'.format(id) + '/jobs?onlyScheduled=false&_start=0'
response = requests.get(url,headers=headers)
response_json = response.json()
converted = json.dumps(response_json)
parsing(converted,ParseData_jobId,list_jobId)
def executionId():
# Their API link:
# https://api.testproject.io/v2/projects/{projectId}/jobs/{jobId}/reports?_start=0
# the for loop below does not work here is where i need the help:
for i in range(0, len(list_projectId)):
project_id = list_projectId[i]
job_id = list_jobId[i]
url = 'https://api.testproject.io/v2/projects/{}'.format(project_id) + '/jobs/{}'.format(job_id) + '/reports?_start=0'
response = requests.get(url,headers=headers)
response_json = response.json()
converted = json.dumps(response_json)
parsing(converted,ParseData_executionId,list_executionId)
projectId()
print("----------LIST PROJECT ID: ----------")
print(list_projectId)
print("")
jobId()
print("----------LIST JOB ID: ----------")
print(list_jobId)
executionId()
print("----------LIST EXECUTION ID: ----------")
print(list_executionId)

you have to use 'in' operator to check the value exist in the list data structure.

REST API requests - Use Concurrent.futures in the right way

The code below is a sample from my complete program, I tried it to make understandable.
It sends requests to a REST API. It starts with an URL and the number of pages for this specific search and tries to catch the content for each page.
Each page has several results. Each result becomes a FinalObject.
Because there are as many API requests as there are pages, I decided to use multi-threading and the concurrent.futures module.
=> It works but, as I'm new in coding and Python, I still have these 2 questions:
How to use ThreadPoolExecutor sequentially in this case,
Is there a better way to handle multi-threading in this case?
from concurrent.futures import ThreadPoolExecutor
from requests import get as re_get
def main_function(global_page_number, headers, url_request):
# create a list of pages number
pages_numbers_list = [i for i in range(global_page_number)]
# for each page, call the page_handler (MultiThreading)
with ThreadPoolExecutor(max_workers=10) as executor:
for item in pages_numbers_list:
executor.submit(
page_handler,
item,
url_request,
headers
)
def page_handler(page_number, url_request, headers):
# we change the page number in the url request
url_request = change_page(url_request, page_number)
# new request with the new url
result = re_get(url_request, headers=headers)
result = result.json()
# in the result, with found the list of dict in order to create the
# final object
final_object_creation(result['results_list'])
def change_page(url_request, new_page_number):
"to increment the value of the 'page=' attribute in the url"
current_nb_page = ''
start_nb = url_request.find("page=") + len('page=')
while 1:
if url_request[start_nb].isdigit():
current_nb_page = url_request[start_nb]
else:
break
new_url_request = url_request.replace("page=" + current_nb_page,
"page=" + str(new_page_number))
return new_url_request
def final_object_creation(results_list):
'thanks to the object from requests.get(), it builts the final object'
global current_id_decision, dict_decisions
# each item in the results lis should be an instance of the final object
for item in results_list:
# On définit l'identifiant du nouvel objet Decision
current_id_decision += 1
new_id = current_id_decision
# On crée l'objet Décision et on l'ajoute au dico des décisions
dict_decisions[new_id] = FinalObject(item)
class FinalObject:
def __init__(self, content):
self.content = content
current_id_decision = 0
dict_decisions = {}
main_function(1000, "headers", "https://api/v1.0/search?page=0&query=test")

KeyError: records when trying to return record count from API

I am trying to check how many records a player has using the Hypixel API friends endpoint (api.hypixel.net/friends)
It keeps giving me a key error when trying to count the records. Here is what the API gives me:
{
"success": true,
"records": [{"_id":"5806841c0cf247f13be18b9d","uuidSender":"71ef88df5f7e482fb472f344965beba8","uuidReceiver":"976129e438b54a839944b1c0703d4da3","started":1476822044856},{"_id":"59589dde0cf250df95af825e","uuidSender":"71ef88df5f7e482fb472f344965beba8","uuidReceiver":"2c5bfce120c04ef1bfb3b798fe0d650e","started":1498979806692},{"_id":"5a444d820cf2604c12e0f6bd","uuidSender":"71ef88df5f7e482fb472f344965beba8","uuidReceiver":"ea703151981a409f8d0ff7cb782ab1c1","started":1514425730370},{"_id":"5aa595800cf24bd1104381cc","uuidSender":"71ef88df5f7e482fb472f344965beba8","uuidReceiver":"27af346e5bde40f0a665e808a331576f","started":1520801152354},{"_id":"5e2511f90cf289be2d0ea273","uuidSender":"71ef88df5f7e482fb472f344965beba8","uuidReceiver":"24a90aca074c4293a656f5fda047f816","started":1579487737061},{"_id":"5e36448c0cf2174287f94af7","uuidSender":"71ef88df5f7e482fb472f344965beba8","uuidReceiver":"1nmHNU1mERJEG452cx1FazPx3RpAuZ9vW","started":1580614796544},{"_id":"5e3f54190cf2ab010c5a21ce","uuidSender":"71ef88df5f7e482fb472f344965beba8","uuidReceiver":"63817f05823945a2b58f4ba1de5589a3","started":1581208601957},{"_id":"55bec5e0c8f2e017bca39176","uuidSender":"1nmHNU1mERJEG452cx1FazPx3RpAuZ9vW","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1438565856136},{"_id":"55e21268c8f21846db2f2566","uuidSender":"8fca5ebf02f74a369b13f3407ad4a9bc","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1440879208434},{"_id":"56c26b190cf2d1a91ec25e83","uuidSender":"1nmHNU1mERJEG452cx1FazPx3RpAuZ9vW","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1455581977070},{"_id":"5755edc20cf2db67507e3a2e","uuidSender":"0ce4597de5484c0e82a067fa0bf171df","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1465249218466},{"_id":"5851ed8d0cf2b9563974034d","uuidSender":"d3dd0059775e46b1b1a63b94a10d2450","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1481764237412},{"_id":"5d087a1f0cf2d7aebbf96fd6","uuidSender":"d6695d1ea7ae480bb2129a3b7d0269ad","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1560836639346},{"_id":"5d703a880cf299e651b71cbb","uuidSender":"cbd9e9009ee94d159d52dff284ad7bf8","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1567636104118},{"_id":"5deeff7f0cf2d87bd75df1a0","uuidSender":"c53c4524ad174fd78134223fddedc484","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1575944063334},{"_id":"5e0638f20cf24f983d2cb02c","uuidSender":"1nmHNU1mERJEG452cx1FazPx3RpAuZ9vW","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1577466098097},{"_id":"5e1e70e70cf2795e4f1322be","uuidSender":"85090a8b495d4856817fd6df1d4da0bd","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1579053287837},{"_id":"5e1fbe4b0cf2795e4f14a36f","uuidSender":"1nmHNU1mERJEG452cx1FazPx3RpAuZ9vW","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1579138635859},{"_id":"5e25910d0cf2a892e569db39","uuidSender":"63ac73044aca4b2f908baff858cf34b9","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1579520269103},{"_id":"5e712f890cf2d292e1148ba6","uuidSender":"1nmHNU1mERJEG452cx1FazPx3RpAuZ9vW","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1584476041946},{"_id":"5e819f550cf2675e4372109b","uuidSender":"8cd38d8f97a24090a43e2e0ce898c521","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1585553237302},{"_id":"5e82c5ad0cf2675e437308aa","uuidSender":"d618457dd6044256bdb287b0df8137d4","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1585628589448},{"_id":"5eae3e420cf26efdbd55b592","uuidSender":"a936c3468bec4a2685199a398212b62d","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1588477506714},{"_id":"5ebd9da40cf22f431e9164d8","uuidSender":"1nmHNU1mERJEG452cx1FazPx3RpAuZ9vW","uuidReceiver":"71ef88df5f7e482fb472f344965beba8","started":1589484964296}]
}
Here is my code:
def get_friend_count(name):
getUUID = f"https://api.mojang.com/users/profiles/minecraft/{name}"
res = requests.get(getUUID)
data = res.json()
if data["id"] is None:
return None
returnUuid = (data["id"])
url1 = f"https://api.hypixel.net/friends?key={API_KEY}&uuid=" + returnUuid
res2 = requests.get(url1)
data2 = res2.json()
if data2["records"] is None:
return None
friend_count = len(data["records"])
return "Friends: " + friend_count
getUUID gets the UUID from the requested username and then uses the UUID to get the players Hypixel stats.
Any help is appreciated, thanks!
Mojang API: https://wiki.vg/Mojang_API
Hypixel API: https://github.com/HypixelDev/PublicAPI/tree/master/Documentation

Did you just forget to request.get(url1)?
In any case, from the mojang API doc, it seems the endpoint you query (https://api.mojang.com/users/profiles/minecraft/) never returns a "records" key ... thus the error when you do if data["records"] is None

Need help adding API PUT method to Python script

I am using the script below to collect inventory information from servers and send it to a product called Device42. The script currently works however one of the APIs that I'm trying to add uses PUT instead of POST. I'm not a programmer and just started using python with this script. This script is using iron python. Can the PUT method be used in this script?
"""
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
##################################################
# a sample script to show how to use
# /api/ip/add-or-update
# /api/device/add-or-update
#
# requires ironPython (http://ironpython.codeplex.com/) and
# powershell (http://support.microsoft.com/kb/968929)
##################################################
import clr
clr.AddReference('System.Management.Automation')
from System.Management.Automation import (
PSMethod, RunspaceInvoke
)
RUNSPACE = RunspaceInvoke()
import urllib
import urllib2
import traceback
import base64
import math
import ssl
import functools
BASE_URL='https://device42_URL'
API_DEVICE_URL=BASE_URL+'/api/1.0/devices/'
API_IP_URL =BASE_URL+'/api/1.0/ips/'
API_PART_URL=BASE_URL+'/api/1.0/parts/'
API_MOUNTPOINT_URL=BASE_URL+'/api/1.0/device/mountpoints/'
API_CUSTOMFIELD_URL=BASE_URL+'/api/1.0/device/custom_field/'
USER ='usernme'
PASSWORD ='password'
old_init = ssl.SSLSocket.__init__
#functools.wraps(old_init)
def init_with_tls1(self, *args, **kwargs):
kwargs['ssl_version'] = ssl.PROTOCOL_TLSv1
old_init(self, *args, **kwargs)
ssl.SSLSocket.__init__ = init_with_tls1
def post(url, params):
"""
http post with basic-auth
params is dict like object
"""
try:
data= urllib.urlencode(params) # convert to ascii chars
headers = {
'Authorization' : 'Basic '+ base64.b64encode(USER + ':' + PASSWORD),
'Content-Type' : 'application/x-www-form-urlencoded'
}
req = urllib2.Request(url, data, headers)
print '---REQUEST---',req.get_full_url()
print req.headers
print req.data
reponse = urllib2.urlopen(req)
print '---RESPONSE---'
print reponse.getcode()
print reponse.info()
print reponse.read()
except urllib2.HTTPError as err:
print '---RESPONSE---'
print err.getcode()
print err.info()
print err.read()
except urllib2.URLError as err:
print '---RESPONSE---'
print err
def to_ascii(s):
# ignore non-ascii chars
return s.encode('ascii','ignore')
def wmi(query):
return [dict([(prop.Name, prop.Value) for prop in psobj.Properties]) for psobj in RUNSPACE.Invoke(query)]
def closest_memory_assumption(v):
return int(256 * math.ceil(v / 256.0))
def add_or_update_device():
computer_system = wmi('Get-WmiObject Win32_ComputerSystem -Namespace "root\CIMV2"')[0] # take first
bios = wmi('Get-WmiObject Win32_BIOS -Namespace "root\CIMV2"')[0]
operating_system = wmi('Get-WmiObject Win32_OperatingSystem -Namespace "root\CIMV2"')[0]
environment = wmi('Get-WmiObject Win32Reg_ESFFarmNode -Namespace "root\CIMV2"')[0]
mem = closest_memory_assumption(int(computer_system.get('TotalPhysicalMemory')) / 1047552)
dev_name = to_ascii(computer_system.get('Name')).upper()
fqdn_name = to_ascii(computer_system.get('Name')).upper() + '.' + to_ascii(computer_system.get('Domain')).lower()
device = {
'memory' : mem,
'os' : to_ascii(operating_system.get('Caption')),
'osver' : operating_system.get('OSArchitecture'),
'osmanufacturer': to_ascii(operating_system.get('Manufacturer')),
'osserial' : operating_system.get('SerialNumber'),
'osverno' : operating_system.get('Version'),
'service_level' : environment.get('Environment'),
'notes' : 'Test w/ Change to Device name collection'
}
devicedmn = ''
for dmn in ['Domain1', 'Domain2', 'Domain3', 'Domain4', 'Domain5']:
if dmn == to_ascii(computer_system.get('Domain')).strip():
devicedmn = 'Domain'
device.update({ 'name' : fqdn_name, })
break
if devicedmn != 'Domain':
device.update({
'name': dev_name,
})
manufacturer = ''
for mftr in ['VMware, Inc.', 'Bochs', 'KVM', 'QEMU', 'Microsoft Corporation', 'Xen']:
if mftr == to_ascii(computer_system.get('Manufacturer')).strip():
manufacturer = 'virtual'
device.update({ 'manufacturer' : 'vmware', })
break
if manufacturer != 'virtual':
device.update({
'manufacturer': to_ascii(computer_system.get('Manufacturer')).strip(),
'hardware': to_ascii(computer_system.get('Model')).strip(),
'serial_no': to_ascii(bios.get('SerialNumber')).strip(),
'type': 'Physical',
})
cpucount = 0
for cpu in wmi('Get-WmiObject Win32_Processor -Namespace "root\CIMV2"'):
cpucount += 1
cpuspeed = cpu.get('MaxClockSpeed')
cpucores = cpu.get('NumberOfCores')
if cpucount > 0:
device.update({
'cpucount': cpucount,
'cpupower': cpuspeed,
'cpucore': cpucores,
})
hddcount = 0
hddsize = 0
for hdd in wmi('Get-WmiObject Win32_LogicalDisk -Namespace "root\CIMV2" | where{$_.Size -gt 1}'):
hddcount += 1
hddsize += hdd.get('Size') / 1073741742
if hddcount > 0:
device.update({
'hddcount': hddcount,
'hddsize': hddsize,
})
post(API_DEVICE_URL, device)
for hdd in wmi('Get-WmiObject Win32_LogicalDisk -Namespace "root\CIMV2" | where{$_.Size -gt 1}'):
mountpoint = {
'mountpoint' : hdd.get('Name'),
'label' : hdd.get('Caption'),
'fstype' : hdd.get('FileSystem'),
'capacity' : hdd.get('Size') / 1024 / 1024,
'free_capacity' : hdd.get('FreeSpace') / 1024 / 1024,
'device' : dev_name,
'assignment' : 'Device',
}
post(API_MOUNTPOINT_URL, mountpoint)
network_adapter_configuration = wmi('Get-WmiObject Win32_NetworkAdapterConfiguration -Namespace "root\CIMV2" | where{$_.IPEnabled -eq "True"}')
for ntwk in network_adapter_configuration:
for ipaddr in ntwk.get('IPAddress'):
ip = {
'ipaddress' : ipaddr,
'macaddress' : ntwk.get('MACAddress'),
'label' : ntwk.get('Description'),
'device' : dev_name,
}
post(API_IP_URL, ip)
def main():
try:
add_or_update_device()
except:
traceback.print_exc()
if __name__ == "__main__":
main()

Ok first things first you need to understand the difference between PUT and POST. I would write it out but another member of the community gave a very good description of the two here.
Now, yes you can use requests with that script. Here is an example of using the requests library by python, in order to install requests if you have pip installed install it like this:
pip install requests
Now, lest go through some examples of using the Requests library, the documentation can be found here.
HTTP Get Request. So for this example, you call the get function from the request library, give the url as parameter, then you can print out the text from the touple that is returned. Since GET will return something, it will generally be in the text portion of the touple allowing you to print it.
r = requests.get('http://urlhere.com/apistuffhere')
print(r.text)
HTTP POST: Posting to a url, depending on how the API was set up will return something, it generally does for error handling, but you also have to pass in parameters. Here is an example for a POST request to a new user entry. And again, you can print the text from the touple to check the response from the API
payload = {'username': 'myloginname', 'password': 'passwordhere'}
r = requests.post('https://testlogin.com/newuserEntry', params=payload)
print(r.text)
Alternatively you can print just r and it should return you a response 200 which should be successful.
For PUT: You have to keep in mind put responses can not be cacheable, so you can post data to the PUT url, but you will not know if there is an error or not but use the same syntax as POST. I have not tried to print out the text response in a PUT request using the Request library as I don't use PUT in any API I write.
requests.put('http://urlhere.com/putextension')
Now for implementing this into your code, you already have the base of the url, in your post for the login just do:
payload = {'username': USERNAME, 'passwd':PASSWORD}
r = requests.post('https://loginurlhere.com/', params=payload)
#check response by printing text
print (r.text)
As for putting data to an extension of your api, let us assume you already have a payload variable ready with the info you need, for example the API device extension:
requests.put(API_DEVICE, params=payload)
And that should PUT to the url. If you have any questions comment below and I can answer them if you would like.

My standard answer would be to replace urllib2 with the Requests package. It makes doing HTTP work a lot easier.
But take a look at this SO answer for a 'hack' to get PUT working.

How can I implement dynamic routing in Python?

I'm attempting to implement dynamic routing for a web framework. At the moment, the goal is to pass arguments into a function by way of the url. So, if user offers a url of "/page/23", then the route function will extract the "23" which will then be used as a parameter for the page function. I am getting a "keyerror", however.
import re
routing_table = {}
url = "/page/23"
def route(url, func):
key = url
key = re.findall(r"(.+?)/<[a-zA-Z_][a-zA-Z0-9_]*>", url)
if key:
params = re.findall(r"<([a-zA-Z_][a-zA-Z0-9_]*)>", url)
routing_table[key[0]] = [params, func]
else:
routing_table[url] = func
def find_path(url):
if url in routing_table:
return routing_table[url]
else:
return None
def page(page_id):
return "this is page %d" % page_id
route("/page/<page_id>", page)
print(routing_table[url])

When you called route, you used a url equal to "/page/<page_id>", but in the last line, url is a global variable equal to "/page/23".
It looks like there are other problems: replace your last line with
print(routing_table)
to see what you're doing.