I have a method which takes one argument, and post to a host.
def fts(searchstring):
search_string="\""+searchstring+"\""
headers = {'Content-Type': 'application/json',}
data = '{ "explain": true,"fields": ["*"],"highlight": {},"query": { "query": '+search_string+'},"size":0}'
response = requests.post('hostname', headers=headers, data=data, auth=('uname', 'password'))
if response.status_code != 200:
raise Exception("{} - {}".format(response.status_code, response.text))
print(response.json())
I want to run this method in several processes at a time with different arguments. Is this possible? I tried to use multiprocessing but couldn't able to get it.
Thanks
Code pool.map(fts, searchstring) will run simultaneously but it will treat searchstring as list of chars and send every single char to different fts
You need list with all strings
all_results = pool.map(fts, [searchstring1, searchstring2, searchstring3, ...])
BTW: If you will need to send more arguments then you will need list with tuples or sublists.
all_results = pool.map(fts, [(searchstring1, param1), (searchstring2, param2), ...])
and define function as
def fts(args):
searchstring, param = args
Related
I am trying to extract data from a REST API using python and put it into one neat JSON file, and having difficulty. The date is rather lengthy, with a total of nearly 4,000 records, but the max record allowed by the API is 100.
I've tried using some other examples to get through the code, and so far this is what I'm using (censoring the API URL and auth key, for the sake of confidentiality):
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while new_results:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("results", [])
vendors.extend(centiblock)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
For the life of me, I cannot figure out why it isn't working. The output keeps coming out as just:
[
"records"
]
If I play around with the print statement at the end, I can get it to print centiblock (so named for being a block of 100 records at a time) just fine - it gives me 100 records in un-formated text. However, if I try printing vendors at the end, the output is:
['records']
...which leads me to guess that somehow, the vendors array is not getting filled with the data. I suspect that I need to modify the get request where I define new_results, but I'm not sure how.
For reference, this is a censored look at how the json data begins, when I format and print out one centiblock:
{
"records": [
{
"id": "XXX",
"createdTime": "2018-10-15T19:23:59.000Z",
"fields": {
"Vendor Name": "XXX",
"Main Phone": "XXX",
"Street": "XXX",
Can anyone see where I'm going wrong?
Thanks in advance!
When you are extending vendors with centiblock, your are giving a dict to the extend function. extend is expecting an Iterable, so that works, but when you iterate over a python dict, you only iterate over the keys of the dict. In this case, ['records'].
Note as well, that your loop condition becomes False after the first iteration, because centiblock.get("results", []) returns [], since "results" is not a key of the output of the API. and [] has a truthiness value of False.
Hence to correct those errors you need to get the correct field from the API into new_results, and extend vendors with new_results, which is itself an array. Note that on the last iteration, new_results will be the empty list, which means vendors won't be extended with any null value, and will contain exactly what you need:
This should look like:
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while len(new_results) > 0:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("records", [])
vendors.extend(new_results)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
Note that I replaced the while new_results with a while len(new_results)>0 which is equivalent in this case, but more readable, and better practice in general.
I have two functions and I'm using similar parameters but for one they work as expected and for the other, they do not:
from fastapi import FastAPI, Header, HTTPException, Body
#app.post("/portfolio/analytics/carbon-footprint", dependencies=[Depends(api_counter)])
async def getPortfolioCarbonFootprint(
tickers: list = Body(...),
func: str = Body(...),
http_client: aiohttp.ClientSession = fastapi.Depends(http_client)
):
print(tickers)
return res
#historical prices
#app.post("/portfolio/analytics/historicalprices", dependencies=[Depends(api_counter)])
async def getPortfolioHistoricalPrices(
tickers: list = Body(...),
http_client: aiohttp.ClientSession = fastapi.Depends(http_client)
):
print(tickers)
jsonResults = await getHistoricalPrices(tickers)
return jsonResults
For both I send it this json:
{"tickers" : [ "IBM.US", "MSFT.US"]}
the first function works perfectly. The second returns this error:
{
"detail": [
{
"loc": [
"body"
],
"msg": "value is not a valid list",
"type": "type_error.list"
}
]
}
Here's where it gets weird. If I send this:
[ "IBM.US", "MSFT.US"]
then it works as expected.
So function 1, works fine. Function 2 is copied from function 1 and it doesn't accept tickers as input but sending it a raw list works.
The difference between the two functions is the number of parameters to be filled in by the user. In the first function you have tickers and func, and in the second one you only have tickers.
From the FastAPI documentation:
But if you have only a single item body parameter from a Pydantic model Item.
By default, FastAPI will then expect its body directly.
But if you want it to expect a JSON with a key item and inside of it the model contents, you can use the special Body parameter embed
So in the second function, if you want to have a key, you must write:
tickers: list = Body(..., embed=True)
I want to get the top artists from a specific country from the last fm API in JSON and save the name and url in the name and url variables. But it always appears "TypeError: byte indices must be integers". Do you know where is the issue?
Working example:
import requests
api_key = "xxx"
for i in range(2,5):
artists = requests.get('http://ws.audioscrobbler.com/2.0/?method=geo.gettopartists&country=spain&format=json&page='+str(i)+'&api_key='+api_key)
for artist in artists:
print(artist)
#name = artist['topartists']['artist']['name']
#url = artist['topartists']['artist']['url']
You want:
response = requests.get(...)
data = response.json()
for artist in data["topartists"]["artist"]:
name = artist["name"]
# etc
Explanation: requests.get() returns a response object. Iterating over the response object is actually iterating over the raw textual response content, line by line. Since this content is actually json, you want to first decode it to Python (response.json() is mainly a shortcut for json.loads(response.content)). You then get a python dict with, in this case, a single key "topartists" which points to a list of "artist" dicts.
A couple hints:
First you may want to learn to use string formatting instead of string concatenation. This :
'http://ws.audioscrobbler.com/2.0/?method=geo.gettopartists&country=spain&format=json&page='+str(i)+'&api_key='+api_key
is ugly and hardly readable. Using string formatting:
urltemplate = "http://ws.audioscrobbler.com/2.0/?method=geo.gettopartists&country=spain&format=json&page={page}&api_key={api_key}"
url = urltemplate.format(page=i, api_key=api_key)
but actually requests knows how to build a querystring from a dict, so you should really use this instead:
query = {
"method": "geo.gettopartists",
"country":"spain",
"format":"json",
"api_key": api_key
}
url = "http://ws.audioscrobbler.com/2.0/"
for pagenum in range(x, y):
query["page"] = pagenum
response = requests.get(url, params=query)
# etc
Then, you may also want to handle errors - there are quite a few things that can go wrong doing an HTTP request.
def enumerate_all_config_objects(baseDN):
url = 'https://www.AwebsiteThatIWontProvide.com'
payload={"ObjectDN":baseDN,"Pattern":"*aws*","Pattern":"*jkb*"}
r = requests.post(url, verify='/PathToKeyForVerification/', headers=headers,data=json.dumps(payload))
response = r.json()
status = r.status_code
print " "
print "Returned status code:", status
print " "
return response ['Objects'][0]['GUID']
Output:
Returned status code: 200
{11871545-8c5b-4c3c-9609-7372fae1add5}
Process finished with exit code 0
I am trying to return ONLY the "GUID" information from a json request. This works (the 1187154...), as I enter values into the index between ['objects'] and ['guid'], each value is successfully produced from the list. My problem is, even though I am printing out the actual response to verify the output is correct, the final script should not require anything being dumped to a CSV file. I have to perform everything in memory. The next function that I need to create will use the returned GUID values and query the server with those values.
How do I get the items in the list to display from the output of enumerate_all_config_objects? I would like to print them to troubleshoot initially. Then I will comment out this feature and have the second function pull each value from that list and use it.
Two problems:
Print out the list which will always have an unknown number of entries.
Create another function to reference / use the values from that list.
The list is populated correctly, I've verified this. I just don't know how to access it or print it.
If I understood correctly, you are looking to do something like:
def enumerate_all_config_objects(baseDN):
url = 'https://www.AwebsiteThatIWontProvide.com'
payload = {"ObjectDN": baseDN, "Pattern": "*aws*", "Pattern": "*jkb*"}
r = requests.post(url, verify='/PathToKeyForVerification/', headers=headers,data=json.dumps(payload))
response = r.json()
status = r.status_code
return map(lambda x: x["GUID"] , response['Objects'])
def use_guids(guid_list):
#do_stuff, for example, to show the guids:
for guid in guid_list:
print(guid)
use_guids(enumerate_all_config_objects(baseDN=<replaceWithYourParameter>))
Edit : To clear out the questions from your comment, I decided to mock the call to the API which you said already works
def enumerate_all_config_objects():
foo = {"GUID" : 1}
bar = {"GUID" : 2}
baz = {"GUID" : 3}
response = {"Objects": [foo, bar, baz] }
mapped = list(map(lambda x: x["GUID"] , response['Objects']))
return map(lambda x: x["GUID"] , response['Objects'])
def use_guids(guid_list):
#do_stuff, for example, to show the guids:
for guid in guid_list:
print(guid)
use_guids(enumerate_all_config_objects())
prints out
1
2
3
When you want to use the value computed from a function, you need to use the return keyword.
For example return map(lambda x: x["GUID"] , response['Objects']), in this new example would return a map object containing [1, 2, 3]. This return value can then be used such as in my first example by passing it to another function.
In the example just above, the list is passed to the use_guids function, which prints the contents of the list.
Edit 2 : If you really insist on calling a function that handles one GUID, you can do that in this way:
def enumerate_all_config_objects(baseDN):
url = 'https://www.AwebsiteThatIWontProvide.com'
payload = {"ObjectDN": baseDN, "Pattern": "*aws*", "Pattern": "*jkb*"}
r = requests.post(url, verify='/PathToKeyForVerification/', headers=headers,data=json.dumps(payload))
response = r.json()
status = r.status_code
for obj in response['Objects']:
use_guid(obj["GUID"])
def use_guid(guid):
print(guid)
# Do some other stuff.
# ...
enumerate_all_config_objects(baseDN=<replaceWithYourParameter>)
I have a function which require to pass some ids as comma seperated to the url string.
My code:
def shipment(self, orderItemIds):
url = "https://api.flipkart.net/sellers/orders/shipments"
payload = {'orderItemsIds':orderItemIds}
return self.session.get(url, data=payload)
I need to pass id1, id2, id3 so that i get a link as:
https://api.flipkart.net/sellers/orders/shipments?orderItemsIds={id1,id2...}
I tried to pass it as a string. and as a list too. But it didn't worked.
oid = 'id1,id2,id3' # or
oid = ['id1',id2'','id3']
How to do it?
I passed an id as oiids = '230501592'.
On running response.url, it gives me:
https://api.flipkart.net/sellers/orders/shipments?orderItemsIds
It shows that parameter values are not passing in the url string.
I believe what you want is this (where orderItemIds is a list of strings):
def shipment(self, orderItemIds):
url = "https://api.flipkart.net/sellers/orders/shipments"
params = {'orderItemsIds': ','.join(orderItemIds)}
return self.session.get(url, params=params)
The differences from the version you have in the question are that orderItemIds is assumed to be a list of strings and is being joined with commas, and the data keyword is replaced with params, which is the right choice for a GET request.
You can easily send get parameters in python requests library-
data = {
'orderItemsIds': ['id1', 'id2', 'id3']
}
response = requests.get(url, data = data)