We Keep Coding

Input CSV file of lat and long coordinates into API to extract the weather data?

Here is my code below where I used long and lat coordinates in locations variable and attached it to the URL via coordinates_str. SInce I have CSV file which has latitude and longitude coordinates of around many locations and then call that CSV file as a input to this API(that needs authentication).
How do I input CSV file into this code instead of locations variable?
import requests
import pprint
locations = [(13.84, -12.57), (12.21, -14.69)]
coordinates_str = ','.join(map(lambda a: ' '.join(f'{f:.3f}' for f in a), locations))
# Replace "poi-settings" with the endpoint you would like to call.
URL = f'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates={coordinates_str}'
TOKEN = 'TOKEN KEY'
# Create session object that can be used for all requests.
session = requests.Session()
session.headers['Authorization'] = 'Token {token}'.format(token=TOKEN)
# Send GET request to UBIconnect.
res = session.get(URL)
res.raise_for_status()
# Decode JSON response.
poi_info = res.json()
pprint.pprint(poi_info, indent=2, compact=True)
Then I tried this way: in place of coordinates_str I did this
import requests
import pprint
import pandas as pd
df = pd.read_csv(r'E:\route_points.csv')
print(df)
# Replace "poi-settings" with the endpoint you would like to call.
URL = f'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?'
TOKEN = 'API TOKEN'
params= {'coordinates':(df)}
# Create session object that can be used for all requests.
session = requests.Session()
session.headers['Authorization'] = 'Token {token}'.format(token=TOKEN)
# Send GET request to UBIconnect.
res = session.get(URL, params= params)
res.raise_for_status()
# Decode JSON response.
poi_info = res.json()
pprint.pprint(poi_info, indent=2, compact=True)
Still not working.
Format needed to call the API from Documentation is:
# Replace "poi-settings" with the endpoint you would like to call.
URL = 'https://ubiconnect-eu.ubimet.com:8090/poi-settings'
TOKEN = '<YOUR TOKEN GOES HERE>'
so I replaced the poi-settings by pinpoint-data
URL = 'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates=longitude<space<latitude'
For Example: I put one coordinate set into API URL
URL = 'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates=132.85 12.84'
then with above URL I get the weather data for that location.

If you just want to submit a block of coordinates at a time from your CSV file then something like the following should suffice:
from itertools import islice
import requests
import pprint
import csv
def grouper(n, iterable):
it = iter(iterable)
return iter(lambda: tuple(islice(it, n)), ())
block_size = 10 # how many pairs to submit per request
TOKEN = 'TOKEN KEY'
# Create session object that can be used for all requests.
session = requests.Session()
session.headers['Authorization'] = 'Token {token}'.format(token=TOKEN)
with open('coordinates.csv', newline='') as f_input:
csv_input = csv.reader(f_input)
header = next(csv_input) # skip the header
for coords in grouper(block_size, csv_input):
coordinates = ','.join(f'{float(long):.3f} {float(lat):.3f}' for long, lat in coords)
print(coordinates)
URL = f'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates={coordinates}'
# Send GET request to UBIconnect.
res = session.get(URL)
res.raise_for_status()
# Decode JSON response.
poi_info = res.json()
pprint.pprint(poi_info, indent=2, compact=True)
(obviously this was not tested - no token). Make sure there are no blank lines in your CSV file.
To output to a file add an output file:
with open('coordinates.csv', newline='') as f_input, open('output.json', 'w', encoding='utf-8') as f_output:
and use this in the pprint() call:
pprint.pprint(poi_info, f_output, indent=2, compact=True)
f_output.write('\n') # add blank line if needed

Hope this is what you are looking for
import csv
locations = list()
with open("foo.csv") as csvf:
csvreader = csv.DictReader(csvf)
for row in csvreader:
locations.append((float(row["lat"]), float(row["long"])))
# now add your code
coordinates_str = ','.join(map(lambda a: ' '.join(f'{f:.3f}' for f in a), locations))

Sleep for X seconds before posting API - Python

I need to post some records to the website. I feel I am done with the complex part - the code itself, now I need to tweak the code so that my account doesn't get blocked when doing the posting - yep, just happened.
#importing libraries
import csv
import json
#changing data type
field_types = [('subject', str),
('description', str),
('email', str)]
output = []
#opening the raw file
with open('file.csv','r',encoding = 'utf-8-sig') as f:
for row in csv.DictReader(f):
row.update((key, conversion(row[key]))
for key, conversion in field_types)
output.append(row) #appending rows
with open('tickets.json','w') as outfile: #saving records as json
json.dump(output,outfile,sort_keys = True, indent = 4)
with open('tickets.json','r')as infile:
indata = json.load(infile)
output =[]
for data in indata:
r= requests.post("https://"+ domain +".domain.com/api/", auth = (api_key, password), headers = headers, json=data)
output.append(json.loads(r.text))
#saving the response code
with open('response.json', 'w') as outfile:
json.dump(output, outfile, indent = 4)
I searched and found time.sleep(5) but now sure how to use it. Will it go before output.append(json.loads(r.text))?

How do I isolate a .json file?

I was trying to split some parts of a .json, to completely isolate parts of a .json file from an API I found.
This is trying to isolate the open share price of any stocks on the internet. I've consulted with Stack Overflow, but I think I may have made a mistake in my paraphrasing.
# example
import sys
import requests
import json
from ticker import *
def main():
stock_ticker = input("Name the stock ticker?\n")
time2 = int(input("How many minutes do you want to view history?\n"))
#separate file to generate URL for API
url = webpage(stock_ticker, time2)
response = requests.get(url)
assert response.status_code == 200
data = json.loads(response.text)
open_share_price = data["Time Series (5min)"]["2019-11-01 16:00:00"]["1. open"]
print(open_share_price)
return 0
if __name__ == "__main__":
sys.exit(main())
Returns
136.800
I've been wanting to get open share prices from different time frames, not just 16 :00:00, and not just at 5 min intervals.
I'm not great at programming, so any help would be gratefully received. Sorry in advance for my conciseness errors
Edit: The link for the data. Sorry I didn't include it the first time around. https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=kmb&interval=5min&apikey=exampleapikey

If you have to more than one element then you should use for-loop
import requests
url = 'https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=kmb&interval=5min&apikey=exampleapikey'
response = requests.get(url)
data = response.json()
for key, val in data["Time Series (5min)"].items():
print(key, val["1. open"])
If you want to keep it as JSON then create new directory to keep values and later save it in file.
import requests
import json
url = 'https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=kmb&interval=5min&apikey=exampleapikey'
response = requests.get(url)
data = response.json()
new_data = dict()
for key, val in data["Time Series (5min)"].items():
new_data[key] = val["1. open"]
#print(new_data)
with open('new_data.json', 'w') as fp:
fp.write(json.dumps(new_data))

Save Scraped Data as CSV file?

I am trying to scrape data from a link that contains JSON data and this is the code:
import requests
import json
parameters = ['a:1','a:2','a:3','a:4','a:3','a:4','a:5','a:6','a:7','a:8','a:9','a:10',]
for item in parameters:
key, value = item.split(':')[0], item.split(':')[1]
url = "https://xxxx.000webhostapp.com/getNamesEnc02Motasel2.php?keyword=%s&type=2&limit=%s" %(key, value)
r = requests.get(url)
cont = json.loads(r.content)
print(cont)
And the output be like
[{'name': 'Absz', 'phone': '66343212'}, {'name': 'ddd ', 'phone': '545432211'}, {'name': 'ezd' 'phone':'54856886'}]
I want to store all the data in a CSV file.
How can I do this?
Also, As you can see I am using parameters list to do multi requests but I think there is a way that I can loop the limit parameter from 1 to 200 without typing every single keyword and number in parameters.
Thanks in advance.

Try the below code it will create csv row wise:
import csv
import json
header = ["name","phone"]
for item in range(1,200):
key, value = 'a', item # Generating key and value from range 1 --> 200
url = "https://xxxx.000webhostapp.com/getNamesEnc02Motasel2.php?keyword=%s&type=2&limit=%s" %(key, value)
r = requests.get(url)
cont = json.loads(r.content)
print(cont)
with open('people.csv', 'a') as writeFile:
writer = csv.writer(writeFile)
writer.writerow(header)
for a_row in cont:
writer.writerow([a_row["name"],a_row["phone"]]) # To write name and phone
Hope this answers your question!!

import requests
import json
import pandas as pd
parameters = ['a:1','a:2','a:3','a:4','a:3','a:4','a:5','a:6','a:7','a:8','a:9','a:10']
results = pd.DataFrame()
for item in parameters:
key, value = item.split(':')
url = "https://xxxx.000webhostapp.com/getNamesEnc02Motasel2.php?keyword=%s&type=2&limit=%s" %(key, value)
r = requests.get(url)
cont = json.loads(r.content)
temp_df = pd.DataFrame(cont)
results = results.append(temp_df)
results.to_csv('path/to/filename.csv', index=False)

Whats the fastest way to get movie info from omdb with python?

I have about 200 thousand imdb_id in a file, and want to get JSON information from these imdb_id using omdb API.
I wrote this code and it works correctly, but it's very slow (3 Seconds for each id, it would take 166 Hours):
import urllib.request
import csv
import datetime
from collections import defaultdict
i = 0
columns = defaultdict(list)
with open('a.csv', encoding='utf-8') as f:
reader = csv.DictReader(f)
for row in reader:
for (k, v) in row.items():
columns[k].append(v)
with open('a.csv', 'r', encoding='utf-8') as csvinput:
with open('b.csv', 'w', encoding='utf-8', newline='') as csvoutput:
writer = csv.writer(csvoutput)
for row in csv.reader(csvinput):
if row[0] == "item_id":
writer.writerow(row + ["movie_info"])
else:
url = urllib.request.urlopen(
"http://www.omdbapi.com/?i=tt" + str(columns['item_id'][i]) + "&apikey=??????").read()
url = url.decode('utf-8')
writer.writerow((row + [url]))
i = i + 1
Whats the fastest way to get movie info from omdb with python ???
**Edited : I wrote this code and after get 1022 url resopnse i hava this error :
import grequests
urls = open("a.csv").readlines()
api_key = '??????'
def exception_handler(request, exception):
print("Request failed")
# read file and put each lines to an LIST
for i in range(len(urls)):
urls[i] = "http://www.omdbapi.com/?i=tt" + str(urls[i]).rstrip('\n') + "&apikey=" + api_key
requests = (grequests.get(u) for u in urls)
responses = grequests.map(requests, exception_handler=exception_handler)
with open('b.json', 'wb') as outfile:
for response in responses:
outfile.write(response.content)
Error is :
Traceback (most recent call last):
File "C:/python_apps/omdb_async.py", line 18, in <module>
outfile.write(response.content)
AttributeError: 'NoneType' object has no attribute 'content'
How can i solve this error ???

This code is IO bound and would benefit greatly from using Python's async/await capabilities. You can loop over your collection of URLs, creating an asynchronously executing request for each, much like the example in this SO question.
Once you're making these requests asynchronously, you may need to throttle your request rate to something within the OMDB API limit.