We Keep Coding

Timestring passed into URL to output JSON file - Python API Call

I'm getting the following error for my python scraper:
import requests
import json
symbol_id = 'COINBASE_SPOT_BTC_USDT'
time_start = '2022-11-20T17:00:00'
time_end = '2022-11-21T05:00:00'
limit_levels = 100000000
limit = 100000000
url = 'https://rest.coinapi.io/v1/orderbooks/{symbol_id}/history?time_start={time_start}limit={limit}&limit_levels={limit_levels}'
headers = {'X-CoinAPI-Key' : 'XXXXXXXXXXXXXXXXXXXXXXX'}
response = requests.get(url, headers=headers)
print(response)
with open('raw_coinbase_ob_history.json', 'w') as json_file:
json.dump(response.json(), json_file)
with open('raw_coinbase_ob_history.json', 'r') as handle:
parsed = json.load(handle)
with open('coinbase_ob_history.json', 'w') as coinbase_ob:
json.dump(parsed, coinbase_ob, indent = 4)
<Response [400]>
And in my written json file, I'm outputted
{"error": "Wrong format of 'time_start' parameter."}
I assume a string goes into a url, so I flattened the timestring to a string. I don't understand why this doesn't work. This is the documentation for the coinAPI call I'm trying to make with 'timestring'. https://docs.coinapi.io/?python#historical-data-get-4

Incorrect syntax for python. To concatenate strings, stick them together like such:
a = 'a' + 'b' + 'c'

string formatting is invalid, and also need use & in between different url params
# python3
url = f"https://rest.coinapi.io/v1/orderbooks/{symbol_id}/history?time_start={time_start}&limit={limit}&limit_levels={limit_levels}"
# python 2
url = "https://rest.coinapi.io/v1/orderbooks/{symbol_id}/history?time_start={time_start}&limit={limit}&limit_levels={limit_levels}".format(symbol_id=symbol_id, time_start=time_start, limit=limit, limit_levels=limit_levels)
https://docs.python.org/3/tutorial/inputoutput.html
https://docs.python.org/2/tutorial/inputoutput.html

Script to check status code of URLs using Python

I want to write script that accept multiple URLs through list or text file and append some string at the end of each URL and check https status code (200, 401 and 403)of each URL and save in separate files.
Here's my code so far:
lst = {'back.sql',
'backup.sql',
'accounts.sql',
'backups.sql',
'clients.sql',
'customers.sql',
'data.sql',
'database.sql',
'database.sqlite',
'users.sql',
'db.sql',
'db.sqlite',
'db_backup.sql',
'dbase.sql',
'dbdump.sql',
'setup.sql',
'sqldump.sql',
'dump.sql',
'mysql.sql',
'sql.sql',
'temp.sql'
}
url_test = 'http://www.Holiday.com/%s/' #This can be modified to accept multiple URLs
for i in lst:
url = url_test %i
print(url) #This can be modified to save results for each http status code

If you want, check status code you have to request each page one by one
from requests import get
lst = {'back.sql',
'backup.sql',
'accounts.sql',
'backups.sql',
'clients.sql',
'customers.sql',
'data.sql',
'database.sql',
'database.sqlite',
'users.sql',
'db.sql',
'db.sqlite',
'db_backup.sql',
'dbase.sql',
'dbdump.sql',
'setup.sql',
'sqldump.sql',
'dump.sql',
'mysql.sql',
'sql.sql',
'temp.sql'
}
url_test = ['http://www.Holiday.com/%s/'] #Create list of url
result_dict = dict()
for i in lst:
for url_from_list in url_test:
url = url_from_list %i
# request and get status code from each page one by one
result_dict[url] = get(url).status_code
result_dict will be a dictionary which will contain url as key and response code as value
Then save it to file
with open('filename.txt', 'w') as file:
for url, status_code in result_dict.items():
line = url+" "+str(status_code)+"\n"
file.write(line)

Input CSV file of lat and long coordinates into API to extract the weather data?

Here is my code below where I used long and lat coordinates in locations variable and attached it to the URL via coordinates_str. SInce I have CSV file which has latitude and longitude coordinates of around many locations and then call that CSV file as a input to this API(that needs authentication).
How do I input CSV file into this code instead of locations variable?
import requests
import pprint
locations = [(13.84, -12.57), (12.21, -14.69)]
coordinates_str = ','.join(map(lambda a: ' '.join(f'{f:.3f}' for f in a), locations))
# Replace "poi-settings" with the endpoint you would like to call.
URL = f'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates={coordinates_str}'
TOKEN = 'TOKEN KEY'
# Create session object that can be used for all requests.
session = requests.Session()
session.headers['Authorization'] = 'Token {token}'.format(token=TOKEN)
# Send GET request to UBIconnect.
res = session.get(URL)
res.raise_for_status()
# Decode JSON response.
poi_info = res.json()
pprint.pprint(poi_info, indent=2, compact=True)
Then I tried this way: in place of coordinates_str I did this
import requests
import pprint
import pandas as pd
df = pd.read_csv(r'E:\route_points.csv')
print(df)
# Replace "poi-settings" with the endpoint you would like to call.
URL = f'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?'
TOKEN = 'API TOKEN'
params= {'coordinates':(df)}
# Create session object that can be used for all requests.
session = requests.Session()
session.headers['Authorization'] = 'Token {token}'.format(token=TOKEN)
# Send GET request to UBIconnect.
res = session.get(URL, params= params)
res.raise_for_status()
# Decode JSON response.
poi_info = res.json()
pprint.pprint(poi_info, indent=2, compact=True)
Still not working.
Format needed to call the API from Documentation is:
# Replace "poi-settings" with the endpoint you would like to call.
URL = 'https://ubiconnect-eu.ubimet.com:8090/poi-settings'
TOKEN = '<YOUR TOKEN GOES HERE>'
so I replaced the poi-settings by pinpoint-data
URL = 'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates=longitude<space<latitude'
For Example: I put one coordinate set into API URL
URL = 'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates=132.85 12.84'
then with above URL I get the weather data for that location.

If you just want to submit a block of coordinates at a time from your CSV file then something like the following should suffice:
from itertools import islice
import requests
import pprint
import csv
def grouper(n, iterable):
it = iter(iterable)
return iter(lambda: tuple(islice(it, n)), ())
block_size = 10 # how many pairs to submit per request
TOKEN = 'TOKEN KEY'
# Create session object that can be used for all requests.
session = requests.Session()
session.headers['Authorization'] = 'Token {token}'.format(token=TOKEN)
with open('coordinates.csv', newline='') as f_input:
csv_input = csv.reader(f_input)
header = next(csv_input) # skip the header
for coords in grouper(block_size, csv_input):
coordinates = ','.join(f'{float(long):.3f} {float(lat):.3f}' for long, lat in coords)
print(coordinates)
URL = f'https://ubiconnect-eu.ubimet.com:8090/pinpoint-data?coordinates={coordinates}'
# Send GET request to UBIconnect.
res = session.get(URL)
res.raise_for_status()
# Decode JSON response.
poi_info = res.json()
pprint.pprint(poi_info, indent=2, compact=True)
(obviously this was not tested - no token). Make sure there are no blank lines in your CSV file.
To output to a file add an output file:
with open('coordinates.csv', newline='') as f_input, open('output.json', 'w', encoding='utf-8') as f_output:
and use this in the pprint() call:
pprint.pprint(poi_info, f_output, indent=2, compact=True)
f_output.write('\n') # add blank line if needed

Hope this is what you are looking for
import csv
locations = list()
with open("foo.csv") as csvf:
csvreader = csv.DictReader(csvf)
for row in csvreader:
locations.append((float(row["lat"]), float(row["long"])))
# now add your code
coordinates_str = ','.join(map(lambda a: ' '.join(f'{f:.3f}' for f in a), locations))

passing value from panda dataframe to http request

I'm not sure how I should ask this question. I'm looping through a csv file using panda (at least I think so). As I'm looping through rows, I want to pass a value from a specific column to run an http request for each row.
Here is my code so far:
def api_request(request):
fs = gcsfs.GCSFileSystem(project=PROJECT)
with fs.open('gs://project.appspot.com/file.csv') as f:
df = pd.read_csv(f,)
value = df[['ID']].to_string(index=False)
print(value)
response = requests.get(REQUEST_URL + value,headers={'accept': 'application/json','ClientToken':TOKEN }
)
json_response = response.json()
print(json_response)
As you can see, I'm looping through the csv file to get the ID to pass it to my request url.
I'm not sure I understand the issue but looking at the console log it seems that print(value) is in the loop when the response request is not. In other words, in the console log I'm seeing all the ID printed but I'm seeing only one http request which is empty (probably because the ID is not correctly passed to it).
I'm running my script with cloud functions.

Actually, forgo the use of the Pandas library and simply iterate through csv
import csv
def api_request(request):
fs = gcsfs.GCSFileSystem(project=PROJECT)
with fs.open('gs://project.appspot.com/file.csv') as f:
reader = csv.reader(f)
next(reader, None) # SKIP HEADERS
for row in reader: # LOOP THROUGH GENERATOR (NOT PANDAS SERIES)
value = row[0] # SELECT FIRST COLUMN (ASSUMED ID)
response = requests.get(
REQUEST_URL + value,
headers={'accept': 'application/json', 'ClientToken': TOKEN }
)
json_response = response.json()
print(json_response)

Give this a try instead:
def api_request(request):
fs = gcsfs.GCSFileSystem(project=PROJECT)
with fs.open('gs://project.appspot.com/file.csv') as f:
df = pd.read_csv(f)
for value in df['ID']:
response = requests.get(
REQUEST_URL + value,
headers = {'accept': 'application/json', 'ClientToken': TOKEN }
)
json_response = response.json()
print(json_response)
As mentioned in my comment, you haven't iterated through the data. What you are seeing is just the string representation of it with linebreaks (which might be why you mistakenly thought to be looping).

loading json from text file

I am trying to run this code but it creates error.
import json
import requests
import pprint
data = []
with open('data.txt') as o1:
for line in o1:
data.append(json.loads(line))
print(data)
print(" \n")
print(data)
url = 'http://xyz.abcdfx.in/devicedata'
body_json=json.dumps(data)
headers = {'Content-Type':'application/json'}
d = requests.post(url, data = body_json, headers=headers)
pprint.pprint(d.json())
it shows
Value Error: No json object could be Decoded
I am new to programming and not able to figure out what is the problem.

It seems like you are trying to parse the json file line by line, but the json objects may (and usually are) span more than one line. You need to have the entire file in order to parse it:
with open('data.txt') as o1:
data = json.loads(o1.read()) # read ALL the file and parse. no loops
print(data)

i solved my problem using this:
data =[]
with open('data.txt') as f:
for line in f:
data = json.loads(line)
print(data)
url = 'http://xyz.abcdfx.cn/devicedata'
body_json=json.dumps(data)
headers = {'Content-Type':'application/json'}
d = requests.post(url, data = body_json, headers=headers)
pprint.pprint(d.json())