We Keep Coding

Timestring passed into URL to output JSON file - Python API Call

I'm getting the following error for my python scraper:
import requests
import json
symbol_id = 'COINBASE_SPOT_BTC_USDT'
time_start = '2022-11-20T17:00:00'
time_end = '2022-11-21T05:00:00'
limit_levels = 100000000
limit = 100000000
url = 'https://rest.coinapi.io/v1/orderbooks/{symbol_id}/history?time_start={time_start}limit={limit}&limit_levels={limit_levels}'
headers = {'X-CoinAPI-Key' : 'XXXXXXXXXXXXXXXXXXXXXXX'}
response = requests.get(url, headers=headers)
print(response)
with open('raw_coinbase_ob_history.json', 'w') as json_file:
json.dump(response.json(), json_file)
with open('raw_coinbase_ob_history.json', 'r') as handle:
parsed = json.load(handle)
with open('coinbase_ob_history.json', 'w') as coinbase_ob:
json.dump(parsed, coinbase_ob, indent = 4)
<Response [400]>
And in my written json file, I'm outputted
{"error": "Wrong format of 'time_start' parameter."}
I assume a string goes into a url, so I flattened the timestring to a string. I don't understand why this doesn't work. This is the documentation for the coinAPI call I'm trying to make with 'timestring'. https://docs.coinapi.io/?python#historical-data-get-4

Incorrect syntax for python. To concatenate strings, stick them together like such:
a = 'a' + 'b' + 'c'

string formatting is invalid, and also need use & in between different url params
# python3
url = f"https://rest.coinapi.io/v1/orderbooks/{symbol_id}/history?time_start={time_start}&limit={limit}&limit_levels={limit_levels}"
# python 2
url = "https://rest.coinapi.io/v1/orderbooks/{symbol_id}/history?time_start={time_start}&limit={limit}&limit_levels={limit_levels}".format(symbol_id=symbol_id, time_start=time_start, limit=limit, limit_levels=limit_levels)
https://docs.python.org/3/tutorial/inputoutput.html
https://docs.python.org/2/tutorial/inputoutput.html

looping through JSON data and inserting all bikestations into a dictionary

I am trying to loop through a JSON file and add data for all the bike stations listed into the dictionary instead of only the final bikestation, which is 502. So if my code was working the way I want it to, The resulting dictionary would have JSON data for all 5 bikestations. I am very new to this and any help is appreciated. here is my code so far:
import json
import urllib.request, urllib.parse, urllib.error
import datetime
import pymongo
stations = (123, 258, 290, 501, 502)
chicagoBikesURL = "https://data.cityofchicago.org/resource/eq45-8inv.json?"
for station in stations:
paramD = dict()
paramD["id"] = station
paramD["$order"] = "timeStamp DESC"
paramD["$limit"] = 2
params = urllib.parse.urlencode(paramD)
print(chicagoBikesURL+params)
document = urllib.request.urlopen(chicagoBikesURL+params)
# get all of the text from the document
text = document.read().decode()
if document.getcode() != 200 :
print("Error code=",document.getcode(), chicagoBikesURL+params)
text = "{}"
# Load the JSON text from the URL into a dictionary using the json library
js = json.loads(text)
# Output first Record
print("\nFirst BikeStation")
print(js[0])
# Write JSON data to a file
fdumps = open('bike_data.txt', "w")
fdumps.write(json.dumps(js).strip())
bikeStation_list.append(js)
# Make sure you close the file otherwise data may not be saved
fdumps.close()
#Process JSON Data
bikeStation_list = []`enter code here`

"w" mode on the file open always creates a new file. So each one overwrites the previous so you will only see the final item.
To append to the file, use:
fdumps = open('bike_data.txt', "a")

Request Status Code 500 when running Python Script

This is what i am suppose to do:
List all files in data/feedback folder
Scan all the files, and make a nested dictionary with Title, Name, Date & Feedback (All the files are in Title,Name, Date & Feedback format with each in a different line of file, that’s why using rstrip function)
Post the dictionary in The given url
Following is my code:
#!/usr/bin/env python3
import os
import os.path
import requests
import json
src = '/data/feedback/'
entries = os.listdir(src)
Title, Name, Date, Feedback = 'Title', 'Name', 'Date', 'Feedback'
inputDict = {}
for i in range(len(entries)):
fileName = entries[i]
completeName = os.path.join(src, fileName)
with open(completeName, 'r') as f:
line = f.readlines ()
line tuple = (line[0],line[1],line[2],line[3])
inputDict[fileName] = {}
inputDict[fileName][Title] = line_tuple[0].rstrip()
inputDict[fileName][Name] = line_tuple[1].rstrip()
inputDict[fileName][Date] = line_tuple[2].rstrip()
inputDict[fileName][Feedback] = line_tuple[3].rstrip()
x = requests.get ("http://website.com/feedback")
print (x.status_code)
r = requests.post ("http://Website.com/feedback” , data=inputDict)
print (r.status_code)
After i run it, get gives 200 code but post gives 500 code.
I just want to know if my script is causing the error or not ?

r = requests.post ("http://Website.com/feedback” , data=inputDict)
If your rest api endpoint is expecting json data then the line above is not doing that; it is sending the dictionary inputDict as form-encoded, as though you were submitting a form on an HTML page.
You can either use the json parameter in the post function, which sets the content-type in the headers to application/json:
r = requests.post ("http://Website.com/feedback", json=inputDict)
or set the header manually:
headers = {'Content-type': 'application/json'}
r = requests.post("http://Website.com/feedback", data=json.dumps(inputDict), headers=headers)

Multiple Json in Flask Response to convert to Pandas DataFrame

I have following JSON structure coming from a Flask Rest API.
It is more than one JSON based on how many assets we query for and I am not able
to convert it to Pandas dataframe.
from flask import Flask
import requests
import pandas as pd
import json
url = "http://localhost:5000/getpqdata"
random_cols = ['AAPL', 'MSFT']
JsonOutput = {'Assets': random_cols}
headers = {'Content-type': 'application/json'}
response = requests.post(url, json=JsonOutput, headers=headers)
rawdata = response.text
rawdata is coming as below:
rawdata = '''[{"APPL": 1.067638}, {"AAPL": -1.996081}]
[{"MSFT": 0.086638}, {"MSFT": -0.926081}]'''
data = json.loads(rawdata)
df = pd.DataFrame(data)
print(df)
It gives following error.
C:\Python36>python D:\Python\pyarrow\RestfulAPI\test.py
Traceback (most recent call last):
File "D:\Python\pyarrow\RestfulAPI\test.py", line 36, in <module>
data = json.loads(rawdata)
File "C:\Python36\lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Python36\lib\json\decoder.py", line 342, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 13 (char 54)

The problem you are having does not have anything to do with pandas but rather with the JSON decoding. json.loads(...) only supports one JSON object. Your rawdata has 2 JSON objects in it. Thus when it reaches the second line it tells you there is extra data. You can see a potential solution to that in this answer.
In short, you can do something like this:
def parse_json_stream(stream):
decoder = json.JSONDecoder()
while stream:
obj, idx = decoder.raw_decode(stream)
yield obj
stream = stream[idx:].lstrip()
parsed_data = list(parse_json_stream(rawdata))
print(parsed_data)
[[{'APPL': 1.067638}, {'AAPL': -1.996081}], [{'MSFT': 0.086638}, {'MSFT': -0.926081}]]
As for converting it to a DataFrame, it depends on how you want to organize your data.

loading json from text file

I am trying to run this code but it creates error.
import json
import requests
import pprint
data = []
with open('data.txt') as o1:
for line in o1:
data.append(json.loads(line))
print(data)
print(" \n")
print(data)
url = 'http://xyz.abcdfx.in/devicedata'
body_json=json.dumps(data)
headers = {'Content-Type':'application/json'}
d = requests.post(url, data = body_json, headers=headers)
pprint.pprint(d.json())
it shows
Value Error: No json object could be Decoded
I am new to programming and not able to figure out what is the problem.

It seems like you are trying to parse the json file line by line, but the json objects may (and usually are) span more than one line. You need to have the entire file in order to parse it:
with open('data.txt') as o1:
data = json.loads(o1.read()) # read ALL the file and parse. no loops
print(data)

i solved my problem using this:
data =[]
with open('data.txt') as f:
for line in f:
data = json.loads(line)
print(data)
url = 'http://xyz.abcdfx.cn/devicedata'
body_json=json.dumps(data)
headers = {'Content-Type':'application/json'}
d = requests.post(url, data = body_json, headers=headers)
pprint.pprint(d.json())