We Keep Coding

JSON format issue in Python

I am trying to executing rest api get using python script.Below the code I am using.But api response doesn't have proper Json format I am getting error.
import requests
# api-endpoint
URL = "end point url"
# sending get request and saving the response as response object
r = requests.get(url = URL)
# extracting data in json format
data = r.json()
print(data)
My api response below
[{:mac_address=>"10:55", :Parameter=>"Device.Info", :status=>"Success", :response_code=>"200", :value=>"2.4Ghz"}]
I need to change response as below
[{"mac_address":"10:55","Parameter":"Device.Info","status":"Success","response_code":"200","value":"2.4Ghz"}]
How to achieve this in python? I am new to python.

test = '[{:mac_address=>"10:55", :Parameter=>"Device.Info", :status=>"Success", :response_code=>"200", :value=>"2.4Ghz"}]'
def to_json(val: str):
val = val.replace("{:", '{"').replace(" :", '"').replace("=>", '":')
return val
res = to_json(test)
print(res)
result:
[{"mac_address":"10:55","Parameter":"Device.Info","status":"Success","response_code":"200","value":"2.4Ghz"}]
in your case:
import requests
import json
def to_json(val: str):
val = val.replace("{:", '{"').replace(" :", '"').replace("=>", '":')
return val
# api-endpoint
URL = "end point url"
# sending get request and saving the response as response object
r = requests.get(url = URL)
# extracting data in string format
data = r.text
# converting string to json object
res = json.loads(to_json(data))
print(res)

Assuming you get a plain text response, you can manually re-format the response using String.replace():
response = '[{:mac_address=>"10:55", :Parameter=>"Device.Info", :status=>"Success", :response_code=>"200", :value=>"2.4Ghz"}]'
desired = {}
response = response[2:-3].replace('"', '').split(', ')
for r in response:
key, value = r.split('=>')
key = key.replace(':','')
desired[key] = value

Python | NameError: name 'null' is not defined

i have a json response as a string inside a json list
as you in the picture
enter image description here
i trying to get the value inside the string i tired to use eval()
but output shows me this error NameError: name 'null' is not defined
i can't read the json values when they are a string
enter image description here
this is my code :
url = "https://api.pipedream.com/v1/sources/code/event_summaries?
expand=event"
headers = {"Authorization": "Bearer hash "}
response = requests.get(url, headers=headers)
data = response.text
datas = json.loads(data)
darts = datas['data']
for i in darts:
trake = i['event']['body']
for docz in trake:
open_time = open_time = docz['open_time']
print(open_time)
enter image description here
the problem is the json values are string i cannot detect values
By the way the Bearer Authorization is just a demo

The data you needed is inside a dict key. So, you need to use .keys() attribute to retrieve it and then you have to use json.loads() to convert it to a dictionary.
Please check the below code:
import requests
import http.client
import json
from ast import literal_eval as evall
url = "https://api.pipedream.com/v1/sources/code/event_summaries?expand=event"
headers = {"Authorization": "Bearer hash"}
response = requests.get(url, headers=headers)
data = response.text
datas = json.loads(data)
darts = datas['data']
for i in darts:
trake = i['event']['body']
for docz in trake:
print(docz)
for tracks in darts:
tracks = json.loads(list(tracks['event']['body'].keys())[0])
print(tracks)
open_time = tracks['event']['trade'].get('open_time', '')
close_time = tracks['event']['trade'].get('close_time', '')
Lots = tracks['event']['trade'].get('lots', '')
balance = tracks['event']['account'].get('balance', '')
symbol = tracks['event']['trade'].get('symbol', '')
profit = tracks['event']['trade'].get('profit', '')
total_profit = tracks['event']['trade'].get('total_profit', '')
msg = """
Open time : """ +open_time + """
Close time : """ +close_time + """
Symbol : """ +symbol + """
lots : """ +Lots + """
Balance : """ +balance + """
"""
print(msg)
print("success")

How to create an excel file if the script stop executing

I am reading an excel file which contain multiple unique numbers, which i use to perform request activity on a REST API. At the end of the program, I am trying to write another excel file where i am writing the status as 'Success' or 'Fail'.
The issue is, I am reading a file which contain data over 100k numbers. So if my program stop at any reason, or even i stop it intentionally, the output excel file never create. How do i make sure to get the output file till my script run.
here is my code.
from openpyxl import Workbook
from openpyxl import load_workbook
import requests
from datetime import datetime
def api_status():
wk = Workbook()
ws = wk.active
start_row = 2
start_column = 1
status_column = 2
wk = load_workbook("Data-File.xlsx")
source = wk["Sheet"]
global IDNO
for id_list in source['A']:
IDNO = id_list.value
url = "someURL"
payload = {'id_no': str(IDNO)}
headers = {}
response = requests.request("POST", json_url, data=json.dumps(payload), headers=headers)
json_obj = response.json()
ws.cell(row=start_row, column=start_column).value = IDNO
json_message = (json_obj.get('message'))
if json_message == "Success":
ws.cell(row=start_row, column=status_column).value = "Success"
start_row += 1
else:
print("NO")
ws.cell(row=start_row, column=status_column).value = "FAIL"
start_row += 1
wb.save("STATUS-FILE-%s.xlsx" % datetime.now().strftime("%d-%m-%Y_%I-%M-%S_%p"))
api_status()

You just fallback on some code by using a try... except. Replace this line:
api_status()
With this block of code:
try:
api_status()
except:
# CODE TO WRITE YOUR "FAIL" STATUS
Or we can do this for the loop inside the function. Of course there's a little more to it than that. You can specify specific actions to be taken for different error types, or you might want to put that try...except block inside of your function to control for specific lines failing and carrying out the rest.
I'm assuming the most likely error would come from your web request. In that case:
try:
response = requests.request("POST", json_url, data=json.dumps(payload), headers=headers)
json_obj = response.json()
except:
json.obj = {}
By making json_obj an empty dict if the request doesn't work, I guarantee that the next lines would write FAIL in your excel instead of Success
Combining both ideas to make sure your code reaches the save as well would look like this (using a finally to make sure it runs in any possible case):
from openpyxl import Workbook
from openpyxl import load_workbook
import requests
from datetime import datetime
def api_status():
wk = Workbook()
ws = wk.active
start_row = 2
start_column = 1
status_column = 2
wk = load_workbook("Data-File.xlsx")
source = wk["Sheet"]
try:
global IDNO
for id_list in source['A']:
IDNO = id_list.value
url = "someURL"
payload = {'id_no': str(IDNO)}
headers = {}
try:
response = requests.request("POST", json_url, data=json.dumps(payload), headers=headers)
json_obj = response.json()
except:
json.obj = {}
ws.cell(row=start_row, column=start_column).value = IDNO
json_message = (json_obj.get('message'))
if json_message == "Success":
ws.cell(row=start_row, column=status_column).value = "Success"
start_row += 1
else:
print("NO")
ws.cell(row=start_row, column=status_column).value = "FAIL"
start_row += 1
finally:
wb.save("STATUS-FILE-%s.xlsx" % datetime.now().strftime("%d-%m-%Y_%I-%M-%S_%p"))
api_status()

Add key-value in JSON when calling an API

I am calling an API to get a list of properties. I obtain the jsons in the 'listings' file (https://api.nestoria.es/show_example?name=search_listings_es&syntax=1).
I am using zip codes to find each property and I need to add to each property its zip code. At the end I transform the json to a csv. I don't know how to add to each property the zip code (so I would need a new key-value to each result from the API call)...
Thanks!!!!!!!!!
Here it is the code:
from requests import get
import json
import pandas as pd
import time
import datetime
import csv
def get_nestoria(type):
#call the api
api = 'http://api.nestoria.es/api?action=search_listings'
place = '&place_name=' + area_name
listing_type = '&listing_type=' + type
json_es = '&encoding=json&pretty=1&country=es'
page = '&page='
api_input = api + place + listing_type + json_es
response = get(api_input)
# Check if the API has worked
if response.status_code == 200:
print("API called successfully")
elif response.status_code == 400:
print("Wrong request for" + area_name + ". Chechk this area is searchable")
elif response.status_code == 403:
print("Forbidden API call. Maximum number of calls reached.")
else:
print("Wrong code", response.status_code)
content_as_string = response.content.decode()
# Decode JSON
content = json.loads(content_as_string)
content_response = content['response']
# Number of total web pages neded for the area
web_pages = content_response['total_pages']
print('Number of pages in that area: ', web_pages)
print("Numer of total properties " + area_name, content_response['total_results'])
#2nd call to the API
homes = pd.DataFrame()
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0'}
for i in range(1, web_pages+1):
api_input = api + place + listing_type + json_es + page + str(i)
response = get(api_input, headers=headers)
content_as_string = response.content.decode()
content = json.loads(content_as_string)
content_response = content['response']
listings = content_response['listings']
listings = pd.DataFrame(listings)
if i==1:
homes = listings
else:
homes = homes.append(listings, sort=True)
time.sleep(3)
if homes.empty:
homes = homes
else:
homes = homes[['bathroom_number','bedroom_number','car_spaces','commission','construction_year','datasource_name', 'img_height','img_url','img_width', 'floor',
'keywords','latitude','lister_url','listing_type','location_accuracy','longitude','price','price_currency','price_formatted','price_high','price_low',
'property_type','room_number','size','size_type','size_unit','summary','thumb_height','thumb_url','thumb_width','title','updated_in_days','updated_in_days_formatted']]
return homes
homes = pd.DataFrame()
codigos_postales = ['01008']
today=datetime.date.today() #to change the name of the file
for i in codigos_postales:
area_name = i
temp = get_nestoria('buy')
if i == 0:
homes =temp
else:
homes = homes.append(temp,sort=True)
print('Number of extracted properties ', len(homes))
print(homes.head())
homes.to_csv('D:\\a000Master Big Data\\Prácticas\\Web scrapping\\Nestoria\\GranadaVenta'+str(today)+'.csv')

data = response.json()
Here data is the parsed response you get from sending the request.
You can then update the listings like this:
for i in data['listings']:
i['ZipCode'] = zipcode
zipcode being the one you want to assign i.e the one you sent in the request.
You can either convert the data object into dataframe and then call pd.to_csv or use python in-built csv.writer
If you don't have the ZipCode you can use the google maps API to get that
http://maps.googleapis.com/maps/api/geocode/json?address=valencia&sensor=true_or_false&key=YOUR_API_KEY
You will have to Sign Up to get your API key and then you will get zip code from the JSON response.
UPDATE:
Here is an example on how to use it.
api = 'http://api.nestoria.es/api?action=search_listings'
place = '&place_name=' + area_name
listing_type = '&listing_type=' + type
json_es = '&encoding=json&pretty=1&country=es'
page = '&page='
api_input = api + place + listing_type + json_es
response = get(api_input)
update = response.json()['response']['listings']
for i in update:
i['Zipcode'] = zipcode

REST Calls in Python

I'm trying to write a script to port my existing database into Firebase. My data is stored in JSON and I thought I could just pull the JSON and then send that as data into a POST to my Firebase.
def Post_And_Recieve_JSON(url, data, headers):
print("Compiling query...")
Post_And_Recieve_JSON.url = url
Post_And_Recieve_JSON.headers = headers
Post_And_Recieve_JSON.data = (data)
print("Sending request...")
request = urllib.request.Request(url=url, data=data,headers=headers)
print("Recieving response...")
response = urllib.request.urlopen(request)
print("Reading response...")
response_data = response.read()
print("Converting into usable format...")
response_data_JSON = json.loads(response_data.decode(encoding='UTF-8'))
return response_data_JSON
for all_users in existing_database:
full_data.append(Post_And_Recieve_JSON(...)
for item in full_data:
url = 'firebaseurlhere ' + item['profileId'] + '.json'
data = json.dumps(item).encode('ascii')
Post_And_Recieve_JSON(url, data, headers)
Where full_data is a list of JSON objects I've properly pulled from teh existing database.
I'm getting "http.client.BadStatusLine: ''"

I've solved this using the firebase python lib found here: http://ozgur.github.io/python-firebase/
I used pip3 to install it. I just wish I could have done it the same way I do other REST calls instead of requiring a new lib.
full_data = []
from firebase import *
firebase = firebase.FirebaseApplication('https://secret_url.firebaseio.com/', None)
for id in id_list:
print(str(id))
url = 'from_url'
try:
result = firebase.put('/users/ ' + str(id) + '/', data=Post_And_Recieve_JSON(url, None, headers)["payload"], name='Data')
except:
print('Skipping a user')