Convert JSon object to Json array/Python List - python

I need to read keys in the Json file to later use them as columns and insert/update with the values pertaining to those Json file keys. The problem is that my Json has the first element as a Json Object (see code below).
Json:
{
"metadata":
{
"namespace": "5.2.0",
"message_id": "3c80151b-fcf3-4cc3-ada0-635be5b5c95f",
"transmit_time": "2020-01-30T11:25:47.247394-06:00",
"message_type": "pricing",
"domain": "Pricing Service",
"version": "1.0.0"
}
,
"prices": [
{
"price": 24.99,
"effective_date": "2019-06-01T00:00:00-05:00",
"strikethrough": 34.99,
"expiration_date": "2019-06-01T00:00:00-05:00",
"modified_date": "2019-08-30T02:14:39.044968-05:00",
"base_price": 25.99,
"sku_id": 341214,
"item_number": 244312,
"trade_base_price": 14.99,
"competitive_price": 20.00
},
{
"price": 24.99,
"effective_date": "2019-06-01T00:00:00-05:00",
"strikethrough": 34.99,
"expiration_date": "2019-06-01T00:00:00-05:00",
"modified_date": "2019-08-30T02:14:39.044968-05:00",
"base_price": 25.99,
"sku_id": 674523,
"item_number": 279412,
"trade_base_price": 14.99,
"competitive_price": 20.00
}
]
}
So when I read the "metadata" using get_data function below
SQL Postgres Table:
DROP TABLE MyTable;
CREATE TABLE IF NOT EXISTS MyTable
(
price numeric(5,2),
effective_date timestamp without time zone,
strikethrough numeric(5,2),
expiration_date timestamp without time zone,
modified_date timestamp without time zone,
base_price numeric(5,2),
sku_id integer CONSTRAINT PK_MyPK PRIMARY KEY NOT NULL,
item_number integer,
trade_base_price numeric(5,2),
competitive_price numeric(5,2),
namespace character varying(50),
message_id character varying(50),
transmit_time timestamp without time zone,
message_type character varying(50),
domain character varying(50),
version character varying(50)
)
Python 3.9:
import psycopg2
import json
# import the psycopg2 database adapter for PostgreSQL
from psycopg2 import connect, Error
with open("./Pricing_test.json") as arq_api:
read_data = json.load(arq_api)
# converts Json oblect "metadata" to a Json Array of Objects/Python list
read_data["metadata"] = [{key:value} for key,value in read_data["metadata"].items()] #this dies not work properly as "post_gre" function below only reads the very last key in the Json Array of Objects
#print(read_data)
data_pricing = []
def get_PricingData():
list_1 = read_data["prices"]
for dic in list_1:
price = dic.get("price")
effective_date = dic.get("effective_date")
strikethrough = dic.get("strikethrough")
expiration_date = dic.get("expiration_date")
modified_date = dic.get("modified_date")
base_price = dic.get("base_price")
sku_id = dic.get("sku_id")
item_number = dic.get("item_number")
trade_base_price = dic.get("trade_base_price")
competitive_price = dic.get("competitive_price")
data_pricing.append([price, effective_date, strikethrough, expiration_date, modified_date, base_price, sku_id, item_number, trade_base_price, competitive_price, None, None, None, None, None, None])
get_PricingData()
data_metadata = []
def get_Metadata():
list_2 = read_data["metadata"]
for dic in list_2:
namespace = dic.get("namespace")
message_id = dic.get("message_id")
transmit_time = dic.get("transmit_time")
message_type = dic.get("message_type")
domain = dic.get("domain")
version = dic.get("version")
#if len(namespace) == 0:
#data_pricing.append([None, None, None, None, None, version])
#else:
#for sub_dict in namespace:
#namespace = sub_dict.get("namespace")
#message_id = sub_dict.get("message_id")
#transmit_time = sub_dict.get("transmit_time")
#message_type = sub_dict.get("message_type")
#domain = sub_dict.get("domain")
#data_pricing.append([group_id, group_name, subgrop_id, subgrop_name, None, None, None])
data_metadata.append([namespace, message_id, transmit_time, message_type, domain, version])
get_Metadata()
conn = connect(
host="MyHost",
database="MyDB",
user="MyUser",
password="MyPassword",
# attempt to connect for 3 seconds then raise exception
connect_timeout = 3
)
cur = conn.cursor()
cur.execute("TRUNCATE TABLE MyTable") #comment this one out to avoid sku_id PK violation error
def post_gre():
for item in data_pricing:
my_Pricingdata = tuple(item)
cur.execute("INSERT INTO MyTable VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", my_Pricingdata)
#upades with metadata
for item2 in data_metadata:
my_Metadata = tuple(item2)
cur.execute("UPDATE MyTable SET namespace = %s, message_id = %s, transmit_time = %s, message_type = %s, domain = %s, version = %s", my_Metadata)
post_gre()
conn.commit()
conn.close()
it throughs me the following error:
namespace = dic.get("namespace") AttributeError: 'str' object has no attribute 'get'
But if I wrap the metadata Json object with array brackets [] (see pic below) it works perfectly fine - It reads every key in the metadata as a separate column (namespace, message_id, transmit_time, message_type, domain, version)
But since I should not modify the JSon source file itself I need to interpret "metadata" to a python List type, so that it could read the keys.
P.S.
Almost right Solution:
read_data["metadata"] = [{key:value} for key,value in read_data["metadata"].items()]
Suggestion provided by Hi #Suraj works, but for some reason it inserts NULL for all "metadata" keys column (namespace, message_id, transmit_time, message_type, domain), except for "version". Any idea why? It does insert correct values when changing the Json by adding []. But should not do it.
I was able to narrow down the issue with not reading other keys in the "metadata", it basically reads only one very last key which happens to "Version", but if you change the order it would read the very last one whatever you change it to (eg.: "domain").

How about now ?
import pandas as pd
import json
with open('stak_flow.json') as f:
data = json.load(f)
data['metadata'] = [{key:value} for key,value in data['metadata'].items()]
print(data)

Related

Querying database for documents within a specified date range using FastAPI and MongoDB

I have a collection in my MongoDB with records that have an attribute "timestamp" which is stored as type datetime
I want to use FastAPI to query and return all records which are within a given range of dates that I am passing on as query parameters.
db = client["tasks"]
async def list_activities(
lower_date: str = "", upper_date: str = ""
):
start_date = datetime.strptime(lower_date, "%d-%m-%Y")
end_date = datetime.strptime(upper_date, "%d-%m-%Y")
activities = (
await db["activities"]
.find({"timestamp": {"$gte": start_date, "$lt": end_date}})
.to_list(1000)
)
return activities
On running this with some parameters for lower and upper dates, I'm being returned an empty list
However I wrote another program to test this
MONGODB_URL = os.getenv("MONGODB_URL")
mc = pymongo.MongoClient(MONGODB_URL)
tgdb = mc["tasks"]
tgcol = tgdb["activities"]
recordno = 1
for x in tgcol.find(
{"timestamp": {"$gte": "2022-01-31 00:00:00", "$lt": "2022-02-06 00:00:00"}}
):
print("Record No: " + str(recordno))
recordno += 1
print(x)
This on the other hand returns all the records within the date range just fine.
Any clue as to what I'm doing wrong with the first piece of code?

How to retrieve customer id from create customer method in Square using Python

I'm creating a customer in square and getting the results as follows. What I need is to get the id of customer.
My code :
from square.client import Client
client = Client(
access_token=settings.SQUARE_ACCESS_TOKEN,
environment=settings.SQUARE_ENVIRONMENT,
)
api_customers = client.customers
request_body = {'idempotency_key': idempotency_key, 'given_name': name, 'company_name': company,'phone_number':phone}
result = api_customers.create_customer(request_body)
And this is the output:
<ApiResponse [{"customer":
{"id": "F8M9KDHWPMYGK2108RMQVQ6FHC",
"created_at": "2020-10-22T09:14:50.159Z",
"updated_at": "2020-10-22T09:14:50Z",
"given_name": "mkv5",
"phone_number": "900000066666",
"company_name": "codesvera",
"preferences": {"email_unsubscribed": false},
"creation_source": "THIRD_PARTY"}
}
]>
Are you using this library ?
https://github.com/square/square-python-sdk/blob/master/square/http/api_response.py
if yes result is an array and APiResponse object.
so first you should do that : result = result.body
then to get the ID: result['customer']['id']
Ps : You have exemple in the github doc :
https://github.com/square/square-python-sdk
# Initialize the customer count
total_customers = 0
# Initialize the cursor with an empty string since we are
# calling list_customers for the first time
cursor = ""
# Count the total number of customers using the list_customers method
while True:
# Call list_customers method to get all customers in this Square account
result = api_customers.list_customers(cursor)
if result.is_success():
# If any customers are returned, the body property
# is a list with the name customers.
# If there are no customers, APIResponse returns
# an empty dictionary.
if result.body:
customers = result.body['customers']
total_customers += len(customers)
# Get the cursor if it exists in the result else set it to None
cursor = result.body.get('cursor', None)
print(f"cursor: {cursor}")
else:
print("No customers.")
break
# Call the error method to see if the call failed
elif result.is_error():
print(f"Errors: {result.errors}")
break
# If there is no cursor, we are at the end of the list.
if cursor == None:
break
print(f"Total customers: {total_customers}")

Is there way to convert pymssql query to json based on first column values?

I am trying to convert pymssql query to json where first column "value" will be "key" and corresponding columns will be list of "key:value".
I have tried using jsondumps but get the "Cursor is not JSON serializable" error:
`conn = pymssql.connect(server, port, db)
cursor = conn.cursor('select u_business_service_display_value as Business, name, host_name as hostname, install_status, ip_address, used_for from cmdb_ci_server where u_patching_director_display_value = <Name> AND install_status <> "Retired" AND install_status <> "Pending Retirement" order by Business, hostname')
for row in cursor:
print("Business=%s, Name=%s, Hostname=%s, install_status=%s, ip_address=%s, used_for=%s" % (row['Business'], row['name'], row['hostname'], row['install_status'], row['ip_address'], row['used_for']))
print json.dumps(results, indent=1)
conn.close()`
Output is
- Business=AAA, Name=Value, Hostname=vaule, install_status=Retired, ip_address=<ip>, used_for=None
- Business=AAA, Name=Value, Hostname=vaule, install_status=Retired, ip_address=<ip>, used_for=None
- Business=BBB, Name=Value, Hostname=vaule, install_status=Installed, ip_address=<ip>, used_for=Prod
- Business=BBB, Name=Value, Hostname=vaule, install_status=Installed, ip_address=<ip>, used_for=Prod
Expected output
{
"AAA":[
{
"Hostname":"Value",
"install_status":"Retired",
"ip_address":"<ip>",
"used_for":"None"
},
{
"Hostname":"Value",
"install_status":"Retired",
"ip_address":"<ip>",
"used_for":"None"
}
],
"BBB":[
{
"Hostname":"Value",
"install_status":"Installed",
"ip_address":"<ip>",
"used_for":"Prod"
},
{
"Hostname":"Value",
"install_status":"Installed",
"ip_address":"<ip>",
"used_for":"Prod"
}
]
}```
from collections import defaultdict
import json
d = defaultdict(list)
for row in cursor:
values = dict()
values['name'] = row['name']
values['hostname'] = row['hostname']
values['install_status'] = row['install_status']
values['ip_address'] = row['ip_address']
values['used_for'] = row['used_for]
d[row['Business']].append(values)
with open('result.json', 'w') as fp:
json.dump(d, fp)
d is defaultdict with list argument so, in case you add key that is not in d it will create list as value when assigned first time.

Dynamodb put_item() overwrites data

Device is my Partition key, table is for putting in multiple different users, under the same Device. However, if I run the following put_item() code, it will overwrite each user if they have the same Device key.
Example: If I put in Monitor, as my device variable, and gomez as my aliasInput variable it runs.
Then run it again as Monitor again as my device variable, but craig as my aliasInput it overwrites my gomez entry.
function to input data into my table :
import boto3
import json
import decimal
import time
import datetime
# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
if o % 1 > 0:
return float(o)
else:
return int(o)
return super(DecimalEncoder, self).default(o)
dynamodb = boto3.resource('dynamodb', region_name='us-west-2', endpoint_url="http://localhost:8000")
table = dynamodb.Table('WishListTest')
device = input('What is the Item being requested?\n')
device = device.upper()
aliasInput = input('What is the Alias of the user?\n')
aliasInput = aliasInput.upper()
date = int((time.strftime("%d%m%Y")))
response = table.put_item(
Item={
'Device': device,
'RequestList': {
'Alias': aliasInput,
'Date': date
},
'AvailableQuanity': 0,
'ReserveQuanity': 0,
}
)
print("PutItem succeeded:")
print(json.dumps(response,
From the docs:
Put item
Creates a new item, or replaces an old item with a new item. If an item that has the same primary key as the new item already exists in the specified table, the new item completely replaces the existing item.
To prevent an overwrite, you need to add a conditional expression
specifying that the partition key does not yet exist.
Something like the following should work (sorry, I didn't quite get your key scheme, so you'll have to modify this).
table.put_item(
Item={'userId': 1, 'productId': 2},
ConditionExpression='userId <> :uid AND productId <> :pid',
ExpressionAttributeValues={':uid': 1, ':pid': 3}
)
You're looking for update_item(). You should use UpdateExpression because AttributeUpdates is deprecated, but this simple example should get you started:
response = table.update_item(
Key={
'Device': device,
},
AttributeUpdates={
'RequestList': {
'Alias': aliasInput,
'Date': date
},
'AvailableQuanity': 0,
'ReserveQuanity': 0,
},
)

Memcache Outputting Null Value

I am trying to implement the pseudocode from the Google documentation, Memcache Examples, so that I can pass it to a dictionary but I am getting a null value. I've researched for solutions, for example, Google App Engine retrieving null values from memcache, but they were unhelpful.
How can I get the output of the_id cached for 500 seconds and returned for use by the update_dict function? What am I doing wrong?
CODE:
def return_id(self):
the_id = str(uuid.uuid1())
data = memcache.get(the_id)
print data
if data is not None:
return data
else:
memcache.add(the_id, the_id, 500)
return data
def update_dict(self):
....
id = self.return_id()
info = {
'id': id,
'time': time
}
info_dump = json.dumps(info)
return info_dump
OUTPUT:
{"id": null, "time": "1506437063"}
This issue has been resolved. The issues were:
my key didn't have a proper string name 'the_id'
I wasn't passing data in my else statement
Solution:
....
the_id = str(uuid.uuid1())
data = memcache.get('the_id') #fix: pass a string for the key name
print data
if data is not None:
return data
else:
data = the_id #fix: added the object that needed to be passed to data
memcache.add('the_id', the_id, 500)
return data
....
OUTPUT:
{"id": "25d853ee-a47d-11e7-8700-69aedf15b2da", "time": "1506437063"}
{"id": "25d853ee-a47d-11e7-8700-69aedf15b2da", "time": "1506437063"}

Categories