I'm trying to convert a dictionary to bytes but facing issues in converting it to a correct format.
First, I'm trying to map an dictionary with an custom schema. Schema is defined as follows -
class User:
def __init__(self, name=None, code=None):
self.name = name
self.code = code
class UserSchema:
name = fields.Str()
code = fields.Str()
#post_load
def create_userself, data):
return User(**data)
My Dictionary structure is as follows-
user_dict = {'name': 'dinesh', 'code': 'dr-01'}
I'm trying to map the dictionary to User schema with the below code
schema = UserSchema(partial=True)
user = schema.loads(user_dict).data
While doing, schema.loads expects the input to be str, bytes or bytearray. Below are the steps that I followed to convert dictionary to Bytes
import json
user_encode_data = json.dumps(user_dict).encode('utf-8')
print(user_encode_data)
Output:
b'{"name ": "dinesh", "code ": "dr-01"}
If I try to map with the schema I'm not getting the required schema object. But, if I have the output in the format given below I can able to get the correct schema object.
b'{\n "name": "dinesh",\n "code": "dr-01"}\n'
Any suggestions how can I convert a dictionary to Bytes?
You can use indent option in json.dumps() to obtain \n symbols:
import json
user_dict = {'name': 'dinesh', 'code': 'dr-01'}
user_encode_data = json.dumps(user_dict, indent=2).encode('utf-8')
print(user_encode_data)
Output:
b'{\n "name": "dinesh",\n "code": "dr-01"\n}'
You can use Base64 library to convert string dictionary to bytes, and although you can convert bytes result to a dictionary using json library. Try this below sample code.
import base64
import json
input_dict = {'var1' : 0, 'var2' : 'some string', 'var1' : ['listitem1','listitem2',5]}
message = str(input_dict)
ascii_message = message.encode('ascii')
output_byte = base64.b64encode(ascii_message)
msg_bytes = base64.b64decode(output_byte)
ascii_msg = msg_bytes.decode('ascii')
# Json library convert stirng dictionary to real dictionary type.
# Double quotes is standard format for json
ascii_msg = ascii_msg.replace("'", "\"")
output_dict = json.loads(ascii_msg) # convert string dictionary to dict format
# Show the input and output
print("input_dict:", input_dict, type(input_dict))
print()
print("base64:", output_byte, type(output_byte))
print()
print("output_dict:", output_dict, type(output_dict))
Output:
>>> print("input_dict:", input_dict, type(input_dict))
input_dict: {'var1': ['listitem1', 'listitem2', 5], 'var2': 'some string'} <class 'dict'>
>>> print()
>>> print("base64:", output_byte, type(output_byte))
base64: b'eyd2YXIxJzogWydsaXN0aXRlbTEnLCAnbGlzdGl0ZW0yJywgNV0sICd2YXIyJzogJ3NvbWUgc3RyaW5nJ30=' <class 'bytes'>
>>> print()
>>> print("output_dict:", output_dict, type(output_dict))
output_dict: {'var1': ['listitem1', 'listitem2', 5], 'var2': 'some string'} <class 'dict'>
Related
I am trying to pass a data back to a URL fetch request. We are using Python 3.x
user_type_data = {'user_type': 'admin',
'user_name': 'myname',
'user_check_flag': 'yes'}
return_data = json.dumps({
l_user_type_data : user_type_data
},default = date_handler)
return return_data
When we do this for a dict I am getting the following error - TypeError("unhashable type: 'dict'"). According to this, it states that we cannot use a dict that is not hashabale - but how do we do this?
How do we fix this?
A valid dictionary key string should be enveloped by quotes or double quotes.
a_dict = {'key': 'value'} # Valid
b_dict = {"key": "value"} # Valid
Or if you wish to assign string that was stored in a variable to be the dictionary key, you can do this instead:
st = "key"
a_dict = dict()
a_dict[st] = 'value'
Since json_dumps requires a valid python dictionary, you may need to rearrange your code.
If the l_user_type_data is a variable contains a string, you should do:
temp_dict = dict()
temp_dict[l_user_type_data] = user_type_data
result = json.dumps(temp_dict, default = date_handler)
Otherwise, if l_user_type_data is a string for the key, just simply enclose that with either single quote or double quotes.
return_data = json.dumps({
"l_user_type_data" : user_type_data
},default = date_handler)
I have a use case where I am reading some data from an API call, but need to transform the data before inserting it into a database. The data comes in a integer format, and I need to save it as a string. The database does not offer a datatype conversion, so the conversion needs to happen in Python before inserting.
Within a config file I have like:
config = {"convert_fields": ["payment", "cash_flow"], "type": "str"}
Then within python I am using the eval() function to check what type to convert the fields to.
So the code ends up being like data['field'] = eval(config['type'])(data['field'])
Does anyone have a better suggestion how I can dynamically change these values, maybe without storing the python class type within a config file.
To add, like sure I could just do str(), but there may be a need to have other fields to convert at some point, which are not string. So I want it to be dynamic, from whatever is defined in the config file for the required conversion fields.
How about using getattr() and __builtins__ that I feel is a little better than exec()/eval() in this instance.
def cast_by_name(type_name, value):
return getattr(__builtins__, type_name)(value)
print(cast_by_name("bool", 1))
Should spit back:
True
You will likely want to include some support for exceptions and perhaps defaults but this should get you started.
#mistermiyagi Points out a critical flaw that of course eval is a bulitin as well. We might want to limit this to safe types:
def cast_by_name(type_name, value):
trusted_types = ["int", "float", "complex", "bool", "str"] ## others as needed
if type_name in trusted_types:
return getattr(__builtins__, type_name)(value)
return value
print(cast_by_name("bool", 1))
Build up a conversion lookup dictionary in advance.
Faster
Easier to debug
config = {"convert_fields":
{"payment" : "str", "cash_flow" : "str", "customer_id" : "int", "name" : "name_it"}
}
def name_it(s : str):
return s.capitalize()
data_in = dict(
payment = 101.00,
customer_id = 3,
cash_flow = 1,
name = "bill",
city = "london"
)
convert_functions = {
#support builtins and custom functions
fieldname : globals().get(funcname) or getattr(__builtins__, funcname)
for fieldname, funcname in config["convert_fields"].items()
if not funcname in {"eval"}
}
print(f"{convert_functions=}")
data_db = {
fieldname :
#if no conversion is specified, use `str`
convert_functions.get(fieldname, str)(value)
for fieldname, value in data_in.items()
}
print(f"{data_db=}")
Output:
convert_functions={'payment': <class 'str'>, 'cash_flow': <class 'str'>, 'customer_id': <class 'int'>, 'name': <function name_it at 0x10f0fbe20>}
data_db={'payment': '101.0', 'customer_id': 3, 'cash_flow': '1', 'name': 'Bill', 'city': 'london'}
if the config could be stored in code, rather than a json-type approach, I'd look into Pydantic though that is not exactly your problem space here:
from pydantic import BaseModel
class Data_DB(BaseModel):
payment : str
customer_id : int
cash_flow : str
#you'd need a custom validator to handle capitalization
name : str
city : str
pydata = Data_DB(**data_in)
print(f"{pydata=}")
print(pydata.dict())
output:
pydata=Data_DB(payment='101.0', customer_id=3, cash_flow='1', name='bill', city='london')
{'payment': '101.0', 'customer_id': 3, 'cash_flow': '1', 'name': 'bill', 'city': 'london'}
The following code is giving me:
Runtime.MarshalError: Unable to marshal response: {'Yes'} is not JSON serializable
from calendar import monthrange
def time_remaining_less_than_fourteen(year, month, day):
a_year = int(input['year'])
b_month = int(input['month'])
c_day = int(input['day'])
days_in_month = monthrange(int(a_year), int(b_month))[1]
time_remaining = ""
if (days_in_month - c_day) < 14:
time_remaining = "No"
return time_remaining
else:
time_remaining = "Yes"
return time_remaining
output = {time_remaining_less_than_fourteen((input['year']), (input['month']), (input['day']))}
#print(output)
When I remove {...} it then throws: 'unicode' object has no attribute 'copy'
I encountered this issue when working with lambda transformation blueprint kinesis-firehose-process-record-python for Kinesis Firehose which led me here. Thus I will post a solution to anyone who also finds this questions when having issues with the lambda.
The blueprint is:
from __future__ import print_function
import base64
print('Loading function')
def lambda_handler(event, context):
output = []
for record in event['records']:
print(record['recordId'])
payload = base64.b64decode(record['data'])
# Do custom processing on the payload here
output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': base64.b64encode(payload)
}
output.append(output_record)
print('Successfully processed {} records.'.format(len(event['records'])))
return {'records': output}
The thing to note is that the Firehose lambda blueprints for python provided by AWS are for Python 2.7, and they don't work with Python 3. The reason is that in Python 3, strings and byte arrays are different.
The key change to make it work with lambda powered by Python 3.x runtime was:
changing
'data': base64.b64encode(payload)
into
'data': base64.b64encode(payload).decode("utf-8")
Otherwise, the lambda had an error due to inability to serialize JSON with byte array returned from base64.b64encode.
David here, from the Zapier Platform team.
Per the docs:
output: A dictionary or list of dictionaries that will be the "return value" of this code. You can explicitly return early if you like. This must be JSON serializable!
In your case, output is a set:
>>> output = {'Yes'}
>>> type(output)
<class 'set'>
>>> json.dumps(output)
Object of type set is not JSON serializable
To be serializable, you need a dict (which has keys and values). Change your last line to include a key and it'll work like you expect:
# \ here /
output = {'result': time_remaining_less_than_fourteen((input['year']), (input['month']), (input['day']))}
I'm trying to serialize the result (a list) of an sqlalchemy query to json.
this is the class:
class Wikilink(Base):
__tablename__='Wikilinks'
__table_args__={'extend_existing':True}
id = Column(Integer,autoincrement=True,primary_key=True)
title = Column(Unicode(350))
user_ip = Column(String(50))
page = Column(String(20))
revision = Column(String(20))
timestamp = Column(String(50))
and I guess my problem is with the __repr__(self): function.
I tried something like:
return '{{0}:{"title":{1}, "Ip":{2}, "page":{3} ,"revision":{4}}}'.format(self.id,self.title.encode('utf-8'),self.user_ip,self.page,self.revision)
or:
return '{"id"={0}, "title"={1}, "Ip"={2}}'.format(self.id,self.title.encode('utf-8'),self.user_ip.encode('utf-8'),self.page,self.revision)
and I got:
TypeError(repr(o) + " is not JSON serializable")
ValueError: Single '}' encountered in format string
I tried:
return '{id=%d, title=%s, Ip=%s}'%(self.id,self.title.encode('utf-8'),self.user_ip.encode('utf-8'))
and I got:
TypeError: {id=8126, title=1 בדצמבר, Ip=147.237.70.106} is not JSON serializable
adding "" around (according to the JSON formatting) like this: "id"="%d", "title"="%s", "Ip"="%s" didn't help either.
I know this is supposed to be dead simple but I just can't get this right
actually bottle is handling the jsonification part automatically, but trying to call json.dumps on the result gives me the same errors.
Instead of trying to convert to json a string, you could define, for example, your own to_dict method that returns the dictionary structure it seems you're trying to create and, after that, generate the json from that structure:
>>> import json
>>> d = {'id':8126, 'title':u'1 בדצמבר', 'ip':'147.237.70.106'}
>>> json.dumps(d)
'{"ip": "147.237.70.106", "id": 8126, "title": "1 \\u05d1\\u05d3\\u05e6\\u05de\\u05d1\\u05e8"}'
I'm not sure I understand what you tried. Couldn't you build the dict and let json.dumps() do the work for you?
Something like:
>>> class Foo:
... id = 1
... title = 'my title'
... to_jsonize = ['id', 'title']
>>>
>>> dct = {name: getattr(Foo,name) for name in Foo.to_jsonize}
>>> import json
>>> json.dumps(dct)
'{"id": 1, "title": "my title"}'
I have A method for format the output as JSON.
My keyword_filter will be pass in this this format:
<QueryDict: {u'customer_type': [u'ABC'], u'tag': [u'2']}>
<QueryDict: {u'customer_type': [u'TDO'], u'tag': [u'3']}>
<QueryDict: {u'customer_type': [u'FRI'], u'tag': [u'2,3']}>
In fact this I got from request.GET (keyword_filter=request.GET)
This is my method: (I am trying)
def save_fiter_to_JSON(self, dest, keyword_filter):
fwrite = open(dest, 'a')
#keyword_filter = <QueryDict: {u'customer_type': [u'FRI'], u'tag': [u'2,3']}>
string_input1 =string.replace(str(keyword_filter), '<QueryDict:', '["name:"')
string_input2 = string.replace(string_input1, '>', '')
fwrite.write(string_input2+",\n")
fwrite.close()
The JSON format that I want:
[
{"name": filter_name, "customer_type": "ABC", "tag": [2,3]},
]
Or the other good one format from you.
import simplejson as json
>>> json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
'["foo", {"bar": ["baz", null, 1.0, 2]}]'
filter_name will be passed from the method save_fiter_to_JSON.
Some tips:
you can convert django's QueryDict to to Python dictionary with dict(keyword_filter) expression,
you can add additional record to the dictionary with dict(keyword_filter, name=filter_name) expression.
Then use json module to dump JSON and write it to the file.
Your question is difficult to understand. I am not sure what you need. Here is my best attempt to solve your problem.
def save_fiter_to_JSON(self, dest, filter_name, keyword_filter):
# start with an empty list
lst = []
# I don't know where you will get your qd (QueryDict instance)
# filter something using keyword_filter? Replace this with actual code
for qd in ??FILTER_SOMETHING??(keyword_filter):
# make a mutable copy of the QueryDict
d = qd.copy()
# update the copy by adding "name"
d["name"] = filter_name
# append dict instance to end of list
lst.append(d)
# get a string with JSON encoding the list
s = json.dumps(lst)
f = open(dest, 'a')
f.write(s + "\n")
f.close()