We Keep Coding

How to get the full XML when decoding bytes to string. It is generating XML but truncated

I am using dicttoxml for generating an XML file in Python.
**xml = dicttoxml(obj, item_func = my_item_func, attr_type=False, custom_root='Benefits')**
and it generates a byte XML which I am again converting to string for my XSD validation.
**strxml = bytes.decode(xml, 'utf-8')**
Issue : When the XML is generated it skips many nodes and are replaced by ..., the reason I think it is doing so because the XML file is very big. I don't want it to skip nodes I want the XML file in its entirety.
However when this "xml" byte object is rendered on the browser or when I print in debug mode there is no issue and I get the XML in it's entirety.
How can I overcome this problem?
Here is the complete code.
from datetime import date
from dicttoxml import dicttoxml
from xml.dom.minidom import parseString
import json
import xmlschema
def response_handler(self,node, mktsegtype):
isvalidxml = False
if node is not None:
obj = json.loads(json.dumps(node,default=lambda o: dict((key, value) for key, value in o.__dict__.items() if value is not None),indent=4,allow_nan=False))
my_item_func = lambda x: 'cvrg' if x == "Covered" else('Insure' if x == "Insurance" else x[:-1])
xml = dicttoxml(obj, item_func = my_item_func, attr_type=False, custom_root='Benefits')
isvalidxml = self.validatexmlwithxsd(xml, mktsegtype)
if(isvalidxml):
return xml
else:
return None
def validatexmlwithxsd(self, xml, mktsegtype):
valid = False
xsd = None
strxml = bytes.decode(xml, 'utf-8')
if(mktsegtype == "XXX"):
xsd = xmlschema.XMLSchema('tests/test_cases/examples/vehicles/vehicles.xsd')
elif(mktsegtype == "YYY"):
xsd = xmlschema.XMLSchema('tests/test_cases/examples/vehicles/boat.xsd')
valid = xsd.is_valid(strxml)
return valid
E.g.of node generated
<cvrg>
<cvrgID>285</cvrgID>
<cvrgCoveredRtl>1</cvrgCoveredRtl>
<cvrgCoveredMail>1</cvrgCoveredMail>
**<cvrgO...goryAgeLimitMax>**
</cvrgAgeLimitMax>
<cvrgOutOfLimitActionAge></cvrgOutOfLimitActionAge>
<cvrgOutOfLimitActionGender></cvrgOutOfLimitActionGender>
</cvrg>
<cvrg>
<cvrgID>559</cvrgID>
<cvrgCoveredRtl>2</cvrgCoveredRtl>
<cvrgCoveredMail>2</cvrgCoveredMail>
<cvrgOutOfLimitActionAge></cvrgOutOfLimitActionAge>
<cvrgOutOfLimitActionGender></cvrgOutOfLimitActionGender>
</cvrg>
Update 1: -
I tried to capture the print output to a variable, based on below url
Python: Assign print output to a variable.
s = StringIO()
print(xml, file=s,flush=True)
result = s.getvalue()
In this case also I getting a truncated XML nodes(with...) but as I mentioned in my original post when I print the byte object in the debug window or render it into the browser I am getting the entire XML. Any suggestion or help!!!

JSON from (RIOT) API Formatted Incorrectly

I am importing JSON data into Python from an API and ran into the following decode error:
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Looking at online examples its immediately clear my JSON data has ' where others have have ".
Ideally, I'd like to know why it's being downloaded in this way. It seems highly likely its an error on my end, not theirs.
I decided it should be easy to correct the JSON format but I have failed here too. Please see the below code for how I obtain the JSON data and my attempt at fixing it.
#----------------------------------------
#---Read this only if you want to download
#---the data yourself.
#----------------------------------------
#Built from 'Towards Data Science' guide
#https://towardsdatascience.com/how-to-use-riot-api-with-python-b93be82dbbd6
#Must first have installed riotwatcher
#Info in my example is made up, I can't supply a real API code or
#I get in trouble. Sorry about this. You could obtain one from their website
#but this would be a lot of faff for what is probably a simple StackOverflow
#question
#If you were to get/have a key you could use the following information:
#<EUW> for region
#<Agurin> for name
#----------------------------------------
#---Code
#----------------------------------------
#--->Set Variables
#Get installed riotwatcher module for
#Python
import riotwatcher
#Import riotwatcher tools.
from riotwatcher import LolWatcher, ApiError
#Import JSON (to read the JSON API file)
import json
# Global variables
# Get new API from
# https://developer.riotgames.com/
api_key = 'RGAPI-XXXXXXX-XXX-XXXX-XXXX-XXXX'
watcher = LolWatcher(api_key)
my_region = 'MiddleEarth'
#need to give path to where records
#are to be stored
records_dir = "/home/solebaysharp/Projects/Riot API/Records"
#--->Obtain initial data, setup new varaibles
#Use 'watcher' to get basic stats and setup my account as a variable (me)
me = watcher.summoner.by_name(my_region, "SolebaySharp")
# Setup retrieval of recent match info
my_matches = watcher.match.matchlist_by_account(my_region, me["accountId"])
print(my_matches)
#--->Download the recent match data
#Define where the JSON data is going to go
recent_matches_index_json = (records_dir + "/recent_matches_index.json")
#get that JSON data
print ("Downloading recent match history data")
file_handle = open(recent_matches_index_json,"w+")
file_handle.write(str(my_matches))
file_handle.close()
#convert it to python
file_handle = open(recent_matches_index_json,)
recent_matches_index = json.load(file_handle)
Except this giver the following error...
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
So instead to correct this I tried:
file_handle = open(recent_matches_index_json)
json_sanitised = json.loads(file_handle.replace("'", '"'))
This returns...
AttributeError: '_io.TextIOWrapper' object has no attribute 'replace'
For the sake of completeness, beneath is a sample of what the JSON looks like. I have added the paragraphs to enhance readability. It does not come this way.
{'matches': [
{'platformId': 'NA1',
'gameId': 5687555181,
'champion': 235,
'queue': 400,
'season': 13,
'timestamp': 1598243995076,
'role': 'DUO_SUPPORT',
'lane': 'BOTTOM'
},
{'platformId': 'NA1',
'gameId': 4965733458,
'champion': 235,
'queue': 400,
'season': 13,
'timestamp': 1598240780841,
'role': 'DUO_SUPPORT',
'lane': 'BOTTOM'
},
{'platformId': 'NA1',
'gameId': 4583215645,
'champion': 111,
'queue': 400,
'season': 13,
'timestamp': 1598236666162,
'role': 'DUO_SUPPORT',
'lane': 'BOTTOM'
}],
'startIndex': 0,
'endIndex': 100,
'totalGames': 186}

This is occurring because Python is converting the JSON to a string (str).
file_handle.write(str(my_matches))
As it doesn't see the difference between ' and " it just goes with its default of '.
We can stop this from happening by using JSON.dumps. This partially (certainly not fully) answers the second part of the question as well, as we're using a correctly formatted JSON command.
The above-mentioned line simply has to be replaced with this:
file_handle.write(json.dumps(my_matches))
This will preserve the JSON formatting.

Convert Bson to Json in python

I have a situation where I can only read BSON file as bytes (it is Apache Beam bytes coder). So I get BSON file content as bytes. Now I try to convert it to JSON. My code is:
from bson import json_util
import apache_beam as beam
class ParseBsontoJson(beam.DoFn):
def process(self, element):
print(type(element))
# data = bson.BSON.decode(bson.BSON(element))
data = element.decode('utf-8')
# data = bson.decode_all(element)
# data2 = json_util.dumps(data)
# print(type(data))
return [data]
p = beam.Pipeline(options=pipeline_options)
# This gives me Pcollection of bytes (elements)
test = (p | 'test_r' >> beam.io.ReadFromText(known_args.input + '/' + 'test.bson', coder=coders.BytesCoder()
| 'test_parse' >> beam.ParDo(ParseBsontoJson())) - here I have problem
data = element.decode('latin-1').encode("utf-8")
data2 = json_util.dumps(data)
print(data2)
Where element is a line in BSON file.
What I get is:
{
"$binary": "w5MBAAAHX2lkAF5cw5/Dvj3Cu3FsIcKMOsO+Am5hbWUABgAAAHNjZW5lAAdhY2NvdW50SWQAXlzDn8OKw6B1OEp4TcKmXQdkYXRhQ2VudGVyAF5cw5/DizpiwrIWY8KLw5EIB3RzaGlydFNpemUAXlzDn8OLPcK7cWwhwow6Sgdvc0ltYWdlAF5cw5/Diz3Cu3FsIcKMOksHcHJvdmlzaW9uZWRCeVdvcmtPcmRlcgBeXMOfw5Q9wrtxbCHCjDpcB3BoeXNpY2FsU2VydmVyTm9kZQBeXMOfw5U9wrtxbCHCjDpmBGJvbmRzAHoAAAADMAByAAAAEGluZGV4AAAAAAAEbmljcwAoAAAAAzAAIAAAAAJtYWMAEgAAADhjOmZkOjFiOjAwOjlmOjk5AAAAB25ldHdvcmsAXlzDn8OmOmLCshZjwovDkTwCaXB2NEFkZHJlc3MADgAAADM4LjEzMy4xNjQuNjAAAAAEbHVucwAUAAAABzAAXlzDn8O+PcK7cWwhwow6w70AAnN0YXR1cwAOAAAAZGVwcm92aXNpb25lZAAJX3VwZGF0ZWQAMMO4w4rCmnABAAAJX2NyZWF0ZWQAMMO4w4rCmnABAAACX2V0YWcAKQAAAGMxN2QyZGNjOGU2ZGQ3ZGQ1NGI1ZGQzMjVlYjkzMDcyZTE2NWVmZjEAAMK5AgAAB19pZABeXMOgCTpiwrIWY8KLw5HCsQJuYW1lAAUAAABhd2F5AAdhY2NvdW50SWQAXlzDn8OKOmLCshZjwovDkQMHZGF0YUNlbnRlcgBeXMOfw4s6YsKyFmPCi8ORCQd0c2hpcnRTaXplAF5cw5/DjDpiwrIWY8KLw5E=",
"$type": "00"
}
I tried another recommendations like for example from similar StackOverflow responses:
bson.decode_all(element)
or
# This give an exception: InvalidBSONbad eoo
data = bson.BSON.decode(bson.BSON(element))
But it doesn't convert it to JSON view.
element.decode('latin-1')
gives
"b²cÓ:nameimpactaccountId^\à=»ql!<RdataCenter^\à=»ql!<UtshirtSize^\à:b²cÒÞosImage^\à:b²cÒßprovisionedByWorkOrder^\à:b²cÒäphysicalServerNode^\à=»ql!<]bonds|0tindexnics(0 mac5d:b1:d1:82:d5:99network^"
Could you help me? I can't find in the internet how I can do this. All my tries receive exceptions.
I am using Python 3.7.

How to split a date time string and replace with date only string?

I am having a problem with splitting this string:
"published": "2018-08-15T08:04:57Z",
I would like to split the 2018-08-15 part from the T08 part. After that the T08... part needs to be removed. This will be applied to every "published": rule in the .json file.
I'll have to do this with Python, as I also convert the XML file to JSON.
So in the convert process I would like to remove the T08... part.
I hope someone can help me and if more clarification is needed, I don't mind giving it :)
Searched the internet, had some look into the .split, .pop etc. methods. I am just a rookie at Python still but I want to learn.
Here is my current code:
import xmltodict
import json
#Searching for .xml file to convert
with open ('../../get_url/chocolatey.xml') as fd:
xmlString = fd.read()
#Converting .xml file
print("XML Input (../../get_url/chocolatey.xml):")
print(xmlString)
#Removing certain Characters from strings in file
jsonString = json.dumps(xmltodict.parse(xmlString), indent=4)
jsonString = jsonString.replace("#", "")
jsonString = jsonString.replace("m:", "")
jsonString = jsonString.replace("d:", "")
#jsonString = jsonString.replace('"', '')
#Printing output in Json format
print("\nJson Output (../../get_url/chocolatey.json):")
print(jsonString)
#Applying output to .json file
with open("chocolatey.json", 'w') as fd:
fd.write(jsonString)
Example of the JSON file
},
"published": "2018-08-15T08:04:57Z",
"updated": "2018-08-15T08:04:57Z",
"author": {
"name": "Microsoft"
},

you can try like this:
timestamp = "2018-08-15T08:04:57Z"
timestamp = timestamp.split("T")[0]
op:
2018-08-15

You can use the dateutil.parser for this.
from dateutil import parser
d = "2018-08-15T08:04:57Z"
dt = parser.parse(d) # parses date string of any format and returns a date time object
print(dt,type(dt))
# outputs 2018-08-15 08:04:57+00:00 <class 'datetime.datetime'>
You can then do use strftime to get the date only or date and time in any format.
print(dt.strftime('%Y-%m-%d')) # You can specify any format you need
# outputs 2018-08-15
Read more about how to get date string from datetime object in any format here.
Example code:
import json
from dateutil import parser
jsonDict = {"published": "2018-08-15T08:04:57Z", "updated": "2018-08-15T08:04:57Z", "author": { "name": "Microsoft"},}
# converting a dictionary object to json String
jsonString = json.dumps(jsonDict)
# converting a json string to json object
jsonObj = json.loads(jsonString)
# replacing the "published" value with date only
jsonObj["published"] = parser.parse("2018-08-15T08:04:57Z").strftime('%Y-%m-%d')
# printing the result
print(jsonObj["published"])
# outputs 2018-08-15
# converting back to json string to print
jsonString = json.dumps(jsonObj)
# printing the json string
print(jsonString)
# ouputs
'''
{"published": "2018-08-15", "updated": "2018-08-15T08:04:57Z", "author":{"name": "Microsoft"}}
'''
You can test the code here

Something like this. Using JSONEncoder
import json
class PublishedEncoder(json.JSONEncoder):
def encode(self, o):
if 'published' in o:
o['published'] = o['published'][:o['published'].find('T')]
return super().encode(o)
data = {1: 'X', 'published': '2018-08-15T08:04:57Z'}
json_str = json.dumps(data, cls=PublishedEncoder)
print(json_str)
output
{"1": "X", "published": "2018-08-15"}

How to parse the JSON coming from all the URL's using Python?

I have a URL which gives me the below JSON String if I hit them on the browser -
Below is my URL, let's say it is URL-A
http://hostnameA:1234/Service/statistics?%24format=json
And below is my JSON String -
{
"description": "",
"statistics": {
"dataCount": 0,
}
}
So like the above URL-A, I have around 5 urls, which also gives me the same JSON String back but it might be possible that dataCount count is not zero in them.
Now what I am supposed to do is, I need to write a Python script which can scan all the 5 URL's and then parse then JSON String to extract the dataCount from it and its count as well. And it should keep on running every few seconds to scan the URL and then parse it.
Below are my URL's
URL-A http://hostnameA:1234/Service/statistics?%24format=json
URL-B http://hostnameB:1234/Service/statistics?%24format=json
URL-C http://hostnameC:1234/Service/statistics?%24format=json
URL-D http://hostnameD:1234/Service/statistics?%24format=json
URL-E http://hostnameE:1234/Service/statistics?%24format=json
And the data I want to see is like this on the console, here dataCount will be actual number
hostnameA - dataCount
hostnameB - dataCount
hostnameC - dataCount
hostnameD - dataCount
hostnameE - dataCount
Is this possible to do it in Python?
UPDATE:-
This is what I have tried -
import urllib, json
url = "http://hostnameA:1234/Service/statistics?%24format=json"
response = urllib.urlopen(url);
data = json.loads(response.read())
print data
But how to do the same thing for other 5 URL's and keep on scanning it every 10 seconds and also extract dataCount from it?

import requests
from time import sleep
def get_data_count(url):
try:
req = requests.get(url)
except requests.ConnectionError:
return 'could not get page'
try:
return int(req.json['statistics']['dataCount'])
except TypeError:
return 'field not found'
except ValueError:
return 'not an integer'
def main():
urls = [
('hostnameA', 'http://hostnameA:1234/Service/statistics?%24format=json'),
('hostnameA', 'http://hostnameB:1234/Service/statistics?%24format=json'),
('hostnameA', 'http://hostnameC:1234/Service/statistics?%24format=json'),
('hostnameA', 'http://hostnameD:1234/Service/statistics?%24format=json'),
('hostnameA', 'http://hostnameE:1234/Service/statistics?%24format=json')
]
while True:
print('')
for name, url in urls:
res = get_data_count(url)
print('{name} - {res}'.format(name=name, res=res))
sleep(10.)
if __name__=="__main__":
main()
Edit: in response to AKIWEB:
Running Python 2.7.6 with requests 0.13.2 on Window 7 x64:
>>> requests.get('http://echo.jsontest.com/key/value/one/two').json
{u'key': u'value', u'one': u'two'}
>>> requests.get('http://echo.jsontest.com/key/value/one/two').json()
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
requests.get('http://echo.jsontest.com/key/value/one/two').json()
TypeError: 'dict' object is not callable
Yes, I am pretty sure.