We Keep Coding

Python code to create JSON with Marathi language Giving Unreadable JSON

I am trying to create JSON file using python code. file is created successfully with English language but not properly working with Marathi Language.
Please check out code:
import os
import json
jsonFilePath = "E:/file/"
captchaImgLocation = "E:/file/captchaimg/"
path_to_tesseract = r"C:/Program Files/Tesseract-OCR/tesseract.exe"
image_path = r"E:/file/captchaimg/captcha.png"
x = {
"FName": "प्रवीण",
}
# convert into JSON:
y = json.dumps(x, ensure_ascii=False).encode('utf8')
# the result is a JSON string:
print(y.decode())
completeName = os.path.join(jsonFilePath, "searchResult_Unicode.json")
print(str(completeName))
file1 = open(completeName, "w")
file1.write(str(y))
file1.close()
O/P on console:
{"FName": "प्रवीण"}
<br>
File created inside folder like this:
b'{"FName": "\xe0\xa4\xaa\xe0\xa5\x8d\xe0\xa4\xb0\xe0\xa4\xb5\xe0\xa5\x80\xe0\xa4\xa3"}'
There is no run time or compile time error but JSON is created with with above format.
Please suggest me any solution.

Open the file in the encoding you need and then json.dump to it:
import os
import json
data = { "FName": "प्रवीण" }
# Writing human-readable. Note some text viewers on Windows required UTF-8 w/ BOM
# to *display* correctly. It's not a problem with writing, but you can use
# encoding='utf-8-sig' to hint to those programs that the file is UTF-8 if
# you see that issue. MUST use encoding='utf8' to read it back correctly.
with open('out.json', 'w', encoding='utf8') as f:
json.dump(data, f, ensure_ascii=False)
# Writing non-human-readable for non-ASCII, but others will have few
# problems reading it back into Python because all common encodings are ASCII-compatible.
# Using the default encoding this will work. I'm being explicit about encoding
# because it is good practice.
with open('out2.json', 'w', encoding='ascii') as f:
json.dump(data, f, ensure_ascii=True) # True is the default anyway
# reading either one is the same
with open('out.json', encoding='utf8') as f:
data2 = json.load(f)
with open('out2.json', encoding='utf8') as f: # UTF-8 is ASCII-compatible
data3 = json.load(f)
# Round-tripping test
print(data == data2, data2)
print(data == data3, data3)
Output:
True {'FName': 'प्रवीण'}
True {'FName': 'प्रवीण'}
out.json (UTF-8-encoded):
{"FName": "प्रवीण"}
out2.json (ASCII-encoded):
{"FName": "\u092a\u094d\u0930\u0935\u0940\u0923"}

You have encoded the JSON string, so you must either open the file in binary mode or decode the JSON before writing to file, so:
file1 = open(completeName, "wb")
file1.write(y)
or
file1 = open(completeName, "w")
file1.write(y.decode('utf-8'))
Doing
file1 = open(completeName, "w")
file1.write(str(y))
writes the string representation of the bytes to the file, which always the wrong thing to do.

Do you want your json to be human readable? It's usually bad practice since you would never know what encoding to use.
You can write/read your json files with the json module without worrying about encoding:
import json
json_path = "test.json"
x = {"FName": "प्रवीण"}
with open(json_path, "w") as outfile:
json.dump(x, outfile, indent=4)
with open(json_path, "r") as infile:
print(json.load(infile))

How to Replace \n, b and single quotes from Raw File from GitHub?

I am trying to download file from GitHub(raw file) and then run this file as .sql file.
import snowflake.connector
from codecs import open
import logging
import requests
from os import getcwd
import os
import sys
#logging
logging.basicConfig(
filename='C:/Users/abc/Documents/Test.log',
level=logging.INFO
)
url = "https://github.com/raw/abc/master/file_name?token=Anvn3lJXDks5ciVaPwA%3D%3D"
directory = getcwd()
filename = os.path.join(getcwd(),'VIEWS.SQL')
r = requests.get(url)
filename.decode("utf-8")
f = open(filename,'w')
f.write(str(r.content))
with open(filename,'r') as theFile, open(filename,'w') as outFile:
data = theFile.read().split('\n')
data = theFile.read().replace('\n','')
data = theFile.read().replace("b'","")
data = theFile.read()
outFile.write(data)
However I get this error
syntax error line 1 at position 0 unexpected 'b'
My converted sql file has b at the beginning and bunch of newline \n characters in the file. Also the entire output file is in single quotes 'text'. Can anyone help me get rid of these? Looks like replace isn't working.
OS: Windows
Python Version: 3.7.0

You introduced a b'.. prefix by converting the response.content bytes value to a string with str():
>>> import requests
>>> r = requests.get("https://github.com/raw/abc/master/file_name?token=Anvn3lJXDks5ciVaPwA%3D%3D")
>>> r.content
b'Not Found'
>>> str(r.content)
"b'Not Found'"
Of course, the specific dummy URL you gave in your question produces a 404 Not Found response, hence the Not Found content of the response body:
>>> r.status_code
404
so the contents in this demonstration are not actually all that useful. However, even for your real URL you probably want to test for a 200 status code before moving to write the data to a file!
What is going wrong in the above is that str(bytesvalue) converts a bytes object to its representation. You'd normally want to decode a bytes value with a text codec, using the bytes.decode() method. But because you are writing the data to a file here, you should instead just open the file in binary mode and write the bytes object without decoding:
r = requests.get(url)
if r.status_code == 200:
with open(filename, 'wb') as f:
f.write(r.content)
The 'wb' mode opens the file for writing in binary mode. Writing binary content to a binary file is the most efficient; decoding it first then writing to a text file requires that it is encoded again. Better to avoid doing double work.
As a side note: there is no need to join a local filename with getcwd(); relative paths always end up in the current working directory, and otherwise it's better to use os.path.abspath(filename).
You could also trust that GitHub sets the correct character set in the Content-Type headers and have response decode the value to str for you in the form of the response.text attribute:
r = requests.get(url)
if r.status_code == 200:
with open(filename, 'w') as f:
f.write(r.text)
but again, that's really doing extra work for nothing, first decoding the binary content from the request, then encoding again when writing to a text file.
Finally, for larger file responses it is better to stream the data and copy it directly to a file. The shutil.copyfileobj() function can take a raw response fileobject directly, provided you enable transparent transport decompression:
import shutil
r = requests.get(url, stream=True)
if r.status_code == 200:
with open(filename, 'wb') as f:
# enable transparent transport decompression handling
r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)

Depending on your version of Python/OS it could be as simple as changing the file to read/write in binary (and if they're still there then altering where you have the replaces):
with open(filename,'rb') as theFile, open(filename,'wb') as outFile:
outfile.write(str(r.content))
data = theFile.read().split('\n')
data = data.replace('\n','')
data = data.replace("b'","")
outFile.write(data)
It would help to have a copy of the file and the line the error is occurring on.

json2html python lib is not working

I'm trying to create new json file with my custom json input and converting JSON to HTML format and saving into .html file. But I'm getting error while generating JSON and HTML file. Please find my below code - Not sure what I'm doing wrong here:
#!/usr/bin/python
# -*- coding: utf-8 -*-
from json2html import *
import sys
import json
JsonResponse = {
"name": "json2html",
"description": "Converts JSON to HTML tabular representation"
}
def create(JsonResponse):
#print JsonResponse
print 'creating new file'
try:
jsonFile = 'testFile.json'
file = open(jsonFile, 'w')
file.write(JsonResponse)
file.close()
with open('testFile.json') as json_data:
infoFromJson = json.load(json_data)
scanOutput = json2html.convert(json=infoFromJson)
print scanOutput
htmlReportFile = 'Report.html'
htmlfile = open(htmlReportFile, 'w')
htmlfile.write(str(scanOutput))
htmlfile.close()
except:
print 'error occured'
sys.exit(0)
create(JsonResponse)
Can someone please help me resolve this issue.
Thanks!

First, get rid of your try / except. Using except without a type expression is almost always a bad idea. In this particular case, it prevented you from knowing what was actually wrong.
After we remove the bare except:, we get this useful error message:
Traceback (most recent call last):
File "x.py", line 31, in <module>
create(JsonResponse)
File "x.py", line 18, in create
file.write(JsonResponse)
TypeError: expected a character buffer object
Sure enough, JsonResponse isn't a character string (str), but is a dictionary. This is easy enough to fix:
file.write(json.dumps(JsonResponse))
Here is a create() subroutine with some other fixes I recommend. Note that writing the dumping the JSON followed immediately by loading the JSON is usually silly. I left it in assuming that your actual program does something slightly different.
def create(JsonResponse):
jsonFile = 'testFile.json'
with open(jsonFile, 'w') as json_data:
json.dump(JsonResponse, json_data)
with open('testFile.json') as json_data:
infoFromJson = json.load(json_data)
scanOutput = json2html.convert(json=infoFromJson)
htmlReportFile = 'Report.html'
with open(htmlReportFile, 'w') as htmlfile:
htmlfile.write(str(scanOutput))

The error is while writing to the JSON file. Instead of file.write(JsonResponse) you should use json.dump(JsonResponse,file). It will work.

How can i filter out mimetype JSON only from no of files in python

I am writing python script to take one by one file from dir and get it mimetype if it's mimetype is not JSON then i want to ignore it. See below part of my script
for filepath in files:
filename = os.path.basename(filepath)
mimetype = mimetypes.guess_type(filepath, strict=False) //here i want to filter out only JSON file and ignore other one
version = "0"
checksum = "0"
fileext = os.path.splitext(filename)[1].lower()
# get raw file data
with open(filepath, "rb") as fr:
filedata = fr.read()
oldfilesize = len(filedata)
See my comment in above code.. Any resolution???

You could try something like this:
for filepath in files:
filename = os.path.basename(filepath)
mimetype = mimetypes.guess_type(filepath, strict=False)
if mimetype != ('application/json', None):
with open(filepath) as f:
try:
json.load(f)
except ValueError:
# It's not json
continue
# do stuff
but this could be inefficient if there are lots of files, and/or they are large.

Well, mimetypes won't help because the mime type application/json for .json files is not inherent to the file metadata. Rather you use it to to provide file type info to whoever is going to to handle it, e.g Content-Type: application/json in the HTTP response header tells the client that it is JSON.
Anyway, the solution might be as follows,
import json
with open("filename", "rt") as f:
try:
d = json.load(f) # no need to name it if you are just checking
except JSONDecodeError:
# handle it or just pass
else:
# Got a json file, do whatever

Saving a file in a directory but also being able to print it whenever i needto

I am running Python 3.x. So i have been working on some code for fetching data on currencies names around the world from a currency website to get information which the code is as follows
def _fetch_currencies():
import urllib.request
import json
f = urllib.request.urlopen('http://openexchangerates.org/api/currencies.json')
charset = f.info().get_param('charset', 'utf8')
data = f.read()
decoded = json.loads(data.decode(charset))
dumps = json.dumps(decoded, indent=4)
return dumps
I then need to save it as a file locally but having some issue and cant see where.
Here is the code for saving the currencies:
def save_currencies(_fetch_currencies, filename):
sorted_currencies = sorted(decoded.items())
with open(filename, 'w') as my_csv:
csv_writer = csv.writer(my_csv, delimiter=',')
csv_writer.writerows(sorted_currencies)
They just don't seem to work together apart from when i remove the line ' dumps = json.dumps(decoded, indent=4) ' but i need that line to be able to print the file in text, how do i get around deleting this line and still be able to save and print? How do i also pick where it saves?
Any Help will be great, thank you very much anyone and everyone who answers/reads this.

I may be mistaken, but your "decoded" variable should be declared as global in both functions.

I would actually have _fetch_currencies() return a dictionary, and then I would pass that dictionary on to saved_currencies(currencies_decoded, filename). For example:
def _fetch_currencies():
import urllib.request
import json
f = urllib.request.urlopen('http://openexchangerates.org/api/currencies.json')
charset = f.info().get_param('charset', 'utf8')
data = f.read()
decoded = json.loads(data.decode(charset))
return decoded
def save_currencies(currencies_decoded, filename):
sorted_currencies = sorted(currencies_decoded.items())
with open(filename, 'w') as my_csv:
csv_writer = csv.writer(my_csv, delimiter=',')
csv_writer.writerows(sorted_currencies)
my_currencies_decoded = _fetch_currencies()
save_currencies(my_currencies_decoded, "filename.csv")
Furthermore, if you would like to save your csv file to a certain location in your filesystem, you can import os and use the os.path.join() function and provide it the FULL path. For example, to save your .csv file to a location called "/Documents/Location/Here", you can do:
import os
def save_currencies(currencies_decoded, filename):
sorted_currencies = sorted(currencies_decoded.items())
with open(os.path.join("Documents","Location","Here"), 'w') as my_csv:
csv_writer = csv.writer(my_csv, delimiter=',')
csv_writer.writerows(sorted_currencies)
You can also use a relative path, so that if you're already in directory "Documents", and you'd like to save a file in "/Documents/Location/Here", you can instead just say:
with open(os.path.join("Location", "Here"), 'w') as my_csv: