We Keep Coding

Convert json query to insert a variable and re-convert it to json query

I am kinda frustrated. I copied the following Metabase query string from the network tab in the browser:
query = "{\"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXX_XXXXXX\"],[\"=\",[\"field\",17599,null],\"**chl-43d813dd-05a7-45b8-a5b0-8eb960289aa5**\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}"
As the next step I wanted to convert it to a String to replace the bold reference with a variable.
The String looks like this:
query = '{"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXXXX-XXXXXXXX\"],[\"=\",[\"field\",17599,null],\"'+channelRef+'\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}'
With
q = json.dumps(query)
the result looks exactly as I want to:
q = "{\"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXXXX-XXXXXXXX\"],[\"=\",[\"field\",17599,null],\"**chl-caabef81-f081-4532-9b6e-ac20b3d4c6cf**\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}"
But when I use this query string to send an API request, I get the following error message(s):
{"via":[{"type":"java.lang.ClassCastException"}],"trace":[],"message":null}
Traceback (most recent call last):
File "c:\Users\XXXX\Documents\XXXXXXXX\Test.py", line 308, in
main()
File "c:\Users\XXXX\Documents\XXXXXXXX\Test.py", line 114, in main some_function(XXXX, window, selected_path)
File "c:\Users\XXXX\Documents\XXXXXXXX\Test.py", line 290, in some_function
dataframe = DataFrame(result)
File "C:\Users\XXXX\AppData\Roaming\Python\Python310\site-packages\pandas\core\frame.py", line 756, in init
raise ValueError("DataFrame constructor not properly called!")
ValueError: DataFrame constructor not properly called!
Does have anyone have an idea?
Thank you very much in advance!

You can use the built-in json module:
import json
query = "{\"database\":17,\"query\":{\"source-table\":963,\"filter\":[\"and\",[\"=\",[\"field\",17580,null],\"XXXXXX_XXXXXX\"],[\"=\",[\"field\",17599,null],\"**chl-43d813dd-05a7-45b8-a5b0-8eb960289aa5**\"]],\"fields\":[[\"field\",17579,null],[\"field\",17569,null],[\"field\",17572,null],[\"field\",17586,null],[\"field\",17592,{\"temporal-unit\":\"default\"}],[\"field\",17611,null],[\"field\",17582,null],[\"field\",17597,null],[\"field\",17603,null],[\"field\",17607,null],[\"field\",17576,null],[\"field\",17588,null],[\"field\",17596,null],[\"field\",17608,null],[\"field\",17587,{\"temporal-unit\":\"default\"}],[\"field\",17578,{\"temporal-unit\":\"default\"}],[\"field\",17602,null],[\"field\",17606,null],[\"field\",17605,{\"temporal-unit\":\"default\"}],[\"field\",17601,null],[\"field\",17590,null],[\"field\",17580,null],[\"field\",17598,{\"temporal-unit\":\"default\"}],[\"field\",17577,null],[\"field\",164910,null],[\"field\",46951,null],[\"field\",46952,{\"temporal-unit\":\"default\"}]]},\"type\":\"query\",\"middleware\":{\"js-int-to-string?\":true,\"add-default-userland-constraints?\":true}}"
my_json = json.loads(query)
# make edit's (works like a dict)
query = json.dumps(my_json)

I don't see a bold reference in your JSON string, but this is all handled with the json library:
import json
query = "YOUR QUERY STRING"
object = json.dumps(query)
# Make your changes to your dict object here
new_query = json.loads(object)

TypeError when trying to get data from JSON

I would like to print specific data in a JSON but I get the following error:
Traceback (most recent call last):
File "script.py", line 47, in <module>
print(link['data.file.url.short'])
TypeError: 'int' object has no attribute '__getitem__'
Here is the JSON:
{
"status":true,
"data":{
"file":{
"url":{
"full":"https://anonfile.com/y000H35fn3/yuh_txt",
"short":"https://anonfile.com/y000H35fn3"
},
"metadata":{
"id":"y000H35fn3",
"name":"yuh.txt",
"size":{
"bytes":0,
"readable":"0 Bytes"
}
}
}
}
}
I'm trying to get data.file.url.short which is the short value of the url
Here is the script in question:
post = os.system('curl -F "file=#' + save_file + '" https://anonfile.com/api/upload')
link = json.loads(str(post))
print(link['data.file.url.short'])
Thanks

Other than os.system() return value mentioned by #John Gordon I think correct syntax to access data.file.url.short is link['data']['file']['url']['short'], since json.loads returns dict.

os.system() does not return the output of the command; it returns the exit status of the command, which is an integer.
If you want to capture the command's output, see this question.

You are capturing the return code of the process created by os.system which is an integer.
Why dont you use the request class in the urllib module to perform that action within python?
import urllib.request
import json
urllib.request.urlretrieve('https://anonfile.com/api/upload', save_file)
json_dict = json.load(save_file)
print(json_dict['data']['file']['url']['short']) # https://anonfile.com/y000H35fn3
Or if you don't need to save the file you can use the requests library:
import requests
json_dict = requests.get('https://anonfile.com/api/upload').json()
print(json_dict['data']['file']['url']['short']) # https://anonfile.com/y000H35fn3

Python3 and hmac . How to handle string not being binary

I had a script in Python2 that was working great.
def _generate_signature(data):
return hmac.new('key', data, hashlib.sha256).hexdigest()
Where data was the output of json.dumps.
Now, if I try to run the same kind of code in Python 3, I get the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.4/hmac.py", line 144, in new
return HMAC(key, msg, digestmod)
File "/usr/lib/python3.4/hmac.py", line 42, in __init__
raise TypeError("key: expected bytes or bytearray, but got %r" %type(key).__name__)
TypeError: key: expected bytes or bytearray, but got 'str'
If I try something like transforming the key to bytes like so:
bytes('key')
I get
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: string argument without an encoding
I'm still struggling to understand the encodings in Python 3.

You can use bytes literal: b'key'
def _generate_signature(data):
return hmac.new(b'key', data, hashlib.sha256).hexdigest()
In addition to that, make sure data is also bytes. For example, if it is read from file, you need to use binary mode (rb) when opening the file.

Not to resurrect an old question but I did want to add something I feel is missing from this answer, to which I had trouble finding an appropriate explanation/example of anywhere else:
Aquiles Carattino was pretty close with his attempt at converting the string to bytes, but was missing the second argument, the encoding of the string to be converted to bytes.
If someone would like to convert a string to bytes through some other means than static assignment (such as reading from a config file or a DB), the following should work:
(Python 3+ only, not compatible with Python 2)
import hmac, hashlib
def _generate_signature(data):
key = 'key' # Defined as a simple string.
key_bytes= bytes(key , 'latin-1') # Commonly 'latin-1' or 'ascii'
data_bytes = bytes(data, 'latin-1') # Assumes `data` is also an ascii string.
return hmac.new(key_bytes, data_bytes , hashlib.sha256).hexdigest()
print(
_generate_signature('this is my string of data')
)

try
codecs.encode()
which can be used both in python2.7.12 and 3.5.2
import hashlib
import codecs
import hmac
a = "aaaaaaa"
b = "bbbbbbb"
hmac.new(codecs.encode(a), msg=codecs.encode(b), digestmod=hashlib.sha256).hexdigest()

for python3 this is how i solved it.
import codecs
import hmac
def _generate_signature(data):
return hmac.new(codecs.encode(key), codecs.encode(data), codecs.encode(hashlib.sha256)).hexdigest()

PYTHON : There is a function similar to ast.literal_eval ()?

I've got a problem with the utilisation of ast.literal_eval(). In the example below, I only want to convert the string (myText) to dictionnary. But ast.literal_eval() try to evaluate <__main__.myClass instance at 0x0000000052D64D88> and give me an error. I completely anderstand this error but I would like to know if there is a way to avoid it (with an other function or with an other way to use the function ast.literal_eval)
import ast
myText = "{<__main__.myClass instance at 0x0000000052D64D88>: value}"
ast.literal_eval(myText)
# Error: invalid syntax
# Traceback (most recent call last):
# File "<maya console>", line 4, in <module>
# File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\ast.py", line 49, in literal_eval
# node_or_string = parse(node_or_string, mode='eval')
# File "C:\Program Files\Autodesk\Maya2016\bin\python27.zip\ast.py", line 37, in parse
# return compile(source, filename, mode, PyCF_ONLY_AST)
# File "<unknown>", line 1
# {<__main__.myClass instance at 0x0000000052D64D88>: value}
# ^
# SyntaxError: invalid syntax #
Thank you in advance for your help !

What you really want to do is dump your data using pickle.dump and load it using pickle.load (or equivalent, such as json, etc.). Using repr(data) to dump the data will cause problems like this.
If you just need to salvage the data you have already generated, you might get away with something like the following:
def my_literal_eval(s):
s = re.sub(r"<__main__.myClass instance at 0x([^>]+)>", r'"<\1>"', s)
dct = ast.literal_eval(s)
return {myClass(): v for v in dct.itervalues()}
Example of usage:
>>> import ast, re
>>> class myClass(object): pass
...
>>> myText = "{<__main__.myClass instance at 0x0000000052D64D88>: {'name': 'theName'}, <__main__.myClass instance at 0x0000000052D73F48>: {'name': 'theName'}}"
>>> my_literal_eval(myText)
{<__main__.myClass object at 0x7fbdc00a4b90>: {'name': 'theName'}, <__main__.myClass object at 0x7fbdc0035550>: {'name': 'theName'}}
This will work only if the myClass instances don't have any useful information, but are only needed for identity. The idea is to first fix up the string by replacing the <__main__.myClass instance ...> strings with something that can be parsed by ast.literal_eval, and then replace those with actual myClass instances - provided these can be constructed without arguments, which hinges on the above assumption.
If this initial assumption doesn't hold, then your data is, as Ignacio put it, irreversibly damaged, and no amount of clever parsing will retrieve the lost bits.

Why I can't use urlencode to encode json format data?

I have a problem about urlencode in python 2.7:
>>> import urllib
>>> import json
>>> urllib.urlencode(json.dumps({'title':"hello world！",'anonymous':False,'needautocategory':True}))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/urllib.py", line 1280, in urlencode
raise TypeError
TypeError: not a valid non-string sequence or mapping object

urlencode can encode a dict, but not a string. The output of json.dumps is a string.
Depending on what output you want, either don't encode the dict in JSON:
>>> urllib.urlencode({'title':"hello world！",'anonymous':False,'needautocategory':True})
'needautocategory=True&anonymous=False&title=hello+world%EF%BC%81'
or wrap the whole thing in a dict:
>>> urllib.urlencode({'data': json.dumps({'title':"hello world！",'anonymous':False,'needautocategory':True})})
'data=%7B%22needautocategory%22%3A+true%2C+%22anonymous%22%3A+false%2C+%22title%22%3A+%22hello+world%5Cuff01%22%7D'
or use quote_plus() instead (urlencode uses quote_plus for the keys and values):
>>> urllib.quote_plus(json.dumps({'title':"hello world！",'anonymous':False,'needautocategory':True}))
'%7B%22needautocategory%22%3A+true%2C+%22anonymous%22%3A+false%2C+%22title%22%3A+%22hello+world%5Cuff01%22%7D'

Because urllib.urlencode "converts a mapping object or a sequence of two-element tuples to a “percent-encoded” string...". Your string is neither of these.
I think you need urllib.quote or urllib.quote_plus.

For those of ya'll getting the error:
AttributeError: module 'urllib' has no attribute 'urlencode'
It's because urllib has been split up in Python 3
Here's the code for Python 3:
import urllib.parse
data = {
"title": "Hello world",
"anonymous": False,
"needautocategory": True
}
urllib.parse.urlencode(data)
# 'title=Hello+world&anonymous=False&needautocategory=True'

(1) Import libraries
import requests
import json
(2) Spec is a dictionary object
spec = {...}
(3) Convert dictionary object to json
data = json.dumps(spec, ensure_ascii=False)
(4) Finally, do request with parameter spec in json format
response = requests.get(
'http://localhost:8080/...',
params={'spec': data}
)
(5) Analyze response ...

json.dumps() returns a string.
urllib.urlencode() expects a query in the format of a mapping object or tuples. Note that it does not expect a string.
You're passing the first as the parameter for the second, resulting in the error.