We Keep Coding

Reading key values a JSON array which is a set in Python

I have the following code
import requests
import json
import sys
credentials_User=sys.argv[1]
credentials_Password=sys.argv[2]
email=sys.argv[3]
def auth_api(login_User,login_Password,):
gooddata_user=login_User
gooddata_password=login_Password
body = json.dumps({
"postUserLogin":{
"login": gooddata_user,
"password": gooddata_password,
"remember":1,
"verify_level":0
}
})
headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}
url="https://reports.domain.com/gdc/account/login"
response = requests.request(
"POST",
url,
headers=headers,
data=body
)
sst=response.headers.get('Set-Cookie')
return sst
def query_api(cookie,email):
url="https://reports.domain.com/gdc/account/domains/domain/users?login="+email
body={}
headers={
'Content-Type': 'application/json',
'Accept': 'application/json',
'Cookie': cookie
}
response = requests.request(
"GET",
url,
headers=headers,
data=body
)
jsonContent=[]
jsonContent.append({response.text})
accountSettings=jsonContent[0]
print(accountSettings)
cookie=auth_api(credentials_User,credentials_Password)
profilehash=query_api(cookie,email)
The code itself works and sends a request to the Gooddata API.
The query_api() function returns JSON similar to below
{
"accountSettings": {
"items": [
{
"accountSetting": {
"login": "user#example.com",
"email": "user#example.com",
"firstName": "First Name",
"lastName": "Last Name",
"companyName": "Company Name",
"position": "Data Analyst",
"created": "2020-01-08 15:44:23",
"updated": "2020-01-08 15:44:23",
"timezone": null,
"country": "United States",
"phoneNumber": "(425) 555-1111",
"old_password": "secret$123",
"password": "secret$234",
"verifyPassword": "secret$234",
"authenticationModes": [
"SSO"
],
"ssoProvider": "sso-domain.com",
"language": "en-US",
"ipWhitelist": [
"127.0.0.1"
],
"links": {
"projects": "/gdc/account/profile/{profile_id}/projects",
"self": "/gdc/account/profile/{profile_id}",
"domain": "/gdc/domains/default",
"auditEvents": "/gdc/account/profile/{profile_id}/auditEvents"
},
"effectiveIpWhitelist": "[ 127.0.0.1 ]"
}
}
],
"paging": {
"offset": 20,
"count": 100,
"next": "/gdc/uri?offset=100"
}
}
}
The issue I am having is reading specific keys from this JSON Dict, I can use accountSettings=jsonContent[0] but that just returns the same JSON.
What I want to do is read the value of the project key within links
How would I do this with a dict?
Thanks

Based on your description, uyou have your value inside a list, (not a set. Foergt about set: sets are not used with JSON). Inside your list, you either your content as a single string, which then you'd have to parse with json.loads, or it is simply a well behaved nested data structure already extracted from JSON, but which is inside a single element list. This seems the most likely.
So, you should be able to do:
accountlink = jsonContent[0]["items"][0]["accountSetting"]["login"]
otherwise, if it is encoded as a a json string, you have to parse it first:
import json
accountlink = json.loads(jsonContent[0])["items"][0]["accountSetting"]["login"]
Now, given your question, I'd say your are on a begginer level as a programmer, or a casual user, just using Python to automatize something either way, I'd recommend you do try some exercising before proceeding: it will save you time (a lot of time). I am not trying to bully or mock anything here: this is the best advice I can offer you. Seek for tutorials that play around on the interactive mode, rather than trying entire programs at once that you'd just copy and paste.

Using the below code fixed the issue
jsonContent=json.loads(response.text)
print(type(jsonContent))
test=jsonContent["accountSettings"]["items"][0]
test2=test["accountSetting"]["links"]["self"]
print(test)
print(test2)
I believe this works because for some reason I didn't notice I was using .append for my jsonContent. This resulted in the data type being something other than it should have been.
Thanks to everyone who tried helping me.

TF400898: An Internal Error Occurred. Activity Id: 1fc05eca-fed8-4065-ae1a-fc8f2741c0ea

i’m trying to push files into git repo via azure API but getting activity_id error. I followed their documentation and trying to add simple file in my repo.
Here is my code:
import requests, base64
pat_token = "xxxx-xxxxx-xxxx"
b64Val = base64.b64encode(pat_token.encode()).decode()
payload = {
"refUpdates": [
{
"name": "refs/heads/main",
"oldObjectId": "505aae1f15ae153b7fc53e8bdb79ac997caa99e6"
}
],
"commits": [
{
"comment": "Added task markdown file.",
"changes": [
{
"changeType": "add",
"item": {
"path": "TimeStamps.txt"
},
"newContent": {
"content": "# Tasks\n\n* Item 1\n* Item 2",
"contentType": "rawtext"
}
}
]
}
]
}
headers = {
'Authorization': 'Basic %s' % b64Val,
'Content-Type': 'application/json',
}
params = (
('api-version', '6.0'),
)
response = requests.post('https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repo}/pushes', headers=headers, data=payload, params=params)
Anyone knows how to solve this issue? I have also added this issue on their developer community

I’ve fixed that error, actually the payload was not in json format so i have to make it as json and after that it worked fine.
Like this
response = requests.post('https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repoId}/pushes', headers=headers, params=params, data=json.dumps(payload))

Getting unecessary quotation marks in Python 2

In a whole context of trying to combine 2 APIs, I need to "combine" two functions results so that everything is more organized.
def descr():
return 88
def name():
return 'Account',descr()
When I print name(), I get this ('Account', 88). Problem with that format is that I can't use this result later in my script.
Here is the whole script :
import requests
import json
url = "https://proxy6.net/api/xxx/getproxy"
def descr():
return 88
def name():
return 'Account',descr()
querystring = {"descr":descr()}
headers = {
'Cache-Control': "no-cache",
'Postman-Token': "xxxx"
}
response = requests.request("GET", url, headers=headers, params=querystring)
data = response.json()
for value in data['list'].values():
host = value['host']
port = value['port']
url = "https://api.multiloginapp.com/v1/profile/create"
querystring = {"token":"xxx"}
payloadobj = {
"generateZeroFingerprintsData": True,
"name": name(),
"OS": "MacOS",
"platform": "MacIntel",
"browserType": "mimic",
"proxyHost": host,
"proxyPort": port,
"proxyIpValidation": False,
"proxyType": "socks5",
"maskFonts": True,
"disablePlugins": True,
"disableWebrtcPlugin": True,
"disableFlashPlugin": True,
"canvasDefType": "noise",
"hardwareConcurrency": 2,
"langHdr": "en-US,en;q=0.8",
"timeZone": "US/Eastern",
"audio": {
"noise": True
},
"geolocation": {
"permitType": "block"
},
"mediaDevices": {
"audioInputs": 1,
"audioOutputs": 1,
"videoInputs": 1
},
"webgl": {
"noise": True
},
"webRtc": {
"type": "block"
},
"shared": False
}
payload = json.dumps(payloadobj)
headers = {
'Content-Type': "application/json",
'Cache-Control': "no-cache",
'Postman-Token': "xxx"
}
response = requests.request("POST", url, data=payload, headers=headers, params=querystring)
print(response.text)
I want the name value in the JSON query above to be the result of name + descr, but it won't work with that returned format.

Looks like you need.
def descr():
return 88
def name():
return '{} {}'.format('Account', descr())
print(name())
Output:
Account 88

name() is returning a tuple object not a string. To return a string you could change it to:
def name():
return "Account {}".format(descr())

If you are using python3.6 or later you can:
def descr():
return 88
def name():
return f"Account {descr()}"

how to feed data to Elasticseach as Integer using Python?

i am using this python script to feed my data to elasticsearch 6.0. How can i store the variable Value with type float in Elasticsearch?
I can't use the metric options for the visualization in Kibana, because all the data is stored automatically as string
from elasticsearch import Elasticsearch
Device=""
Value=""
for key, value in row.items():
Device = key
Value = value
print("Dev",Device, "Val:", Value)
doc = {'Device':Device, 'Measure':Value , 'Sourcefile':filename}
print(' doc: ', doc)
es.index(index=name, doc_type='trends', body=doc)
Thanks
EDIT:
After the advice of #Saul, i could fix this problem with the following code:
import os,csv
import time
from elasticsearch import Elasticsearch
#import pandas as pd
import requests
Datum = time.strftime("%Y-%m-%d_")
path = '/home/pi/Desktop/Data'
os.chdir(path)
name = 'test'
es = Elasticsearch()
#'Time': time ,
#url = 'http://localhost:9200/index_name?pretty'
doc = {
"mappings": {
"doc": {
"properties": {
"device": { "type": "text" },
"measure": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
#headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
#r = requests.post(url, data=json.dumps(data), headers=headers)
r= es.index(index=name, doc_type='trends', body=doc)
print(r)

You need to send a HTTP Post request using python request, as follows:
url = "http://localhost:9200/index_name?pretty”
data = {
"mappings": {
"doc": {
"properties": {
"title": { "type": "text" },
"name": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(data), headers=headers)
Please replace index_name in the URL with the name of the index you are defining in to elasticsearch engine.
If you want to delete the index before creating it again, please do as follows:
url = "http://localhost:9200/index_name”
data = { }
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.delete(url, data=json.dumps(data), headers=headers)
please replace index_name in the URL with your actual index name. After deleting the index, create it again with the first code example above including the mappings that you would need. Enjoy.

Elasticsearch defines field types in the index mapping. It looks like you probably have dynamic mapping enabled, so when you send data to Elasticsearch for the first time, it makes an educated guess about the shape of your data and the field types.
Once those types are set, they are fixed for that index, and Elasticsearch will continue to interpret your data according to those types no matter what you do in your python script.
To fix this you need to either:
Define the index mapping before you load any data. This is the better option as it gives you complete control over how your data is interpreted. https://www.elastic.co/guide/en/elasticsearch/reference/6.0/mapping.html
Make sure that, the first time you send data into the index, you use the correct data types. This will rely dynamic mapping generation, but it will typically do the right thing.
Defining the index mapping is the best option. It's common to do that once off, in Kibana or with curl, or if you create a lot of indices, with a template.
However if you want to use python, you should look at the create or put_mapping functions on IndicesClient

How do I pass raw POST data into urllib3?

Trying to use urllib3 to post JSON-encoded data.
Just want my POST payload to be raw JSON string, with content type application/json.
I just cannot see how to do this.
The urllib3 documentation describes posting data in "fields", i.e. dicts with (key,value) pairs, like how HTML forms are URL-encoded with the URL. But I don't want to do that.
The closest I've been able to get is this (I just guessed where to put the data, as it's not documented anywhere that I can find):
http = urllib3.PoolManager()
headers = urllib3.util.make_headers(basic_auth=key+":")
r = http.request_encode_body('POST', path, json.dumps(payload), headers=headers)
which causes this urllib3 error:
File "C:\Python27\lib\site-packages\urllib3-1.7.1-py2.7.egg\urllib3\filepost.py", line 44, in iter_field_objects
yield RequestField.from_tuples(*field)
TypeError: from_tuples() takes exactly 3 arguments (2 given)
Thanks for any pointers!

you can't use PoolManager.request for that, it tries to concoct the body iself, use the lower level urlopen:
In [16]: pool = urllib3.PoolManager()
In [17]: print pool.urlopen('POST', 'http://httpbin.org/post', headers={'Content-Type':'application/json'}, body='{"sup":"son"}').data
{
"data": "{\"sup\":\"son\"}",
"form": {},
"json": {
"sup": "son"
},
"origin": "50.74.23.243",
"args": {},
"url": "http://httpbin.org/post",
"files": {},
"headers": {
"Host": "httpbin.org",
"Content-Length": "13",
"Content-Type": "application/json",
"Accept-Encoding": "identity",
"Connection": "close"
}
}