We Keep Coding

Python passing variable with multiple lines to payload

Below is the part of code I am trying to execute.
msg="""line1
line 2"""
msg=f"\"{msg}\""
payload="{\"Additional comments\" :"+msg+",\n \"Short description\" : \"tests\",\n \"Requester\" : \"test1\",\n \"Impact\" : \"3\",\n \"Contact type\" : \"test1 \"\n\n}\n"
My code throws this error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
But when I send a message as:
#without a new line \n, my code works fine
msg="Line one" # code works fine

Put your data in a Python dictionary and use the json module to generate the text. It will write valid JSON much more easily that trying to format it manually:
import json
msg = '''line1
line2'''
data = {'Additional comments' : msg,
'Short description' : 'tests',
'Requester' : 'test1',
'Impact' : '3',
'Contact type' : 'test1'}
text = json.dumps(data,indent=2)
print(text)
Output:
{
"Additional comments": "line1\nline2",
"Short description": "tests",
"Requester": "test1",
"Impact": "3",
"Contact type": "test1"
}

Parse a file with JSON objects in Python

I have a file with this type of structure:
{
"key" : "A",
"description" : "1",
"uninterestingInformation" : "whatever"
}
{
"key" : "B",
"description" : "2",
"uninterestingInformation" : "whatever"
}
{
"key" : "C",
"description" : "3",
"uninterestingInformation" : "whatever"
}
I want to build a dictionary in Python that contains the key as key and the description as value. I have more fields, but just the 2 of them are interesting for me.
This file is not exactly a .json file, is a file with a lot of similar json objects.
json.loads is not working, obviously.
Any suggestion on how to read the data?
I've already read this post, but my json object is not on one line...
EDIT:
If it wasn't clear in my explanations, the example is quite accurate, I have a lot of similar JSON objects, one after another, separated by new line (\n), with no comma. So, overall the file is not a valid JSON file, while each object is a valid JSON object.
The solution I've applied finally was:
api_key_file = open('mongo-config.json').read()
api_key_file = '[' + api_key_file + ']'
api_key_file= api_key_file.replace("}\n{", "},\n{")
api_key_data = json.loads(api_key_file)
api_key_description = {}
for data in api_key_data:
api_key_description[data['apiKey']] = data['description']
It worked well for my situation. There are maybe better ways of doing this explained in the comments bellow.

Another option would be to use the literal_eval function from the ast module, after making the necessary changes so that it fits the format of a valid type:
from ast import literal_eval
inJson = '''{
"key" : "A"
"description" : "1"
"uninterestingInformation" : "whatever"
}
{
"key" : "B"
"description" : "2"
"uninterestingInformation" : "whatever"
}
{
"key" : "C"
"description" : "3"
"uninterestingInformation" : "whatever"
}'''
inJson = "[" + inJson.replace("}", "},")[:-1] + "]"
inJson = inJson.replace("\"\n ","\",")
newObject = literal_eval(inJson)
print(newObject)
Output:
[{'key': 'A', 'description': '1', 'uninterestingInformation': 'whatever'}, {'key': 'B', 'description': '2', 'uninterestingInformation': 'whatever'}, {'key': 'C', 'description': '3', 'uninterestingInformation': 'whatever'}]

You can use re.split to split the file content into appropriate JSON strings for parsing:
import re
import json
j='''{
"key" : "A",
"description" : "1",
"uninterestingInformation" : "whatever"
}
{
"key" : "B",
"description" : "2",
"uninterestingInformation" : "whatever"
}
{
"key" : "C",
"description" : "3",
"uninterestingInformation" : "whatever"
}'''
print(list(map(json.loads, re.split(r'(?<=})\n(?={)', j))))
This outputs:
[{'key': 'A', 'description': '1', 'uninterestingInformation': 'whatever'}, {'key': 'B', 'description': '2', 'uninterestingInformation': 'whatever'}, {'key': 'C', 'description': '3', 'uninterestingInformation': 'whatever'}]

How to use find() nested documents for two levels or more?

Here is my sample mongodb database
database image for one object
The above is a database with an array of articles. I fetched only one object for simplicity purposes.
database image for multiple objects ( max 20 as it's the size limit )
I have about 18k such entries.
I have to extract the description and title tags present inside the (articles and 0) subsections.
The find() method is the question here.. i have tried this :
for i in db.ncollec.find({'status':"ok"}, { 'articles.0.title' : 1 , 'articles.0.description' : 1}):
for j in i:
save.write(j)
After executing the code, the file save has this :
_id
articles
_id
articles
and it goes on and on..
Any help on how to print what i stated above?
My entire code for reference :
import json
import newsapi
from newsapi import NewsApiClient
import pymongo
from pymongo import MongoClient
client = MongoClient()
db = client.dbasenews
ncollec = db.ncollec
newsapi = NewsApiClient(api_key='**********')
source = open('TextsExtractedTemp.txt', 'r')
destination = open('NewsExtracteddict.txt', "w")
for word in source:
if word == '\n':
continue
all_articles = newsapi.get_everything(q=word, language='en', page_size=1)
print(all_articles)
json.dump(all_articles, destination)
destination.write("\n")
try:
ncollec.insert(all_articles)
except:
pass

Okay, so I checked a little to update my rusty memory of pymongo, and here is what I found.
The correct query should be :
db.ncollec.find({ 'status':"ok",
'articles.title' : { '$exists' : 'True' },
'articles.description' : { '$exists' : 'True' } })
Now, if you do this :
query = { 'status' : "ok",
'articles.title' : { '$exists' : 'True' },
'articles.description' : { '$exists' : 'True' } }
for item in db.ncollect.find(query):
print item
And that it doesn't show anything, the query is correct, but you don't have the right database, or the right tree, or whatever.
But I assure you, that with the database you showed me, that if you do...
query = { 'status' : "ok",
'articles.title' : { '$exists' : 'True' },
'articles.description' : { '$exists' : 'True' } }
for item in db.ncollect.find(query):
save.write(item[0]['title'])
save.write(item[0]['description'])
It'll do what you wished to do in the first place.
Now, the key item[0] might not be good, but for this, I can't really be of any help since it is was you are showing on the screen. :)
Okay, now. I have found something for you that is a bit more complicated, but is cool :)
But I'm not sure if it'll work for you. I suspect you're giving us a wrong tree, since when you do .find( {'status' : 'ok'} ), it doesn't return anything, and it should return all the documents with a 'status' : 'ok', and since you have lots...
Anyways, here is the query, that you should use with .aggregate() method, instead of .find() :
elem = { '$match' : { 'status' : 'ok', 'articles.title' : { '$exists' : 'True'}, 'articles.description' : { '$exists' : 'True'}} }
[ elem, { '$unwind' : '$articles' }, elem ]
If you want an explanation as to how this works, I invite you to read this page.
This query will return ONLY the elements in your array that have a title, and a description, with a status OK. If an element doesn't have a title, or a description, it will be ignored.

Issue with Parsing a Json Array

I am trying to Parse a Json array, A sample of the array i get is below with my code.
I can not seem to workout what my issue is, please forgive my question if I have included too much
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import json
map = "[{'network' : 'networkA','ycoord' : '73','zcoord' : '-2612','xcoord' : '-4461','owner' : 'PlayerA','name' : 'PlaceA'}, {'network' : 'NetworkB','ycoord' : '66','zcoord' : '-1915','xcoord' : '1156','owner' : 'PlayerB','name' : 'PlaceB'}, {'network' : 'NetWorkB','ycoord' : '71','zcoord' : '3091','xcoord' : '4541','owner' : 'PlayerB','name' : 'PlaceC'}, {'network' : 'NetworkB','ycoord' : '118','zcoord' : '-66','xcoord' : '5','owner' : 'PlayerB','name' : 'PlaceD'}, {'network' : 'networkA','ycoord' : '71','zcoord' : '761','xcoord' : '-248','owner' : 'PlayerA','name' : 'PlaceE'}]"
data = json.load(map)
for item in data:
print "Network : "+ str(item['network'])
print "Name : "+ str(item['name'])
print "Owner : "+ str(item['owner'])
print "Co ords : ("+ str(item['ycoord']+", "+ str(item['xcoord']+", "+ str(item['Zcoord']+")"
I get The error
File "test.py", line 8, in <module>
data = json.load(map)
File "/usr/lib/python2.7/json/__init__.py", line 274, in load
return loads(fp.read(),
AttributeError: 'str' object has no attribute 'read'
Readable Json Data (because I know what's in the code isn't)
[{
'network' : 'networkA',
'ycoord' : '73',
'zcoord' : '-2612',
'xcoord' : '-4461',
'owner' : 'PlayerA',
'name' : 'PlaceA'
}, {
'network' : 'NetworkB',
'ycoord' : '66',
'zcoord' : '-1915',
'xcoord' : '1156',
'owner' : 'PlayerB',
'name' : 'PlaceB'
}, {
'network' : 'NetWorkB',
'ycoord' : '71',
'zcoord' : '3091',
'xcoord' : '4541',
'owner' : 'PlayerB',
'name' : 'PlaceC'
}, {
'network' : 'NetworkB',
'ycoord' : '118',
'zcoord' : '-66',
'xcoord' : '5',
'owner' : 'PlayerB',
'name' : 'PlaceD'
}, {
'network' : 'networkA',
'ycoord' : '71',
'zcoord' : '761',
'xcoord' : '-248',
'owner' : 'PlayerA',
'name' : 'PlaceE'
}]

You want loads() instead of load(). Read the documentation, load() takes a filename, loads() takes actual JSON data.

json.load() function will require filename as a parameter. In your case, You don't want filename but an actual JSON array.
Use json.loads() instead of json.load()
Also, remember The functions with an s take string parameters. The others take file
streams. This applies to json.dump() and json.dumps() too.

How to use "suggest" in elasticsearch pyes?

How to use the "suggest" feature in pyes? Cannot seem to figure it out due to poor documentation. Could someone provide a working example? None of what I tried appears to work. In the docs its listed under query, but using:
query = Suggest(fields="fieldname")
connectionobject.search(query=query)

Since version 5:
_suggest endpoint has been deprecated in favour of using suggest via _search endpoint. In 5.0, the _search endpoint has been optimized for suggest only search requests.
(from https://www.elastic.co/guide/en/elasticsearch/reference/5.5/search-suggesters.html)
Better way to do this is using search api with suggest option
from elasticsearch import Elasticsearch
es = Elasticsearch()
text = 'ra'
suggest_dictionary = {"my-entity-suggest" : {
'text' : text,
"completion" : {
"field" : "suggest"
}
}
}
query_dictionary = {'suggest' : suggest_dictionary}
res = es.search(
index='auto_sugg',
doc_type='entity',
body=query_dictionary)
print(res)
Make sure you have indexed each document with suggest field
sample_entity= {
'id' : 'test123',
'name': 'Ramtin Seraj',
'title' : 'XYZ',
"suggest" : {
"input": [ 'Ramtin', 'Seraj', 'XYZ'],
"output": "Ramtin Seraj",
"weight" : 34 # a prior weight
}
}

Here is my code which runs perfectly.
from elasticsearch import Elasticsearch
es = Elasticsearch()
text = 'ra'
suggDoc = {
"entity-suggest" : {
'text' : text,
"completion" : {
"field" : "suggest"
}
}
}
res = es.suggest(body=suggDoc, index="auto_sugg", params=None)
print(res)
I used the same client mentioned on the elasticsearch site here
I indexed the data in the elasticsearch index by using completion suggester from here