We Keep Coding

How do i get the recent inserted document in MongoDB with all it's fields?

I'm working on this REST application in python Flask and a driver called pymongo. But if someone knows mongodb well he/she maybe able to answer my question.
Suppose Im inserting a new document in a collection say students. I want to get the whole inserted document as soon as the document is saved in the collection. Here is what i've tried so far.
res = db.students.insert_one({
"name": args["name"],
"surname": args["surname"],
"student_number": args["student_number"],
"course": args["course"],
"mark": args["mark"]
})
If i call:
print(res.inserted_id) ## i get the id
How can i get something like:
{
"name": "student1",
"surname": "surname1",
"mark": 78,
"course": "ML",
"student_number": 2
}
from the res object. Because if i print res i am getting <pymongo.results.InsertOneResult object at 0x00000203F96DCA80>

Put the data to be inserted into a dictionary variable; on insert, the variable will have the _id added by pymongo.
from pymongo import MongoClient
db = MongoClient()['mydatabase']
doc = {
"name": "name"
}
db.students.insert_one(doc)
print(doc)
prints:
{'name': 'name', '_id': ObjectId('60ce419c205a661d9f80ba23')}

Unfortunately, the commenters are correct. The PyMongo pattern doesn't specifically allow for what you are asking. You are expected to just use the inserted_id from the result and if you needed to get the full object from the collection later do a regular query operation afterwards

Python: Get Jira Tickets that returned an error

I'm using the atlassian rest API and creating issues through it. For accessing the API I'm using the JIRA-API-Wrapper see: https://pypi.org/project/jira/.
In my application I'm uploading a bunch of tickets. Due to performance reasons I'm using concurrent.futures. The tickets are uploaded via the following code:
fields = [{'project': 'Project', 'summary': 'New summary', 'issuetype': {'name': 'Task'}}, ....]
with concurrent.futures.ThreadPoolExecutor() as executor:
data = executor.map(jira.create_issue, fields)
My problem is, that I'm not really sure how to get the information, when a ticket couldn't be uploaded for some reason. Everytime a ticket couldn't be uploaded, the JIRA-Wrapper returns a JIRAError-Exception. Therefore, I somehow have to count whenever I get a JIRAError. But unfortunally I'm not sure how to count the errors.
I know that the result can be retrieved via:
for i in data:
counter = counter + 1
print(i)
But because data contains the JIRAErrors the above code fails. This is why I tried the following.
try:
for i in data:
print(i)
except:
print(fields[counter])
But when the exception appears the code just continues. Therefore I tried solutions with a while-loop, but they also didn't get the right solution.
Is there a way to get the tickets, which couldn't be uploaded?

I haven't used jira-python myself. I wrote my own python client that I've been using for years. I'll have to give this a try myself.
According to the documentation to create bulk issues:
https://jira.readthedocs.io/en/latest/examples.html#issues
issue_list = [
{
'project': {'id': 123},
'summary': 'First issue of many',
'description': 'Look into this one',
'issuetype': {'name': 'Bug'},
},
{
'project': {'key': 'FOO'},
'summary': 'Second issue',
'description': 'Another one',
'issuetype': {'name': 'Bug'},
},
{
'project': {'name': 'Bar'},
'summary': 'Last issue',
'description': 'Final issue of batch.',
'issuetype': {'name': 'Bug'},
}]
issues = jira.create_issues(field_list=issue_list)
Additionally, there is a note about the failures which you are interested in:
Using bulk create will not throw an exception for a failed issue
creation. It will return a list of dicts that each contain a possible
error signature if that issue had invalid fields. Successfully created
issues will contain the issue object as a value of the issue key.
So to see the ones that failed, you would iterate through issues and look for error signatures.
As far as performance issues, you could look at jira.create_issues(data, prefectch=false)
https://jira.readthedocs.io/en/latest/api.html#jira.JIRA.create_issues
prefetch (bool) – whether to reload the created issue Resource for
each created issue so that all of its data is present in the value
returned from this method.
However, if you must use concurrent.futures, note that it will likely fail when calling via jira.create_issue if jira object has a state that needs to be saved between calls to create_issue when being run async.
If a func call raises an exception, then that exception will be raised
when its value is retrieved from the iterator.
I would recommend using a separate jira object for each create_issue if you do not trust the create_issues() function.
def create_issue(fields):
print(fields)
j_obj = JIRA(...)
try:
ret = j_obj.create_issue(fields)
except:
# Do something here
ret = False
return ret
with concurrent.futures.ThreadPoolExecutor() as executor:
data = executor.map(create_issue, issue_list)
items = [item for item in data]
print(items)
# Interact with result
pdb.set_trace()
When you break into the trace, any successful issues created will be an Issue type, any failures will show up as False. This is just an example, and you can decide what you want to return, in whatever format you need.

Can't reach nested JSON in python 3.8

Working with a response from a Websockets subscription.
The response reads like this:
{'jsonrpc': '2.0', 'method': 'subscription', 'params': {'channel': 'book.BTC-PERPETUAL.none.1.100ms', 'data': {'timestamp': 1588975154127, 'instrument_name': 'BTC-PERPETUAL', 'change_id': 19078703948, 'bids': [[10019.5, 8530.0]], 'asks': [[10020.0, 506290.0]]}}}
And I'm trying to reach the first and only values inside "bids" and "asks" arrays via json.loads()
Code looks like this:
async def __async__get_ticks(self):
async with self.ws as echo:
await echo.send(json.dumps(self.request))
while True:
response = await echo.receive()
responseJson = json.loads(response)
print(responseJson["params"]["data"])
And error says:
print(responseJson["params"]["data"])
KeyError: 'params'
However I try, it doesn't want to catch any of the JSON after "jsonprc", for which it successfully returns 2.0. Anything beyond that always comes up with an error.
I tried using .get(), and it helps to go one level deeper, but still not more.
Any ideas on how to format this properly and reach the bids and asks ?
Thank you in advance.

I would suggest using the dict.get() method, but make sure that you set it to return an empty dictionary when querying dictionaries that are expected to have nested dicts.
By default (if you don't specify a second argument to dict.get()), it will return None. This explains why you were only able to go one level deep.
Here's an example:
empty_dict = {}
two_level_dict = {
"one": {
"level": "deeper!"
}
}
# This will return None and the second get call will not fail, because
# the first get returned an empty dict for the .get("level") call to succeed.
first_get = empty_dict.get("one", {}).get("level")
# This will return 'deeper!'
second_get = two_level_dict.get("one", {}).get("level")
print(first_get)
print(second_get)

Python extract json structure from html page

in python i'm reading an html page content which contains a lot of stuff.
To do this i read the webpage as string by this way:
url = 'https://myurl.com/'
reqq = req.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
reddit_file = req.urlopen(reqq)
reddit_data = reddit_file.read().decode('utf-8')
if i print the reddit_data i can see correctly the whole html contents.
Now, inside it there's a structure like json that i would like to read and extract some fields from that.
Below the structure:
"dealDetails" : {
"f240141a" : {
"egressUrl" : "https://ccc.com",
"title" : "ZZZ",
"type" : "ghi",
},
"5f9ab246" : {
"egressUrl" : "https://www.bbb.com/",
"title" : "YYY",
"type" : "def",
},
"2bf6723b" : {
"egressUrl" : "https://www.aaa.com//",
"title" : "XXX",
"type" : "abc",
},
}
What i want to do is: find the dealDetails field and then for each f240141a 5f9ab246 2bf6723b
get the egressURL, title and type values.
Thanks

Try this,
[nested_dict['egressUrl'] for nested_dict in reddit_data['dealDetails'].keys()]
To access the values of JSON, you can consider as dictionary and use the same syntax to access values as well.
Edit-1:
Make sure your type of reddit_data is a dictionary.
if type(reddit_data) is str.
You need to do..
import ast
reddit_data = ast.literal_eval(reddit_data)
OR
import json
reddit_data = json.loads(reddit_data)

If you just wanted to know how to access the egressURL, title and the type. You might just wanna read the answer below! Be careful however, cause the following code won't work unless you converted your HTML file reddit_data in something like a dictionary ( Modified shaik moeed's answer a tiny bit to also return title and type) :
[(i['egressUrl'], i['title'], i['type']) for i in reddit_data['dealDetails'].keys()]
However, If I got it right, the part you're missing is the conversion from HTML to a JSON friendly file. What I personally use, even though it's quite unpopular, is the eval function
dictionary = eval(reddit_data)
This will convert the whole file into a dictionary, I recommend that you only use it on the part of the text that 'looks' like a dictionary! (One of the reason eval is unpopular, is because it won't convert strings like 'true'/'false' to Python's True/False, be careful with that :) )
Hope that helped!

how to get Parameters from a json file with python

I am working in Python with a JSON file. When I print date it comes out like this :
print (date['slot'])
[{'slot': 1, 'type': {'url': 'https://pokeapi.co/api/v2/type/11/', 'name':
'water'}}]
But I just want to get 'water'. I have been using different methods and it did not work. Can someone please help me to get just 'water' from the code?

print (date['slot'][0]['type']['name'])
OR
print (date['slot'][0]['type']['name'] if date['slot'] else None)
OR More generic
for sd in date['slot']:
print (sd['type']['name'])