We Keep Coding

Python: Add subelement to an element based on value of a sibling subelement?

Lots of python XML parsing tutorials out there, but not that many on updating XML, and none I can find that match my needs. Sorry for the N00B.
I have a need to add subelements to a particular element based on the value of another subelement.
<CadData>
<FireIncidentCollection>
<FireIncident>
<IncidentNo>12345</IncidentNo>
<ApparatusCollection>
<Apparatus>
<Unit>E29</Unit>
</Apparatus>
<Apparatus>
<Unit>TW29</Unit>
</Apparatus>
<Apparatus>
<Unit>R29</Unit>
</Apparatus>
</ApparatusCollection>
</FireIncident>
</FireIncidentCollection>
</CadData>
I have values and even other subtrees I need to add based on the "Unit" value of an "Apparatus" element. For example, I may need to add this snippet in that "Apparatus" element when the "Unit"=="TW29":
<DispatchTime>20221115T06:05:04-6.00</DispatchTime>
<ApparatusPersonnelCollection>
<ApparatusPersonnel>
<ID>23456</ID>
</ApparatusPersonnel>
<ApparatusPersonnel>
<ID>78901</ID>
</ApparatusPersonnelCollection>
So far I'm resisting the urge to dump everything to a DB and re-writing the whole file each time :). I'm sure there's a way in ElementTree or DOM, but I can't figure it out (not for lack of effort). Any pointers are greatly appreciated. Thanks!
(Oh, and no, I don't control the schema- I just have to adhere to it).

The first step would be to put this information into a dictionary - then it will be much easier to update your data
I'd recommend using xmltodict library with a mixture of this tutorial - which will allow you to convert to a dictionary that you can traverse.
From there, just traverse down the dictionary. The nice thing about the xmltodict library is that it will convert xml with the same tag into a list, so your ApparatusCollection might look something like this once it's converted:
>>> data['CadData']['FireIncidentCollection']['FireIncident']['ApparatusCollection']
{'Apparatus': [{'Unit': 'E29'}, {'Unit': 'TW29'}, {'Unit': 'R29'}]}
Just find the "ApparatusCollect" key, and update the "Apparatus" section. My guess is that it would look close to this:
{'Apparatus': [
{
'Unit': 'E29',
'DispatchTime':20221115T06:05:04-6.00},
'ApparatusPersonnelCollection': {
'ApparatusPersonnel': [23456, 78901]
} ...
After you've added what you need, just convert the dictionary back into XML.
Hope this helps!

Zipline-trader: Unknown syntax [{'sid': Equity(1576 [JPM])} -

I am relatively new to Python and am working my way through the zipline-trader library. I came across a data structure that I am unfamiliar with and was wondering if you could help me access a certain element out of it.
I ran a backtest on zipline-trader and have the results-DataFrame that has a column "positions" which includes the portfolio positions for a given day.
Here is an example of the content of that column:
[{'sid': Equity(1576 [JPM]), 'amount': 39, 'cost_basis': 25.95397, 'last_sale_price': 25.94}, {'sid': Equity(2942 [UNH]), 'amount': 11, 'cost_basis': 86.62428999999999, 'last_sale_price': 86.58}]
The syntax I am unfamiliar with is the part "Equity (1576 [JPM])" - can anybody explain to me what this is? Also, can you please let me know how to access the "[JPM]"-part of it? Ultimately, what I am trying to do is access that cell of the DataFrame using a loc-function and producing the result "{JPM: 1576, UNH: 2942}"
Thank you!

That is (likely to be) an object of type Equity. If the structure you showed us was stored in a variable data then the object can be fetched using
eq = data[0]['sid']
The text when it's printed will be coming from the __str__ method defined in the Equity class, so it doesn't really tell us anything about how to access it. You would have to look up the documentation.
If you are able to access the object in an interactive session then you could run the help command against it and that might contain something useful. Again, if the structure you showed us was stored in a variable data then you could do:
help(data[0]['sid'])

transform JSON file to be usable

Long story short, i get the query from spotify api which is JSON that has data about newest albums. How do i get the specific info from that like let's say every band name or every album title. I've tried a lot of ways to get that info that i found on the internet and nothing seems to work for me and after couple of hours im kinda frustrated
JSON data is on jsfiddle
here is the request
endpoint = "https://api.spotify.com/v1/browse/new-releases"
lookup_url = f"{endpoint}"
r = requests.get(lookup_url, headers=headers)
print(r.json())
you can find the

When you make this request like the comments have mentioned you get a dictionary which you can then access the keys and values. For example if you want to get the album_type you could do the following:
print(data["albums"]["items"][0]["album_type"])
Since items contains a list you would need to get the first values 0 and then access the album_type.
Output:
single
Here is a link to the code I used with your json.
I suggest you look into how to deal with json data in python, this is a good place to start.

I copied the data from the jsfiddle link.
Now try the following code:
import ast
pyobj=ast.literal_eval(str_cop_from_src)
later you can try with keys
pyobj["albums"]["items"][0]["album_type"]
pyobj will be a python dictionary will all data.

Reading a dictionary from within a dictionary

I have a json file for tweet data. The data that I want to look at is the text of the tweet. For some reason, some of the tweets are too long to put into the normal text part of the dictionary.
It seems like there is a dictionary within another dictionary and I can't figure out how to access it very well.
Basically, what I want in the end is one column of a data frame that will have all of the text from each individual tweet. Here is a link to a small sample of the data that contains a problem tweet.
Here is the code I have so far:
import json
import pandas as pd
tweets = []
#This writes the json file so that I can work with it. This part works correctly.
with open("filelocation.txt") as source
for line in source:
if line.strip():
tweets.append(json.loads(line))
print(len(tweets)
df = pd.DataFrame.from_dict(tweets)
df.info()
When looking at the info you can see that there will be a column called extended_tweet that only encompasses one of the two sample tweets. Within this column, there seems to be another dictionary with one of those keys being full_text.
I want to add another column to the dataframe that just has this information along with the normal text column when the full_text is null.
My first thought was to try and read that specific column of the dataframe as a dictionary again using:
d = pd.DataFrame.from_dict(tweets['extended_tweet]['full_text])
But this doesn't work. I don't really understand why that doesn't work as that is how I read the data the first time.
My guess is that I can't look at the specific names because I am going back to the list and it would have to read all or none. The error it gives me says "KeyError: 'full_text' "
I also tried using the recommendation provided by this website. But this gave me a None value no matter what.
Thanks in advance!
I tried to do what #Dan D. suggested, however, this still gave me errors. But it gave me the idea to try this:
tweet[0]['extended_tweet']['full_text']
This works and gives me the value that I am looking for. But I need to run through the whole thing. So I tried this:
df['full'] = [tweet[i]['extended_tweet']['full_text'] for i in range(len(tweet))
This gives me "Key Error: 'extended_tweet' "
Does it seem like I am on the right track?

I would suggest to flatten out the dictionaries like this:
tweet = json.loads(line)
tweet['full_text'] = tweet['extended_tweet']['full_text']
tweets.append(tweet)

I don't know if the answer suggested earlier works. I never got that successfully. But I did figure out something else that works well for me.
What I really needed was a way to display the full text of a tweet. I first loaded the tweets from the json with what I posted above. Then I noticed that in the data file, there is something called truncated. If this value is true, the tweet is cut short and the full tweet is placed within the
tweet[i]['extended_tweet]['full_text]
In order to access it, I used this:
tweet_list = []
for i in range(len(tweets)):
if tweets[i]['truncated'] == 'True':
tweet_list.append(tweets[i]['extended_tweet']['full_text']
else:
tweet_list.append(tweets[i]['text']
Then I can work with the data using the whol text from each tweet.

one-many relationship-google datastore-python

I have two models like below:-
class Food(db.Model):
foodname=db.StringProperty()
cook=db.StringProperty()
class FoodReview(db.Model):
thereview=db.StringProperty()
reviews=db.ReferenceProperty(Food,collections_name='thefoodreviews')
I go ahead and create an entity:-
s=Food(foodname='apple',cook='Alice')`
s.put()
When someone writes a review, the function which does the below comes in play:
theentitykey=db.Query(Food,keys_only=True).filter('foodname =','apple').get()
r=FoodReview()
r.reviews=theentitykey #this is the key of the entity retrieved above and stored as a ref property here
r.thereview='someones review' #someone writes a review
r.put()
Now the problem is how to retrieve these reviews. If I know the key of the entity, I can just do the below:-
theentityobject=db.get(food-key) # but then the issue is how to know the key
for elem in theentityobject.thefoodreviews:
print elem.thereview
else I can do something like this:-
theentityobj=db.Query(Food).filter('foodname =','apple').get()
and then iterate as above, but are the above two ways the correct ones?

If to get the food you're always doing db.Query(Food).filter('foodname =','apple') then it looks like your foodname is your key...
Why not just use it as a key_name?
Then, you can even fetch the reviews without fetching the food itself:
key = db.Key.from_path('food', 'apple')
reviews = FoodReview.all().filter("reviews =", key)

The second method looks exactly like what AppEngine tutorial advices.
Seems like the right thing to do, if you want to find all reviews for a particular foodname.