We Keep Coding

ElementTree does not seem to get some texts/elements in the tree

I have a wikipedia dump that I want to parse, and I am having some difficulties/mysterious problems while using Python xml parser, ElementTree.
My recent problem is, ElementTree does not seem to find the texts that are actually there. This is an example data:
<page>
<title>Cengiz Han</title>
<ns>0</ns>
<id>10</id>
<revision>
<id>20337884</id>
<parentid>20218916</parentid>
<timestamp>2019-01-29T14:02:43Z</timestamp>
<contributor>
<username>CommonsDelinker</username>
<id>31545</id>
</contributor>
<comment>China_11b.jpg dosyası Map_of_China_1142.jpg ile değiştirildi</comment>
<model>wikitext</model>
<format>text/x-wiki</format>
<text xml:space="preserve">
...some long Genghis Khan stuff...
</text>
</page>
Now when I parse it with this:
for event, elem in et.iterparse('dataset/wiki_test', events=('start', 'end', 'start-ns', 'end-ns')):
if event == 'start':
if elem.tag == 'page':
if len(list(elem)) == 0:
continue
title = elem.find('title').text
if title == None or 'MediaWiki' in title:
elem.clear()
continue
wiki_id = elem.find('id')
if wiki_id == None:
elem.clear()
continue
wiki_id = wiki_id.text
revision = elem.find('revision')
if revision != None:
print(list(revision))
text = revision.find('text').text
print(text)
if text != None:
count += 1
titles += title + '\n'
page = {'wiki_id': wiki_id, 'title': title, 'text': text.text}
pages += json.dumps(page, ensure_ascii=False) + '\n'
elem.clear()
revision.find('text').text line seems to find no text for some elements, including the one above, and that some makes like one seventh of my data, which is annoying. This was also the case for page->id for some other entries, in which it claimed that element does not exist at all. I solved that problem by ignoring that ones, but I don't really want to do that, also this error does not make sense to me at all.
Here is another page, which works totally fine.
<page>
<title>Mustafa Suphi</title>
<ns>0</ns>
<id>22</id>
<revision>
<id>20077185</id>
<parentid>20017115</parentid>
<timestamp>2018-10-14T08:31:32Z</timestamp>
<contributor>
<username>Vikiçizer</username>
<id>90501</id>
</contributor>
<comment>/* top */düzeltme [[Vikipedi:AWB|AWB]] ile</comment>
<model>wikitext</model>
<format>text/x-wiki</format>
<text xml:space="preserve">
...some Mustafa Suphi stuff...
</text>
<sha1>m5finh6h2kr8h2fbtmsatp5fhz1siwq</sha1>
</revision>
</page>
What am I doing wrong?

You have posted two examples: "working" and "not working".
In the "not working" one there is no
</revision>
Are you sure this the XML you have or it is just copy & paste mistake.

Print nested element of xml with python etree

I'm trying to build a script to read an xml file.
This is my first time parsing an xml and i'm doing it using python with xml.etree.ElementTree. The section of the file that i would like to process looks like:
<component>
<section>
<id root="42CB916B-BB58-44A0-B8D2-89B4B27F04DF" />
<code code="34089-3" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC" displayName="DESCRIPTION SECTION" />
<title mediaType="text/x-hl7-title+xml">DESCRIPTION</title>
<text>
<paragraph>Renese<sup>®</sup> is designated generically as polythiazide, and chemically as 2<content styleCode="italics">H</content>-1,2,4-Benzothiadiazine-7-sulfonamide, 6-chloro-3,4-dihydro-2-methyl-3-[[(2,2,2-trifluoroethyl)thio]methyl]-, 1,1-dioxide. It is a white crystalline substance, insoluble in water but readily soluble in alkaline solution.</paragraph>
<paragraph>Inert Ingredients: dibasic calcium phosphate; lactose; magnesium stearate; polyethylene glycol; sodium lauryl sulfate; starch; vanillin. The 2 mg tablets also contain: Yellow 6; Yellow 10.</paragraph>
</text>
<effectiveTime value="20051214" />
</section>
</component>
<component>
<section>
<id root="CF5D392D-F637-417C-810A-7F0B3773264F" />
<code code="42229-5" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC" displayName="SPL UNCLASSIFIED SECTION" />
<title mediaType="text/x-hl7-title+xml">ACTION</title>
<text>
<paragraph>The mechanism of action results in an interference with the renal tubular mechanism of electrolyte reabsorption. At maximal therapeutic dosage all thiazides are approximately equal in their diuretic potency. The mechanism whereby thiazides function in the control of hypertension is unknown.</paragraph>
</text>
<effectiveTime value="20051214" />
</section>
</component>
The full file can be downloaded from:
https://dailymed.nlm.nih.gov/dailymed/getFile.cfm?setid=abd6ecf0-dc8e-41de-89f2-1e36ed9d6535&type=zip&name=Renese
Here my code:
import xml.etree.ElementTree as ElementTree
import re
with open("ABD6ECF0-DC8E-41DE-89F2-1E36ED9D6535.xml") as f:
xmlstring = f.read()
# Remove the default namespace definition (xmlns="http://some/namespace")
xmlstring = re.sub(r'\sxmlns="[^"]+"', '', xmlstring, count=1)
tree = ElementTree.fromstring(xmlstring)
for title in tree.iter('title'):
print(title.text)
So far I'm able to print the titles but I would like to print also the corresponding text that is captured in the tags.
I have tried this:
for title in tree.iter('title'):
print(title.text)
for paragraph in title.iter('paragraph'):
print(paragraph.text)
But I have no output from the paragraph.text
Doing
for title in tree.iter('title'):
print(title.text)
for paragraph in tree.iter('paragraph'):
print(paragraph.text)
I print the text of the paragraphs but (obviously) it is printed all together for each title found in the xml structure.
I would like to find a way to 1) identify the title; 2) print the corresponding paragraph(s).
How can I do it?

If you are willing to use lxml, then the following is a solution which uses XPath:
import re
from lxml.etree import fromstring
with open("ABD6ECF0-DC8E-41DE-89F2-1E36ED9D6535.xml") as f:
xmlstring = f.read()
xmlstring = re.sub(r'\sxmlns="[^"]+"', '', xmlstring, count=1)
doc = fromstring(xmlstring.encode()) # lxml only accepts bytes input, hence we encode
for title in doc.xpath('//title'): # for all title nodes
title_text = title.xpath('./text()') # get text value of the node
# get all text values of the paragraph nodes that appear lower (//paragraph)
# in the hierarchy than the parent (..) of <title>
paragraphs_for_title = title.xpath('..//paragraph/text()')
print(title_text[0] if title_text else '')
for paragraph in paragraphs_for_title:
print(paragraph)

Search xml for text and return element/node

I'd like to be able to search an xml formatted file by the text value and return the id it is part of. I've looked through the python library at the xml commands but only saw examples for searching by elements/nodes. I have a simplified xml sample below and I'd like search for "3x3 Eyes" for example and return "2". It should also search for the exact text minus case. There will normally be multiple entries for title under each anime so the search can stop at the first match. Thanks
<?xml version="1.0" encoding="UTF-8"?>
<animetitles>
<anime aid="1">
<title type="official" xml:lang="fr">Crest of the Stars</title>
<title type="official" xml:lang="fr">Crest of the Stars</title>
</anime>
<anime aid="2">
<title type="official" xml:lang="en">3x3 Eyes</title>
</anime>
<anime aid="3">
<title type="official" xml:lang="en">3x3 Eyes: Legend of the Divine Demon</title>
</anime>
</animetitles>

tree = et.parse( ... )
# Unique match
results = []
for anime in tree.findall('anime'):
for title in anime.findall('title'):
if title.text == '3x3 Eyes':
results.append(anime.get('aid'))
print results
# Everything that starts with
results = []
for anime in tree.findall('anime'):
for title in anime.findall('title'):
if title.text.startswith('3x3 Eyes'):
results.append(anime.get('aid'))
print results
First one returns [2], the second one [2, 3].
Or a little bit more cryptic but, hey, why not :)
results = [anime.get('aid') for anime in tree.findall('anime')
for title in anime.findall('title') if title.text == '3x3 Eyes']

You can use ElementTree for your purpose.
import xml.etree.ElementTree as ET
tree = ET.parse('a.xml')
root = tree.getroot()
def findParentAttrib(string):
for neighbor in root.iter():
for parent in neighbor.getiterator():
for child in parent:
if child.text == string:
return parent.attrib['aid']
print findParentAttrib("3x3 Eyes") # returns 2
Also refer to this page.

Getting subelements using lxml and iterparse

I am trying to write a parsing algorithm to efficiently pull data from an xml document. I am currently rolling through the document based on elements and children, but would like to use iterparse instead. One issue is that I have a list of elements that when found, I want to pull the child data from them, but it seems like using iterparse my options are to filter based on either one element name, or get every single element.
Example xml:
<?xml version="1.0" encoding="UTF-8"?>
<data_object xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<source id="0">
<name>Office Issues</name>
<datetime>2012-01-13T16:09:15</datetime>
<data_id>7</data_id>
</source>
<event id="125">
<date>2012-11-06</date>
<state_id>7</state_id>
</event>
<state id="7">
<name>Washington</name>
</state>
<locality id="2">
<name>Olympia</name>
<state_id>7</state_id>
<type>City</type>
</locality>
<locality id="3">
<name>Town</name>
<state_id>7</state_id>
<type>Town</type>
</locality>
</data_object>
Code example:
from lxml import etree
fname = "test.xml"
ELEMENT_LIST = ["source", "event", "state", "locality"]
with open(fname) as xml_doc:
context = etree.iterparse(xml_doc, events=("start", "end"))
context = iter(context)
event, root = context.next()
base = False
b_name = ""
for event, elem in context:
if event == "start" and elem.tag in ELEMENT_LIST:
base = True
bname = elem.tag
children = elem.getchildren()
child_list = []
for child in children:
child_list.append(child.tag)
print bname + ":" + str(child_list)
elif event == "end" and elem.tag in ELEMENT_LIST:
base = False
root.clear()

With iterparse you cannot limit parsing to some types of tags, you may do this only with one tag (by passing argument tag). However it is easy to do manually what you would like to achieve. In the following snippet:
from lxml import etree
fname = "test.xml"
ELEMENT_LIST = ["source", "event", "state", "locality"]
with open(fname) as xml_doc:
context = etree.iterparse(xml_doc, events=("start", "end"))
for event, elem in context:
if event == "start" and elem.tag in ELEMENT_LIST:
print "this elem is interesting, do some processing: %s: [%s]" % (elem.tag, ", ".join(child.tag for child in elem))
elem.clear()
you limit your search to interesting tags only. Important part of iterparse is the elem.clear() which clears memory when item is obsolete. That is why it is memory efficient, see http://lxml.de/parsing.html#modifying-the-tree

I would use XPath instead. It's much more elegant than walking the document on your own and certainly more efficient I assume.

Use tag='{http://www.sitemaps.org/schemas/sitemap/0.9}url'
Similar question with right answer https://stackoverflow.com/a/7019273/1346222
#!/usr/bin/python
# coding: utf-8
""" Parsing xml file. Basic example """
from StringIO import StringIO
from lxml import etree
import urllib2
sitemap = urllib2.urlopen(
'http://google.com/sitemap.xml',
timeout=10
).read()
NS = {
'x': 'http://www.sitemaps.org/schemas/sitemap/0.9',
'x2': 'http://www.google.com/schemas/sitemap-mobile/1.0'
}
res = []
urls = etree.iterparse(StringIO(sitemap), tag='{http://www.sitemaps.org/schemas/sitemap/0.9}url')
for event, url in urls:
t = []
t = url.xpath('.//x:loc/text() | .//x:priority/text()', namespaces=NS)
t.append(url.xpath('boolean(.//x2:mobile)', namespaces=NS))
res.append(t)

Parsing XML file etree module

I'm reading XML file using Etree module. Im using following code to print the value of <page> and <title> tags. My code working fine. But I want little change. If the <page id='...'> attribute id is exists then print the value of tag. Is it possible? thanks
import xml.etree.cElementTree as etree
from pprint import pprint
tree = etree.parse('find_title.xml')
for value in tree.getiterator(tag='title'):
print value.text
for value in tree.getiterator(tag='page'):
pprint(value.attrib)
Here is my xml File.
<mediawiki>
<siteinfo>
<sitename>Wiki</sitename>
<namespaces>
<namespace key="-2" case="first-letter">Media</namespace>
</namespaces>
</siteinfo>
<page id="31239628" orglength="6822" newlength="4524" stub="0" categories="0" outlinks="1" urls="10">
<title>Title</title>
<categories></categories>
<links>15099779</links>
<urls>
</urls>
<text>
Books
</text>
</page>
</mediawiki>

for el in tree.getiterator(tag='page'):
page_id = el.get('id', None) # returns second arg if id not exists
if page_id:
print page_id, el.find('title').text
else:
pprint(el.attrib)
Edit: Updated for commment: "Thanks can i print page_id and title at same time? Means 31239628 - Title"

The element.get() method is used to retrieve option attribute values in a tag:
>>> page_id = tree.find('page').get('id')
>>> if page_id:
print page_id
31239628