We Keep Coding

Parsing XML Dom to a Python List using mindom - python

I have a XML file that I fetch from a website. I have placed the XML in a DOM and I am able to get most of the needed information from it except where I have the following:
<response>
<result name="response" numFound="2567888" start="0">
<doc>
<int name="ImageCount">3</int>
<arr name="Images">
<str>binder/jnws/jnws40/images/p1120.jpg</str>
<str>binder/jnws/jnws40/images/g0753.jpg</str>
<str>binder/jnws/jnws40/images/p0754.jpg</str>
</arr>
</doc>
</result>
</response>
My Code is:
for node in solardom.getElementsByTagName('doc'):
# Get the Image Count & Video Counts for this doc element ..."
imageCount = int(getMyElementValue(node, "int", "ImageCount"))
videoCount = int(getMyElementValue(node, "int", "VideoCount"))
if imageCount > 0:
print "Image Count is: " + str(imageCount)
imageList = getMyList(node, "arr", "Images", imageCount)
def getMyList(n, ntype, s, num):
list = []
i = 0
for node in n.getElementsByTagName(ntype):
if node.getAttribute("name") == s:
print "Found Image Path!!!"
I see that I am at the correct level in the XML but I can't figure out how to stuff the string value for the image paths into a Python list.
Thanks for any assistance or pointers you can give me.
Jake

Try return [child.nodeValue for child in node.childNodes].

Try xmltodict module.
>>> import xmltodict
>>> obj = xmltodict.parse(xml)
>>> print(obj['response']['result']['doc']['arr']['str'])
>>> ['binder/jnws/jnws40/images/p1120.jpg', 'binder/jnws/jnws40/images/g0753.jpg', 'binder/jnws/jnws40/images/p0754.jpg']

Ok, try this
xml = '''
<response>
<result name="response" numFound="2567888" start="0">
<doc>
<int name="ImageCount">3</int>
<arr name="Images">
<str>binder/jnws/jnws40/images/p1120.jpg</str>
<str>binder/jnws/jnws40/images/g0753.jpg</str>
<str>binder/jnws/jnws40/images/p0754.jpg</str>
</arr>
</doc>
</result>
</response>
'''
>>> import xml.etree.ElementTree as ET
>>> root = ET.fromstring(xml)
>>> imgs = [img.text for img in root.findall(".//*[#name='Images']/str")]
>>> ['binder/jnws/jnws40/images/p1120.jpg', 'binder/jnws/jnws40/images/g0753.jpg', 'binder/jnws/jnws40/images/p0754.jpg']
You can read more here

Related

I'm using minidom for read XML document. I need find the "Member" tag name and if the attributes start with "W" or "F" I need read the text between "MultilanguajeTex" tag name.
It's possible?
I try this:
mydoc = minidom.parse('DB.xml')
items = mydoc.getElementsByTagName('Member')
for elem in items:
valor = elem.attributes['Name'].value
if (valor[0] == "F" or valor[0] == "W") and len(valor)<4:
print(len(valor))
print(valor)
texto = elem.SelectNodes('MultiLanguageText')
print(texto.localName)
XML text:
<Member Name="F00" Datatype="Bool">
<Comment>
<MultiLanguageText Lang="en-GB">
BottomSidePanels_Feeding_Zone_41 Fa 00. Material Jam. Material too much time going out Fw
</MultiLanguageText>
</Comment>
Thanks!

Using minidom it could be done as follows:
from xml.dom.minidom import parseString
xml = '''<root>
<Member Name="F00" Datatype="Bool">
<Comment>
<MultiLanguageText Lang="en-GB">
BottomSidePanels_Feeding_Zone_41 Fa 00. Material Jam. Material too much time going out Fw
</MultiLanguageText>
</Comment>
</Member>
</root>'''
root = parseString(xml)
items = root.getElementsByTagName('Member')
for elem in items:
valor = elem.attributes['Name'].value
if (valor[0] == "F" or valor[0] == "W") and len(valor) < 4:
texts = elem.getElementsByTagName('MultiLanguageText')
for text in texts:
print(text.firstChild.nodeValue)
A bit simpler it could be achieved using lxml as it allows usage of XPath expressions:
import lxml.etree as etree
root = etree.fromstring(xml)
text_elements = root.xpath('.//Member[starts-with(#Name, "F") or starts-with(#Name, "W")]//MultiLanguageText')
for text_element in text_elements:
print(text_element.text.strip())

I have an XML with a number of strings:
<?xml version="1.0" encoding="UTF-8"?>
<Strings>
<String id="TEST_STRING_FROM_XML">
<en>Test string from XML</en>
<de>Testzeichenfolge aus XML</de>
<es>Cadena de prueba de XML</es>
<fr>Tester la chaîne à partir de XML</fr>
<it>Stringa di test da XML</it>
<ja>XMLからのテスト文字列</ja>
<ko>XML에서 테스트 문자열</ko>
<nl>Testreeks van XML</nl>
<pl>Łańcuch testowy z XML</pl>
<pt>Cadeia de teste de XML</pt>
<ru>Тестовая строка из XML</ru>
<sv>Teststräng från XML</sv>
<zh-CHS>从XML测试字符串</zh-CHS>
<zh-CHT>從XML測試字符串</zh-CHT>
<Comment>A test string that comes from a shared XML file.</Comment>
</String>
<String id="TEST_STRING_FROM_XML_2">
<en>Another test string from XML.</en>
<de></de>
<es></es>
<fr></fr>
<it></it>
<ja></ja>
<ko></ko>
<nl></nl>
<pl></pl>
<pt></pt>
<ru></ru>
<sv></sv>
<zh-CHS></zh-CHS>
<zh-CHT></zh-CHT>
<Comment>Another test string that comes from a shared XML file.</Comment>
</String>
</Strings>
And I would like to append these strings to a resx file with a long list of strings in the following format:
<?xml version="1.0" encoding="utf-8"?>
<root>
<!--
Microsoft ResX Schema
Version 2.0
**a bunch of schema and header stuff...**
-->
<data name="STRING_NAME_1" xml:space="preserve">
<value>This is a value 1</value>
<comment>This is a comment 1</comment>
</data>
<data name="STRING_NAME_2" xml:space="preserve">
<value>This is a value 2</value>
<comment>This is a comment 2</comment>
</data>
</root>
But using the following snippet of python code:
import sys, os, os.path, re
import xml.etree.ElementTree as ET
from xml.dom import minidom
existingStrings = []
newStrings = {}
languages = []
resx = '*path to resx file*'
def LoadAllNewStrings():
src_root = ET.parse('Strings.xml').getroot()
for src_string in src_root.findall('String'):
src_id = src_string.get('id')
src_value = src_string.findtext("en")
src_comment = src_string.findtext("Comment")
content = [src_value, src_comment]
newStrings[src_id] = content
def ExscludeExistingStrings():
dest_root = ET.parse(resx)
for stringName in dest_root.findall('Name'):
for stringId in newStrings:
if stringId == stringName:
newStrings.remove(stringId)
def PrettifyXML(element):
roughString = ET.tostring(element, 'utf-8')
reparsed = minidom.parseString(roughString)
return reparsed.toprettyxml(indent=" ")
def AddMissingStringsToLocalResource():
ExscludeExistingStrings()
with open(resx, "a") as output:
root = ET.parse(resx).getroot()
for newString in newStrings:
data = ET.Element("data", name=newString)
newStringContent = newStrings[newString]
newStringValue = newStringContent[0]
newStringComment = newStringContent[1]
ET.SubElement(data, "value").text = newStringValue
ET.SubElement(data, "comment").text = newStringComment
output.write(PrettifyXML(data))
if __name__ == "__main__":
LoadAllNewStrings()
AddMissingStringsToLocalResource()
I get the following XML appended to the end of the resx file:
<data name="STRING_NAME_2" xml:space="preserve">
<value>This is a value 1</value>
<comment>This is a comment 1</comment>
</data>
</root><?xml version="1.0" ?>
<data name="TEST_STRING_FROM_XML">
<value>Test string from XML</value>
<comment>A test string that comes from a shared XML file.</comment>
</data>
<?xml version="1.0" ?>
<data name="TEST_STRING_FROM_XML_2">
<value>Another test string from XML.</value>
<comment>Another test string that comes from a shared XML file.</comment>
</data>
I.e. the root ends and then my new strings are added after. Any ideas on how to add the data tags to the existing root properly?

with open(resx, "a") as output:
No. Don't open XML files as text files. Not for reading, not for writing, not for appending. Never.
The typical life cycle of an XML file is:
parsing (with an XML parser)
reading or Modification (with a DOM API)
if there were changes: Serializition (also with a DOM API)
At no point should you ever call open() on an XML file. XML files are not supposed to be treated as if they were plain text. They are not.
# parsing
resx = ET.parse(resx_path)
root = resx.getroot()
# modification
for newString in newStrings:
newStringContent = newStrings[newString]
# create node
data = ET.Element("data", name=newString)
ET.SubElement(data, "value").text = newStringContent[0]
ET.SubElement(data, "comment").text = newStringContent[1]
# append node, e.g. to the top level element
root.append(data)
# serialization
resx.write(resx_path, encoding='utf8')

I have the following Input XML:
<?xml version="1.0" encoding="utf-8"?>
<Scenario xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Scenario.xsd">
<TestCase>test_startup_0029</TestCase>
<ShortDescription>Restart of the EVC with missing ODO5 board.</ShortDescription>
<Events>
<Event Num="1">Switch on the EVC</Event>
</Events>
<HW-configuration>
<ELBE5A>true</ELBE5A>
<ELBE5K>false</ELBE5K>
</HW-configuration>
<SystemFailure>true</SystemFailure>
</Scenario>
My Program does add three Tags to the XML but they are formatted false.
The Output XML looks like the following:
<Scenario xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Scenario.xsd">
<TestCase>test_startup_0029</TestCase>
<ShortDescription>Restart of the EVC with missing ODO5 board.</ShortDescription>
<Events>
<Event Num="1">Switch on the EVC</Event>
</Events>
<HW-configuration>
<ELBE5A>true</ELBE5A>
<ELBE5K>false</ELBE5K>
</HW-configuration>
<SystemFailure>true</SystemFailure>
<Duration>12</Duration><EVC-SW-Version>08.02.0001.0027</EVC-SW-Version><STAC-Release>08.02.0001.0027</STAC-Release></Scenario>
Thats my Source-Code:
class XmlManager:
#staticmethod
def write_xml(xml_path, duration, evc_sw_version):
xml_path = os.path.abspath(xml_path)
if os.path.isfile(xml_path) and xml_path.endswith(".xml"):
# parse XML into etree
root = etree.parse(xml_path).getroot()
# add tags
duration_tag = etree.SubElement(root, "Duration")
duration_tag.text = duration
sw_version_tag = etree.SubElement(root, "EVC-SW-Version")
sw_version_tag.text = evc_sw_version
stac_release = evc_sw_version
stac_release_tag = etree.SubElement(root, "STAC-Release")
stac_release_tag.text = stac_release
# write changes to the XML-file
tree = etree.ElementTree(root)
tree.write(xml_path, pretty_print=False)
else:
XmlManager.logger.log("Invalid path to XML-file")
def main():
xml = r".\Test_Input_Data_Base\blnmerf1_md1czjyc_REL_V_08.01.0001.000x\Test_startup_0029\Test_startup_0029.xml"
XmlManager.write_xml(xml, "12", "08.02.0001.0027")
My Question is how to add the new tags to the XML in the right format. I guess its working that way for parsing again the changed XML but its not nice formated. Any Ideas? Thanks in advance.

To ensure nice pretty-printed output, you need to do two things:
Parse the input file using an XMLParser object with remove_blank_text=True.
Write the output using pretty_print=True
Example:
from lxml import etree
parser = etree.XMLParser(remove_blank_text=True)
tree = etree.parse("Test_startup_0029.xml", parser)
root = tree.getroot()
duration_tag = etree.SubElement(root, "Duration")
duration_tag.text = "12"
sw_version_tag = etree.SubElement(root, "EVC-SW-Version")
sw_version_tag.text = "08.02.0001.0027"
stac_release_tag = etree.SubElement(root, "STAC-Release")
stac_release_tag.text = "08.02.0001.0027"
tree.write("output.xml", pretty_print=True)
Contents of output.xml:
<Scenario xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Scenario.xsd">
<TestCase>test_startup_0029</TestCase>
<ShortDescription>Restart of the EVC with missing ODO5 board.</ShortDescription>
<Events>
<Event Num="1">Switch on the EVC</Event>
</Events>
<HW-configuration>
<ELBE5A>true</ELBE5A>
<ELBE5K>false</ELBE5K>
</HW-configuration>
<SystemFailure>true</SystemFailure>
<Duration>12</Duration>
<EVC-SW-Version>08.02.0001.0027</EVC-SW-Version>
<STAC-Release>08.02.0001.0027</STAC-Release>
</Scenario>
See also http://lxml.de/FAQ.html#why-doesn-t-the-pretty-print-option-reformat-my-xml-output.

I have the following code to try to parse an XML file such that it reads from external text files (if found) and inserts its contents into newly introduced tags and saves a new XML file with the resultant manipulations.
The code looks like this:
try:
import xml.etree.cElementTree as ET
except ImportError:
import xml.etree.ElementTree as ET
import os
# define our data file
data_file = 'test2_of_2016-09-19.xml'
tree = ET.ElementTree(file=data_file)
root = tree.getroot()
for element in root:
if element.find('File_directory') is not None:
directory = element.find('File_directory').text
if element.find('Introduction') is not None:
introduction = element.find('Introduction').text
if element.find('Directions') is not None:
directions = element.find('Directions').text
for element in root:
if element.find('File_directory') is not None:
if element.find('Introduction') is not None:
intro_tree = directory+introduction
with open(intro_tree, 'r') as f:
intro_text = f.read()
f.closed
intro_body = ET.SubElement(element,'Introduction_Body')
intro_body.text = intro_text
if element.find('Directions') is not None:
directions_tree = directory+directions
with open(directions_tree, 'r') as f:
directions_text = f.read()
f.closed
directions_body = ET.SubElement(element,'Directions_Body')
directions_body.text = directions_text
tree.write('new_' + data_file)
The problem is that it seems like the last found instance of file_directory, introduction, and directions is saved and spread out to multiple entries, which is not desired as each entry has its own individual record so to speak.
The source XML file looks like this:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Row>
<Entry_No>1</Entry_No>
<Waterfall_Name>Bridalveil Fall</Waterfall_Name>
<File_directory>./waterfall_writeups/1_Bridalveil_Fall/</File_directory>
<Introduction>introduction-bridalveil-fall.html</Introduction>
<Directions>directions-bridalveil-fall.html</Directions>
</Row>
<Row>
<Entry_No>52</Entry_No>
<Waterfall_Name>Switzer Falls</Waterfall_Name>
<File_directory>./waterfall_writeups/52_Switzer_Falls/</File_directory>
<Introduction>introduction-switzer-falls.html</Introduction>
<Directions>directions-switzer-falls.html</Directions>
</Row>
</Root>
The desired output XML should look like this:
<Root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Row>
<Entry_No>1</Entry_No>
<Waterfall_Name>Bridalveil Fall</Waterfall_Name>
<File_directory>./waterfall_writeups/1_Bridalveil_Fall/</File_directory>
<Introduction>introduction-bridalveil-fall.html</Introduction>
<Directions>directions-bridalveil-fall.html</Directions>
<Introduction_Body>Text from ./waterfall_writeups/1_Bridalveil_Fall/introduction-bridalveil-fall.html</Introduction_Body>
<Directions_Body>Text from ./waterfall_writeups/1_Bridalveil_Fall/directions-bridalveil-fall.html</Directions_Body>
</Row>
<Row>
<Entry_No>52</Entry_No>
<Waterfall_Name>Switzer Falls</Waterfall_Name>
<File_directory>./waterfall_writeups/52_Switzer_Falls/</File_directory>
<Introduction>introduction-switzer-falls.html</Introduction>
<Directions>directions-switzer-falls.html</Directions>
<Introduction_Body>Text from ./waterfall_writeups/52_Switzer_Falls/introduction-switzer-falls.html</Introduction_Body>
<Directions_Body>Text from ./waterfall_writeups/52_Switzer_Falls/directions-switzer-falls.html</Directions_Body>
</Row>
</Root>
But what I end up getting is:
<Root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Row>
<Entry_No>1</Entry_No>
<Waterfall_Name>Bridalveil Fall</Waterfall_Name>
<File_directory>./waterfall_writeups/1_Bridalveil_Fall/</File_directory>
<Introduction>introduction-bridalveil-fall.html</Introduction>
<Directions>directions-bridalveil-fall.html</Directions>
<Introduction_Body>Text from ./waterfall_writeups/52_Switzer_Falls/introduction-switzer-falls.html</Introduction_Body>
<Directions_Body>Text from ./waterfall_writeups/52_Switzer_Falls/directions-switzer-falls.html</Directions_Body>
</Row>
<Row>
<Entry_No>52</Entry_No>
<Waterfall_Name>Switzer Falls</Waterfall_Name>
<File_directory>./waterfall_writeups/52_Switzer_Falls/</File_directory>
<Introduction>introduction-switzer-falls.html</Introduction>
<Directions>directions-switzer-falls.html</Directions>
<Introduction_Body>Text from ./waterfall_writeups/52_Switzer_Falls/introduction-switzer-falls.html</Introduction_Body>
<Directions_Body>Text from ./waterfall_writeups/52_Switzer_Falls/directions-switzer-falls.html</Directions_Body>
</Row>
</Root>
As an aside, is there any way to introduce the body tags' content without it all being printed on one line (for readability)?

The first for loop iterates over the Row elements of your document, assigning new values to your directory, introduction, and directions variables respectively, with each iteration, ending up with the values from the last occurring Row element.
What I would do is create a dictionary to map tag names to text contents, and then use that mapping to add the new sub-elements on the fly. Example (without reading the referenced files):
for row in root:
elements = {}
for node in row:
elements[node.tag] = node.text
directory = elements['File_directory']
intro_tree = directory + elements['Introduction']
intro_body = ET.SubElement(row, 'Introduction_Body')
intro_body.text = 'Text from %s' % intro_tree
directions_tree = directory + elements['Directions']
directions_body = ET.SubElement(row, 'Directions_Body')
directions_body.text = 'Text from %s' % directions_tree

I have an XML file similar to this:
<root>
<a>Some <b>bad</b> text <i>that</i> I <u>do <i>not</i></u> want to keep.</a>
</root>
I want to remove all text in <b> or <u> elements (and descendants), and print the rest. This is what I tried:
from __future__ import print_function
import xml.etree.ElementTree as ET
tree = ET.parse('a.xml')
root = tree.getroot()
parent_map = {c:p for p in root.iter() for c in p}
for item in root.findall('.//b'):
parent_map[item].remove(item)
for item in root.findall('.//u'):
parent_map[item].remove(item)
print(''.join(root.itertext()).strip())
(I used the recipe in this answer to build the parent_map). The problem, of course, is that with remove(item) I'm also removing the text after the element, and the result is:
Some that I
whereas what I want is:
Some text that I want to keep.
Is there any solution?

If you won't end up using anything better, you can use clear() instead of remove() keeping the tail of the element:
import xml.etree.ElementTree as ET
data = """<root>
<a>Some <b>bad</b> text <i>that</i> I <u>do <i>not</i></u> want to keep.</a>
</root>"""
tree = ET.fromstring(data)
a = tree.find('a')
for element in a:
if element.tag in ('b', 'u'):
tail = element.tail
element.clear()
element.tail = tail
print ET.tostring(tree)
prints (see empty b and u tags):
<root>
<a>Some <b /> text <i>that</i> I <u /> want to keep.</a>
</root>
Also, here's a solution using xml.dom.minodom:
import xml.dom.minidom
data = """<root>
<a>Some <b>bad</b> text <i>that</i> I <u>do <i>not</i></u> want to keep.</a>
</root>"""
dom = xml.dom.minidom.parseString(data)
a = dom.getElementsByTagName('a')[0]
for child in a.childNodes:
if getattr(child, 'tagName', '') in ('u', 'b'):
a.removeChild(child)
print dom.toxml()
prints:
<?xml version="1.0" ?><root>
<a>Some text <i>that</i> I want to keep.</a>
</root>