I have an XML document in which I want to search for some elements and if they match some criteria
I would like to delete them
However, I cannot seem to be able to access the parent of the element so that I can delete it
file = open('test.xml', "r")
elem = ElementTree.parse(file)
namespace = "{http://somens}"
props = elem.findall('.//{0}prop'.format(namespace))
for prop in props:
type = prop.attrib.get('type', None)
if type == 'json':
value = json.loads(prop.attrib['value'])
if value['name'] == 'Page1.Button1':
#here I need to access the parent of prop
# in order to delete the prop
Is there a way I can do this?
Thanks
You can remove child elements with the according remove method. To remove an element you have to call its parents remove method. Unfortunately Element does not provide a reference to its parents, so it is up to you to keep track of parent/child relations (which speaks against your use of elem.findall())
A proposed solution could look like this:
root = elem.getroot()
for child in root:
if child.name != "prop":
continue
if True:# TODO: do your check here!
root.remove(child)
PS: don't use prop.attrib.get(), use prop.get(), as explained here.
You could use xpath to select an Element's parent.
file = open('test.xml', "r")
elem = ElementTree.parse(file)
namespace = "{http://somens}"
props = elem.findall('.//{0}prop'.format(namespace))
for prop in props:
type = prop.get('type', None)
if type == 'json':
value = json.loads(prop.attrib['value'])
if value['name'] == 'Page1.Button1':
# Get parent and remove this prop
parent = prop.find("..")
parent.remove(prop)
http://docs.python.org/2/library/xml.etree.elementtree.html#supported-xpath-syntax
Except if you try that it doesn't work: http://elmpowered.skawaii.net/?p=74
So instead you have to:
file = open('test.xml', "r")
elem = ElementTree.parse(file)
namespace = "{http://somens}"
search = './/{0}prop'.format(namespace)
# Use xpath to get all parents of props
prop_parents = elem.findall(search + '/..')
for parent in prop_parents:
# Still have to find and iterate through child props
for prop in parent.findall(search):
type = prop.get('type', None)
if type == 'json':
value = json.loads(prop.attrib['value'])
if value['name'] == 'Page1.Button1':
parent.remove(prop)
It is two searches and a nested loop. The inner search is only on Elements known to contain props as first children, but that may not mean much depending on your schema.
I know this is an old thread but this kept popping up while I was trying to figure out a similar task. I did not like the accepted answer for two reasons:
1) It doesn't handle multiple nested levels of tags.
2) It will break if multiple xml tags are deleted in the same level one-after-another. Since each element is an index of Element._children you shouldn't delete while forward iterating.
I think a better more versatile solution is this:
import xml.etree.ElementTree as et
file = 'test.xml'
tree = et.parse(file)
root = tree.getroot()
def iterator(parents, nested=False):
for child in reversed(parents):
if nested:
if len(child) >= 1:
iterator(child)
if True: # Add your entire condition here
parents.remove(child)
iterator(root, nested=True)
For the OP, this should work - but I don't have the data you're working with to test if it's perfect.
import xml.etree.ElementTree as et
file = 'test.xml'
tree = et.parse(file)
namespace = "{http://somens}"
props = tree.findall('.//{0}prop'.format(namespace))
def iterator(parents, nested=False):
for child in reversed(parents):
if nested:
if len(child) >= 1:
iterator(child)
if prop.attrib.get('type') == 'json':
value = json.loads(prop.attrib['value'])
if value['name'] == 'Page1.Button1':
parents.remove(child)
iterator(props, nested=True)
A solution using lxml module
from lxml import etree
root = ET.fromstring(xml_str)
for e in root.findall('.//{http://some.name.space}node'):
parent = e.getparent()
for child in parent.find('./{http://some.name.space}node'):
try:
parent.remove(child)
except ValueError:
pass
Using the fact that every child must have a parent, I'm going to simplify #kitsu.eb's example. f using the findall command to get the children and parents, their indices will be equivalent.
file = open('test.xml', "r")
elem = ElementTree.parse(file)
namespace = "{http://somens}"
search = './/{0}prop'.format(namespace)
# Use xpath to get all parents of props
prop_parents = elem.findall(search + '/..')
props = elem.findall('.//{0}prop'.format(namespace))
for prop in props:
type = prop.attrib.get('type', None)
if type == 'json':
value = json.loads(prop.attrib['value'])
if value['name'] == 'Page1.Button1':
#use the index of the current child to find
#its parent and remove the child
prop_parents[props.index[prop]].remove(prop)
I also used XPath for this issue, but in a different way:
root = elem.getroot()
elementName = "YourElement"
#this will find all the parents of the elements with elementName
for elementParent in root.findall(".//{}/..".format(elementName)):
#this will find all the elements under the parent, and remove them
for element in elementParent.findall("{}".format(elementName)):
elementParent.remove(element)
I like to use an XPath expression for this kind of filtering. Unless I know otherwise, such an expression must be applied at the root level, which means I can't just get a parent and apply the same expression on that parent. However, it seems to me that there is a nice and flexible solution that should work with any supported XPath, as long as none of the sought nodes is the root. It goes something like this:
root = elem.getroot()
# Find all nodes matching the filter string (flt)
nodes = root.findall(flt)
while len(nodes):
# As long as there are nodes, there should be parents
# Get the first of all parents to the found nodes
parent = root.findall(flt+'/..')[0]
# Use this parent to remove the first node
parent.remove(nodes[0])
# Find all remaining nodes
nodes = root.findall(flt)
I would like only to add a comment on the accepted answer, but my lack of reputation doesn't allow me to. I wanted to add that it is important to add .findall("*")to the iterator to avoid issues, as stated in the documentation:
Note that concurrent modification while iterating can lead to problems, just like when iterating and modifying Python lists or dicts. Therefore, the example first collects all matching elements with root.findall(), and only then iterates over the list of matches.
Therefore, in the accepted answer the iteration should be for child in root.findal("*"):instead of for child in root:. Not doing so made my code skip some elements from the list.
Related
I parsed a mvn dependency tree to create a list storing info. I want to be able to go through this list & store in a new list the parent + child combos. An excerpt of how the parsed mvn tree looks is below (using pprint) & I added comments with # to show the relationships more explicitly.
[({'name': '"org.antlr antlr4"'}, #parent1
{'children': [({'name': '"org.antlr antlr4-runtime"'}, #child1-1
({'name': '"org.antlr antlr-runtime"'}, #child1-2
({'name': '"org.antlr ST4"'}, #child1-3
({'name': '"org.abego.treelayout org.abego.treelayout.core"'}, child1-4 & parent2
{'children': [({'name': '"org.hamcrest hamcrest-core"'}, #child2-1
({'name': '"org.slf4j slf4j-log4j12"'}, #parent3
{'children': [({'name': '"org.apache.commons commons-lang3"'})] #child3-1
Here's my messy attempt:
def relate(tree):
for name, subtree in tree.items():
group, artifact = name.split(":")
g = "groupId:" + group
a = "artifactId:" + artifact
c = {"children": "children"}
family = []
parent = name.group + name.artifact
if subtree:
for c in subtree:
child = name.group + name.artifact
family.append((parent, child))
return family
Is there a way to iterate through this and return a new list that returns info like shown below?
[[nameParent1, nameChild1-1],
[nameParent1, nameChild1-2],
[nameParent1, nameChild1-3],
[nameParent1, nameChild1-4],
[nameParent2, nameChild2-1],
[nameParent3, nameChild3-1]]
So for this excerpt it would be
[[org.antlr antlr4, org.antlr antlr4-runtime],
[org.antlr antlr4, org.antlr antlr-runtime],
[org.antlr antlr4, org.antlr ST4],
[org.antlr antlr4, org.abego.treelayout org.abego.treelayout.core],
[org.abego.treelayout org.abego.treelayout.core, org.hamcrest hamcrest-core],
[org.slf4j slf4j-log4j12, org.apache.commons commons-lang3]]
I'm unsure of how to iterate through this while keeping track of the relationships & it also has it be general enough to handle any amount of children with children with children (let me know if this needs clarification).
Thanks in advance!
**#FINAL CODE -> based off of Michael Bianconi's answer**
def getParentsChildren(mvn: tuple) -> list:
result = []
parent = mvn[1]['oid']
children = mvn[5]['children']
for child in children:
result.append([parent, child[1]['oid']])
if len(child) >= 2: **# MODIFIED LINE**
result.extend(getParentsChildren(child))
return result
def getAll(mvn: list) -> list:
result = []
for m in mvn:
result.extend(getParentsChildren(m))
return result **# MODIFIED LINE**
The whole thing is a list of tuples, so loop through. The first item in the tuple is the parent, and the second item is an array of tuples (technically it's a bunch of tuples nested inside each other but I'll assume that's a typo since you never close them).
def getParentsChildren(mvn: tuple) -> list:
result = []
parent = mvn[0]['name']
children = mvn[1]['children']
for child in children:
result.append([parent, child[0]['name'])
if child.length == 2: # has children
result.extend(getParentsChildren(child))
return result
def getAll(mvn: list) -> list:
result = []
for m in mvn:
result.extend(getParentsChildren(m))
Using lxml library I have objectified some elements (sample code below)
config = objectify.Element("config")
gui = objectify.Element("gui")
color = objectify.Element("color")
gui.append(color)
config.append(gui)
config.gui.color.active = "red"
config.gui.color.readonly = "black"
config.gui.color.match = "grey"
the result is the following structure
config
config.gui
config.gui.color
config.gui.color.active
config.gui.color.readonly
config.gui.color.match
I can get a full path for each of the objects
for element in config.iter():
print(element.getroottree().getpath(element))
The path elements are separated by slash but that is not a problem. I do not know how can I get only the parent part of the path so I can use setattr to change the value of given element
For example for element
config.gui.color.active
I would like to enter the command
setattr(config.gui.color, 'active', 'something')
But have no idea how get the "parent" part of full path.
You can get the parent of an element using the getparent function.
for element in config.iter():
print("path:", element.getroottree().getpath(element))
if element.getparent() is not None:
print("parent-path:", element.getroottree().getpath(element.getparent()))
You could also just remove the last part of the element path itself.
for element in config.iter():
path = element.getroottree().getpath(element)
print("path:", path)
parts = path.split("/")
parent_path = "/".join(parts[:-1])
print("parent-path:", parent_path)
So, I'm parsing this xml file of moderate size (about 27K lines). Not far into it, I'm seeing unexpected behavior from ElementTree.Element where I get Element.text for one entry but not the next, yet it's there in the source XML as you can see:
<!-- language: lang-xml -->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:enumeration value="24">
<xs:annotation>
<xs:documentation>UPC12 (item-specific) on cover 2</xs:documentation>
<xs:documentation>AKA item/price; ‘cover 2’ is defined as the inside front cover of a book</xs:documentation>
</xs:annotation>
</xs:enumeration>
<xs:enumeration value="25">
<xs:annotation>
<xs:documentation>UPC12+5 (item-specific) on cover 2</xs:documentation>
<xs:documentation>AKA item/price; ‘cover 2’ is defined as the inside front cover of a book</xs:documentation>
</xs:annotation>
</xs:enumeration>
When I encounter an enumeration tag I call this function:
import xml.etree.cElementTree as ElementTree
...
def _parse_list_item(xmlns: str, list_id: int, itemElement: ElementTree.Element) -> ListItem:
if isinstance(itemElement, ElementTree.Element):
if itemElement.attrib['value'] is not None:
item_id = itemElement.attrib['value'] # string
if list_id == 6 and (item_id == '25' or item_id=='24'):
print(list_id, item_id) # <== debug break point here
desc = None
notes = ""
for child in itemElement:
if child.tag == (xmlns + 'annotation'):
for grandchild in child:
if grandchild.tag == (xmlns + 'documentation'):
if desc is None:
desc = grandchild.text
else:
if len(notes)>0:
notes += " " # add a space
notes += grandchild.text or ""
if item_id is not None and desc is not None:
return Codex.ListItem({'itemId': item_id, 'listId': list_id, 'description': desc, 'notes': notes})
If I place a breakpoint at the print statement, when I get to the enumeration node for "24" I can look at the text for the grandchild nodes and they are as shown in the XML, i.e. "UPC12..." or "AKA item...", but when I get to the enumeration node for "25", and look at the grandchild text, it's None.
When I remove the xs: namespace by pre-filtering the XML file, the grandchild text comes through fine.
Is it possible I'm over some size limit or is there some syntax problem?
Sorry for less-than-pythonic code but I wanted to be able to examine all the intermediate values in pycharm. It's python 3.6.
Thanks for any insights you may have!
In the for loop, this condition is never met: if child.tag == (xmlns + 'annotation'):.
Why?
Try to output the child's tag. If we suppose your namespace (xmlns) is 'Steve' then:
print(child.tag) will output: {Steve}annotation, not Steveannotation.
So given this fact, if child.tag == (xmlns + 'annotation'): is always False.
You should change it to: if child.tag == ('{'+xmlns+'}annotation'):
With the same logic, you will find out you will also have to change this condition:
if grandchild.tag == (xmlns + 'documentation'):
to:
if grandchild.tag == ('{'+xmlns+'}documentation'):
So, ultimately, I solved my problem by running a pre-process on the XML file to remove the xs: namespace from all of the open/close XML tags and then I was able to successfully process the file using the function as defined above. Not sure why namespaces are causing problems, but perhaps there is a bug in cElementTree for namespace prefixes in large XML files. To #mzjn - I expect that it would be difficult to construct a minimal example as it does process hundreds of items correctly before it fails, so I would at least have to provide a fairly large XML file. Nevertheless, thanks for being a sounding board.
I use python with lxml to process the xml. After I query/filter to get to a nodes I want but I have some problem. How to get its attribute's value by xpath ? Here is my input example.
>print(etree.tostring(node, pretty_print=True ))
<rdf:li xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" rdf:resource="urn:miriam:obo.chebi:CHEBI%3A37671"/>
The value I want is in resource=... . Currently I just use the lxml to get the value. I wonder if it is possible to do in pure xpath ? thanks
EDIT: Forgot to said, this is not a root nodes so I can't use // here. I have like 2000-3000 others in xml file. My first attempt was playing around with ".#attrib" and "self::*#" but those does not seems to work.
EDIT2: I will try my best to explain (well, this is my first time to deal with xml problem using xpath. and english is not one of my favorite field....). Here is my input snippet http://pastebin.com/kZmVdbQQ (full one from here http://www.comp-sys-bio.org/yeastnet/ using version 4).
In my code, I try to get speciesTypes node with resource link chebi (<rdf:li rdf:resource="urn:miriam:obo.chebi:...."/>). and then I tried to get value from rdf:resource attribute in rdf:li. The thing is, I am pretty sure it would be easy to get attribute in child node if I start from parent node like speciesTypes, but I wonder how to do if I start from rdf:li. From my understanding, the "//" in xpath will looking for node from everywhere not just only in the current node.
below is my code
import lxml.etree as etree
tree = etree.parse("yeast_4.02.xml")
root = tree.getroot()
ns = {"sbml": "http://www.sbml.org/sbml/level2/version4",
"rdf":"http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"body":"http://www.w3.org/1999/xhtml",
"re": "http://exslt.org/regular-expressions"
}
#good enough for now
maybemeta = root.xpath("//sbml:speciesType[descendant::rdf:li[starts-with(#rdf:resource, 'urn:miriam:obo.chebi') and not(starts-with(#rdf:resource, 'urn:miriam:uniprot'))]]", namespaces = ns)
def extract_name_and_chebi(node):
name = node.attrib['name']
chebies = node.xpath("./sbml:annotation//rdf:li[starts-with(#rdf:resource, 'urn:miriam:obo.chebi') and not(starts-with(#rdf:resource, 'urn:miriam:uniprot'))]", namespaces=ns) #get all rdf:li node with chebi resource
assert len(chebies) == 1
#my current solution to get rdf:resource value from rdf:li node
rdfNS = "{" + ns.get('rdf') + "}"
chebi = chebies[0].attrib[rdfNS + 'resource']
#do protein later
return (name, chebi)
metaWithChebi = map(extract_name_and_chebi, maybemeta)
fo = open("metabolites.txt", "w")
for name, chebi in metaWithChebi:
fo.write("{0}\t{1}\n".format(name, chebi))
Prefix the attribute name with # in the XPath query:
>>> from lxml import etree
>>> xml = """\
... <?xml version="1.0" encoding="utf8"?>
... <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
... <rdf:li rdf:resource="urn:miriam:obo.chebi:CHEBI%3A37671"/>
... </rdf:RDF>
... """
>>> tree = etree.fromstring(xml)
>>> ns = {'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'}
>>> tree.xpath('//rdf:li/#rdf:resource', namespaces=ns)
['urn:miriam:obo.chebi:CHEBI%3A37671']
EDIT
Here's a revised version of the script in the question:
import lxml.etree as etree
ns = {
'sbml': 'http://www.sbml.org/sbml/level2/version4',
'rdf':'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
'body':'http://www.w3.org/1999/xhtml',
're': 'http://exslt.org/regular-expressions',
}
def extract_name_and_chebi(node):
chebies = node.xpath("""
.//rdf:li[
starts-with(#rdf:resource, 'urn:miriam:obo.chebi')
]/#rdf:resource
""", namespaces=ns)
return node.attrib['name'], chebies[0]
with open('yeast_4.02.xml') as xml:
tree = etree.parse(xml)
maybemeta = tree.xpath("""
//sbml:speciesType[descendant::rdf:li[
starts-with(#rdf:resource, 'urn:miriam:obo.chebi')]]
""", namespaces = ns)
with open('metabolites.txt', 'w') as output:
for node in maybemeta:
output.write('%s\t%s\n' % extract_name_and_chebi(node))
To select off the current node its attribute named rdf:resource, use this XPath expression:
#rdf:resource
In order for this to "work correctly" you must register the association of the prefix "rdf:" to the corresponding namespace.
If you don't know how to register the rdf namespace, it is still possible to select the attribute -- with this XPath expression:
#*[name()='rdf:resource']
Well, I got it. The xpath expression I need here is "./#rdf:resource" not ".#rdf:resource". But why ? I thought "./" indicate the child of current node.
I have an XML structure which looks similar to:
<Store>
<foo>
<book>
<isbn>123456</isbn>
</book>
<title>XYZ</title>
<checkout>no</checkout>
</foo>
<bar>
<book>
<isbn>7890</isbn>
</book>
<title>XYZ2</title>
<checkout>yes</checkout>
</bar>
</Store>
Using xml.dom.minidom only (restrictions) i would like to
1)traverse through the XML file
2)Search/Get for particular element, depending on its parent
Example: checkout element for author1, isbn for author2
3)Change/Set that element's value
4)Write the new XML structure to a file
Can anyone help here?
Thank you!
UPDATE:
This is what i have done till now
import xml.dom.minidom
checkout = "yes"
def getLoneChild(node, tagname):
assert ((node is not None) and (tagname is not None))
elem = node.getElementsByTagName(tagname)
if ((elem is None) or (len(elem) != 1)):
return None
return elem
def getLoneLeaf(node, tagname):
assert ((node is not None) and (tagname is not None))
elem = node.getElementsByTagName(tagname)
if ((elem is None) or (len(elem) != 1)):
return None
leaf = elem[0].firstChild
if (leaf is None):
return None
return leaf.data
def setcheckout(node, tagname):
assert ((node is not None) and (tagname is not None))
child = getLoneChild(node, 'foo')
Check = getLoneLeaf(child[0],'checkout')
Check = tagname
return Check
doc = xml.dom.minidom.parse('test.xml')
root = doc.getElementsByTagName('Store')[0]
output = setcheckout(root, checkout)
tmp_config = '/tmp/tmp_config.xml'
fw = open(tmp_config, 'w')
fw.write(doc.toxml())
fw.close()
I'm not entirely sure what you mean by "checkout". This script will find the element and alter the value of that element. Perhaps you can adapt it to your specific needs.
import xml.dom.minidom as DOM
# find the author as a child of the "Store"
def getAuthor(parent, author):
# by looking at the children
for child in [child for child in parent.childNodes
if child.nodeType != DOM.Element.TEXT_NODE]:
if child.tagName == author:
return child
return None
def alterElement(parent, attribute, newValue):
found = False;
# look through the child elements, skipping Text_Nodes
#(in your example these hold the "values"
for child in [child for child in parent.childNodes
if child.nodeType != DOM.Element.TEXT_NODE]:
# if the child element tagName matches target element name
if child.tagName == attribute:
# alter the data, i.e. the Text_Node value,
# which is the firstChild of the "isbn" element
child.firstChild.data = newValue
return True
else:
# otherwise look at all the children of this node.
found = alterElement(child, attribute, newValue)
if found:
break
# return found status
return found
doc = DOM.parse("test.xml")
# This assumes that there is only one "Store" in the file
root = doc.getElementsByTagName("Store")[0]
# find the author
# this assumes that there are no duplicate author names in the file
author = getAuthor(root, "foo")
if not author:
print "Author not found!"
else:
# alter an element
if not alterElement(author, "isbn", "987654321"):
print "isbn not found"
else:
# output the xml
tmp_config = '/tmp/tmp_config.xml'
f = open(tmp_config, 'w')
doc.writexml( f )
f.close()
The general idea is that you match the name of the author against the tagNames of the children of the "Store" element, then recurse through the children of the author, looking for a match against a target element tagName. There are a lot of assumptions made in this solution, but it may get you started. It's painful to try and deal with hierarchical structures like XML without using recursion.
In retrospect there was an error in the "alterElement" function. I've fixed this (note the "found" variable")