We Keep Coding

How to get rid of None Type in my output - I only want to select valid IP addresses

Any idea how to not include anything with None? I am trying to just pull in IP addresses at this point, but I don’t want to include empty elements.
My API Response
[{'name': '', 'serial': 'Q2KN-xxxx-438Z', 'mac': '0c:8d:db:c3:ad:c8', 'networkId': 'L_6575255xxx96096977', 'model': 'MX64', 'address': '', 'lat': 38.4180951010362, 'lng': -92.098531723022, 'notes': '', 'tags': '', 'wan1Ip': '47.134.13.195', 'wan2Ip': None}, {'name': '', 'serial': 'Q2PD-xxx-QQ9Y', 'mac': '0c:8d:db:dc:ed:f6', 'networkId': 'L_657525545596096977', 'model': 'MR33', 'address': '', 'lat': 38.4180951010362, 'lng': -92.098531723022, 'notes': '', 'tags': '', 'lanIp': '10.0.0.214'}]
Iterating through elements and selecting certain fields
response = requests.request("GET", url + id + '/devices', headers=headers)
data = response.json()
for item in data:
keys = [item.get(x) for x in ['wan1Ip', 'model', 'lanIp', 'wan2Ip']]
print(*keys, sep="\n", file=sys.stdout)
My output is:
47.134.13.195
MX64
None
None
None
MR33
10.0.0.214
None
My desired output is:
47.134.13.195
10.0.0.214
I’ve tried adding a re.findall for ip addresses, but not sure that’s going to work for me. I’ve also tried to add operators for not in None and several other things.
re.findall(“(?:[\d]{1,3}).(?:[\d]{1,3}).(?:[\d]{1,3}).(?:[\d]{1,3})?“,string2 )
Update
I've changed my line to
keys = [item.get(x) for x in ['wan1Ip', 'model', 'lanIp', 'wan2Ip', '{}']if x in item]
Obviously, I still have non IP addresses in my output, but I can select the elements that have IP addresses only. My main issue was None. I will also try some of the other suggestions.

Search each key ipv4 with regex(https://stackoverflow.com/a/5284410/6250402) and check key is None with item.get(x) or ''
import re
myRegex = re.compile(r'\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.|$)){4}\b')
# data here
for item in data:
keys = [item.get(x) for x in ['wan1Ip', 'model', 'lanIp', 'wan2Ip'] if myRegex.search(item.get(x) or '')]
print(*keys, sep="\n", file=sys.stdout)

Add a filter to your list comprehension to check to see if key x is in item.
for item in data:
keys = [item.get(x) for x in ['wan1Ip', 'model', 'lanIp', 'wan2Ip'] if x in item]
print(*keys, sep="\n", file=sys.stdout)

You could add this condition to the end of your list comprehension:
keys = [item.get(x) for x in ['wan1Ip', 'model', 'lanIp', 'wan2Ip'] if item.get(x) is not None]
The output will be what you want.

Could you try this:
>> data = response.json()
>> keys = [x.get(key) for key in ['wan1Ip', 'model', 'lanIp', 'wan2Ip'] for x in data if x.get(key)]
>> print(keys)
>> ['47.134.13.195', 'MX64', 'MR33', '10.0.0.214']

Is there any way to link a field in a Scapy Packet to its portion in the built str?

I'm using Scapy to manipulate CoAP packets. Now I found one thing I want to do which I actually can't figure out how.
I have the following CoAP Packet:
>>> pkt
<CoAP ver=1L type=CON tkl=8L code=PUT msg_id=<RandShort> token=<RandBin> options=[('If-Match', '\x04\xfc\xbbL'), ('Content-Format', ''), ('Uri-Path', '.well-known'), ('Uri-Path', 'core')] paymark='\xff' |<Raw load='CC22 at 12:39:31.495' |>>
>>> str(pkt)
'H\x03\xf4\x08\xbdR_\x85\xa3\x04\xe9\xe9\x14\x04\xfc\xbbL\xab.well-known\x04core\x10\xffCC22 at 12:39:31.495'
What I'd like to do now is to replace the whole part (not the Option Value '') representing the option at index 1 ('Content-Format', '') to null (\x00) chars. I know this part is represented by str(pkt)[-22:-21] near the end of the str(pkt), which has value \x10 to represent the Content-Format option with no actual value on it.
So, I wonder: is there a way to obtain \x10 (actually, to obtain the reference str(pkt)[-22:-21]) from the high-level field (pkt.options[1]), so I can replace this part of the string by another value?
Thanks in advance!

Ok, a few hours on the problem and I came up with a solution which, if I'm correct to assume there's no easy way to do this through Scapy itself (probably due to the way CoAP Options are implemented in the CoAP contrib layer), seems to be a fair enough one.
The following function can be used to "nullify" an entire CoAP Option field:
def coap_option_to_null(pkt, opt_idx):
''' Changes the whole CoAP Option field at #opt_idx to null '''
opt = pkt.options[opt_idx]
l = pkt.options
# Sort the list of options by option number
l.sort( lambda x, y: cmp(coap_options[1][x[0]], coap_options[1][y[0]]) )
# Replace options for the sorted one and grabs new opt_idx
pkt.options = l
opt_idx = pkt.options.index(opt)
cur_delta = 0
opt_total_len = []
for opt in pkt.options:
if (option_model[opt[0]][0] - cur_delta) < 13:
delta_extended = 0
elif (option_model[opt[0]][0] - cur_delta) < 269:
delta_extended = 1
else:
delta_extended = 2
cur_delta += (option_model[opt[0]][0] - cur_delta)
if len(opt[1]) < 13:
len_extended = 0
elif len(opt[1]) < 269:
len_extended = 1
else:
len_extended = 2
opt_total_len.append(1+delta_extended+len_extended+len(opt[1]))
opt_lidx = sum(opt_total_len[:opt_idx])
opt_hidx = sum(opt_total_len[opt_idx+1:])
new_pkt = str(pkt)[:4+pkt.tkl+opt_lidx] + '\x00'*opt_total_len[opt_idx] + str(pkt)[-(len(pkt.payload)+1+sum(opt_total_len[opt_idx+1:])):]
return CoAP(new_pkt)
Below is a demonstration:
>>> pkt
<CoAP ver=1L type=CON tkl=8L code=PUT msg_id=<RandShort> token=<RandBin> options=[('If-Match', '\x04\xfc\xbbL'), ('Content-Format', ''), ('Uri-Path', 'separate')] paymark='\xff' |<Raw load='CC22 at 12:39:31.495' |>>
>>> str(pkt)
'H\x03*\x88%T\xbe\xac\xd9\xee\xcd\xbd\x14\x04\xfc\xbbL\xa8separate\x10\xffCC22 at 12:39:31.495'
>>> npkt = coap_option_to_null(pkt, 1)
>>> npkt
<CoAP ver=1L type=CON tkl=8L code=PUT msg_id=47784 token='\x90L\x0f\xd4u\xbe&\xd7' options=[('If-Match', '\x04\xfc\xbbL'), ('Uri-Path', 'separate'), ('Uri-Path', '')] paymark='\xff' |<Raw load='CC22 at 12:39:31.495' |>>
>>> str(npkt)
'H\x03\xba\xa8\x90L\x0f\xd4u\xbe&\xd7\x14\x04\xfc\xbbL\xa8separate\x00\xffCC22 at 12:39:31.495'
>>> pkt
<CoAP ver=1L type=CON tkl=8L code=PUT msg_id=<RandShort> token=<RandBin> options=[('If-Match', '\x04\xfc\xbbL'), ('Content-Format', ''), ('Uri-Path', 'separate')] paymark='\xff' |<Raw load='CC22 at 12:39:31.495' |>>
>>> npkt = coap_option_to_null(pkt, 0)
>>> npkt
<CoAP ver=1L type=CON tkl=8L code=PUT msg_id=14606 token='s\xad\xd0\xf5\x05\xac\x87C' options=[(0L, ''), (0L, ''), (0L, ''), (0L, ''), (0L, ''), (10L, 'separate'), ('Uri-Path', '')] paymark='\xff' |<Raw load='CC22 at 12:39:31.495' |>>
>>> str(npkt)
'H\x039\x0es\xad\xd0\xf5\x05\xac\x87C\x00\x00\x00\x00\x00\xa8separate\x10\xffCC22 at 12:39:31.495'

How to get results from pyparsing Forward object?

Let us assume we have the following string
string = """
object obj1{
attr1 value1;
object obj2 {
attr2 value2;
}
}
object obj3{
attr3 value3;
attr4 value4;
}
"""
There is a nested object, and we use Forward to parse this.
from pyparsing import *
word = Word(alphanums)
attribute = word.setResultsName("name")
value = word.setResultsName("value")
object_grammar = Forward()
attributes = attribute + value + Suppress(";") + LineEnd().suppress()
object_type = Suppress("object ") + word.setResultsName("object_type") + Suppress('{') + LineEnd().suppress()
object_grammar <<= object_type+\
OneOrMore(attributes|object_grammar) + Suppress("}") | Suppress("};")
for i, (obj, _, _) in enumerate(object_grammar.scanString(string)):
print('\n')
print('Enumerating over object {}'.format(i))
print('\n')
print('This is the object type {}'.format(obj.object_type))
print(obj.asXML())
print(obj.asDict())
print(obj.asList())
print(obj)
print(obj.dump())
These are the results. The obj.asXML() function contains all the information, however since it has been flattened, the order of the information is essential to parsing the result. Is this the best way to do it? I must be missing something. I would like a solution that works for both nested and not nested objects, i.e. for obj1, obj2 and obj3.
Also, setResultsName('object_type') doesn't return the object_type for the parent object. The output of the program above is shown below. Any suggestions?
Enumerating over object 0
This is the object type obj2
<ITEM>
<object_type>obj1</object_type>
<name>attr1</name>
<value>value1</value>
<object_type>obj2</object_type>
<name>attr2</name>
<value>value2</value>
</ITEM>
{'object_type': 'obj2', 'name': 'attr2', 'value': 'value2'}
['obj1', 'attr1', 'value1', 'obj2', 'attr2', 'value2']
['obj1', 'attr1', 'value1', 'obj2', 'attr2', 'value2']
['obj1', 'attr1', 'value1', 'obj2', 'attr2', 'value2']
- name: attr2
- object_type: obj2
- value: value2
Enumerating over object 1
This is the object type obj3
<ITEM>
<object_type>obj3</object_type>
<name>attr3</name>
<value>value3</value>
<name>attr4</name>
<value>value4</value>
</ITEM>
{'object_type': 'obj3', 'name': 'attr4', 'value': 'value4'}
['obj3', 'attr3', 'value3', 'attr4', 'value4']
['obj3', 'attr3', 'value3', 'attr4', 'value4']
['obj3', 'attr3', 'value3', 'attr4', 'value4']
- name: attr4
- object_type: obj3
- value: value4

While you have successfully processed your input string, let me suggest some refinements to your grammar.
When defining a recursive grammar, one usually wants to maintain some structure in the output results. In your case, the logical piece to structure is the content of each object, which is surrounded by opening and closing braces. Conceptually:
object_content = '{' + ZeroOrMore(attribute_defn | object_defn) + '}'
Then the supporting expressions are just (still conceptually):
attribute_defn = identifier + attribute_value + ';'
object_defn = 'object' + identifier + object_content
The actual Python/pyparsing for this looks like:
LBRACE,RBRACE,SEMI = map(Suppress, "{};")
word = Word(alphas, alphanums)
attribute = word
# expand to include other values if desired, such as ints, reals, strings, etc.
attribute_value = word
attributeDefn = Group(word("name") + value("value") + SEMI)
OBJECT = Keyword("object")
object_header = OBJECT + word("object_name")
object_grammar = Forward()
object_body = Group(LBRACE
+ ZeroOrMore(object_grammar | attributeDefn)
+ RBRACE)
object_grammar <<= Group(object_header + object_body("object_content"))
Group does two things for us: it structures the results into sub-objects; and it keeps the results names at one level from stepping on those at a different level (so no need for listAllMatches).
Now instead of scanString, you can just process your input using OneOrMore:
print(OneOrMore(object_grammar).parseString(string).dump())
Giving:
[['object', 'obj1', [['attr1', 'value1'], ['object', 'obj2', [['attr2', 'value2']]]]], ['object', 'obj3', [['attr3', 'value3'], ['attr4', 'value4']]]]
[0]:
['object', 'obj1', [['attr1', 'value1'], ['object', 'obj2', [['attr2', 'value2']]]]]
- object_content: [['attr1', 'value1'], ['object', 'obj2', [['attr2', 'value2']]]]
[0]:
['attr1', 'value1']
- name: attr1
- value: value1
[1]:
['object', 'obj2', [['attr2', 'value2']]]
- object_content: [['attr2', 'value2']]
[0]:
['attr2', 'value2']
- name: attr2
- value: value2
- object_name: obj2
- object_name: obj1
[1]:
['object', 'obj3', [['attr3', 'value3'], ['attr4', 'value4']]]
- object_content: [['attr3', 'value3'], ['attr4', 'value4']]
[0]:
['attr3', 'value3']
- name: attr3
- value: value3
[1]:
['attr4', 'value4']
- name: attr4
- value: value4
- object_name: obj3
I started to just make simple changes to your code, but there was a fatal flaw in your original. Your parser separated the left and right braces into two separate expressions - while this "works", it defeats the ability to define the group structure of the results.

I was able to work around this by using listAllMatches=True in the setResultsNames function. This gave me as asXML() result that had structure that I could retrieve information from. It still relies on the order of the XML and requires using zip for get the name and value for a attribute together. I'll leave this question open to see if I get a better way of doing this.

Edit keys from a python object - getting TypeError: iteration over non-sequence

When I print out my variable
print myOptions
I get
{'verbose': False,
'number': '1',
'asgMaxSize': '?dev = 2 | ?qa = 2 | ? prd = 8',
'availabilityZone': 'us-east-1a,us-east-1e',
'securityGroups': ['bouncebox-member', 'bouncebox-member'],
'\
instanceType': '?dev = m3.medium | ?qa = m3.medium | ?prd = m3.medium',
'runcommon': False,
'zone': 'inpwrd.net',
'bubblewrapp': False,
'healthCheckTarget': 'HTTP:8080/:NAME:/api/health-check',
'environment': 'dev',
'application': 'tint',
'nscustomerdata': None,
'ami': 'ami-8a5d81e2',
'userData': 'defaultUserData.txt',
'outputFile': None,
'description': 'tint - replacement thumbscraper stack',
'desiredCapacity': '1',
'securityPorts': 'httpPort8080,sshPort',
'instanceProfile': 'default-:ENV:1',
'elbListeners': 'http8080',
'name': 'tint',
'blockDeviceMap': 'Empty',
'instanceMonitoring': 'false',
'asgMinSize': '1',
'keyname': None,
'alarms': 'diskspace,memory,tint'}
(line breaks added for readability)
So it looks like a key/value pair to me. I want to iterate through it like this:
for key in myOptions:
if key == "instanceProfile":
myOptions.value = myOptions.value.replace(":ENV:", "dev")
But when I run the code I get "TypeError: iteration over non-sequence"
So, how do I determine what Type the object is and how can I walk through the key/value pair (dictionary?) that is there?
I am using python version 2.7.8 if that make a difference.
When I print out "print myOptions.class" I get "optparse.Values"
The construction of myOptions is
from optparse import OptionParser
_opt = OptionParser("%prog [options] configfiles...")
_opt.add_option("-v", "--verbose", ...)
_opt.add_option("-r", "--runcommon", ...)
.... a bunch of _opt.add_option
import shlex
def parse(filename, defaults):
myopt, args = _opt.parse_args(shlex.split(open(filename).read()), defaults)
return myopt
if __name__ == "__main__":
import sys
myOptions, args = _opt.parse_args()
myOptions = parse("default.ini", myOptions)
for arg in args:
myOptions = parse(arg, myOptions)
myOptions, args = _opt.parse_args(values = myOptions)
argConfigurationName = args[-1]
if (myOptions.name == None):
myOptions.name = argConfigurationName
if myOptions.verbose == True:
print "Configuration Variables"
for name in _opt.defaults:
print "\t", name, "=", getattr(myOptions, name)

pretty-printing OrderedDicts using pprint

I'm using pprint to nicely print a dict and it's working fine. Now I switch to using an OrderedDict from module collections. Unfortunately, the pprint routing does not seem to recognize that such objects are more or less dicts as well and falls back to printing that as a long line.
>>> d = { i:'*'*i for i in range(8) }
>>> pprint.pprint(d)
{0: '',
1: '*',
2: '**',
3: '***',
4: '****',
5: '*****',
6: '******',
7: '*******'}
>>> pprint.pprint(collections.OrderedDict(d))
OrderedDict([(0, ''), (1, '*'), (2, '**'), (3, '***'), (4, '****'), (5, '*****'), (6, '******'), (7, '*******')])
Any way to get a nicer representation of OrderedDicts as well? Maybe even if they are nested inside a normal dict or list?

I found a relatively simple solution for this, but it includes the risk of making the output for your ordered dictionary appear exactly as if it were a regular dict object.
The original solution for using a context manager to prevent pprint from sorting dictionary keys comes from this answer.
#contextlib.contextmanager
def pprint_OrderedDict():
pp_orig = pprint._sorted
od_orig = OrderedDict.__repr__
try:
pprint._sorted = lambda x:x
OrderedDict.__repr__ = dict.__repr__
yield
finally:
pprint._sorted = pp_orig
OrderedDict.__repr__ = od_orig
(You could also just patch the OrderedDict.__repr__ method with dict.__repr__, but please don't.)
Example:
>>> foo = [('Roger', 'Owner'), ('Diane', 'Manager'), ('Bob', 'Manager'),
... ('Ian', 'Associate'), ('Bill', 'Associate'), ('Melinda', 'Associate')]
>>> d = OrderedDict(foo)
>>> pprint.pprint(d)
OrderedDict([('Roger', 'Owner'), ('Diane', 'Manager'), ('Bob', 'Manager'), ('Ian', 'Associate'), ('Bill', 'Associate'), ('Melinda', 'Associate')])
>>> pprint.pprint(dict(d))
{'Bill': 'Associate',
'Bob': 'Manager',
'Diane': 'Manager',
'Ian': 'Associate',
'Melinda': 'Associate',
'Roger': 'Owner'}
>>> with pprint_OrderedDict():
... pprint.pprint(d)
...
{'Roger': 'Owner',
'Diane': 'Manager',
'Bob': 'Manager',
'Ian': 'Associate',
'Bill': 'Associate',
'Melinda': 'Associate'}

Try this on:
d = collections.OrderedDict({ i:'*'*i for i in range(8) })
EDIT
pprint.pprint(list(d.items()))

If you are specifically targeting CPython* 3.6 or later, then you can just use regular dictionaries instead of OrderedDict. You'll miss out on a few methods exclusive to OrderedDict, and this is not (yet) guaranteed to be portable to other Python implementations,** but it is probably the simplest way to accomplish what you are trying to do.
* CPython is the reference implementation of Python which may be downloaded from python.org.
** CPython stole this idea from PyPy, so you can probably depend on it working there too.

I realize this is sort of necroposting, but I thought I'd post what I use. Its main virtue is that its aoutput can be read back into python, thus allowing, for instance, to shutlle between representations (which I use, for instance, on JSON files). Of course it breaks pprint encapsulation, by ripping some code off its inner _format function.
#!/bin/env python
from __future__ import print_function
import pprint;
from collections import OrderedDict
import json
import sys
class MyPP (pprint.PrettyPrinter):
def _format(self, object, stream, indent, allowance, context, level):
if not isinstance(object, OrderedDict) :
return pprint.PrettyPrinter._format(self, object, stream, indent, allowance, context, level)
level = level + 1
objid = id(object)
if objid in context:
stream.write(_recursion(object))
self._recursive = True
self._readable = False
return
write = stream.write
_len=len
rep = self._repr(object, context, level - 1)
typ = type(object)
sepLines = _len(rep) > (self._width - 1 - indent - allowance)
if self._depth and level > self._depth:
write(rep)
return
write('OrderedDict([\n%s'%(' '*(indent+1),))
if self._indent_per_level > 1:
write((self._indent_per_level - 1) * ' ')
length = _len(object)
#import pdb; pdb.set_trace()
if length:
context[objid] = 1
indent = indent + self._indent_per_level
items = object.items()
key, ent = items[0]
rep = self._repr(key, context, level)
write('( ')
write(rep)
write(', ')
self._format(ent, stream, indent + _len(rep) + 2,
allowance + 1, context, level)
write(' )')
if length > 1:
for key, ent in items[1:]:
rep = self._repr(key, context, level)
if sepLines:
write(',\n%s( %s , ' % (' '*indent, rep))
else:
write(', ( %s , ' % rep)
self._format(ent, stream, indent + _len(rep) + 2,
allowance + 1, context, level)
write(' )')
indent = indent - self._indent_per_level
del context[objid]
write('])')
return
pp = MyPP(indent=1)
handle=open(sys.argv[1],"r")
values=json.loads(handle.read(),object_pairs_hook=OrderedDict)
pp.pprint(values)