Related
how to use nse in python? for example
nmap -p80 google.com --script=http-enum
Output:
PORT STATE SERVICE
80/tcp open http
| http-enum:
|_ /partners/: Potentially interesting folder
Python:
nm.scan('google.com', arguments=f'-p80 --script=http-enum')
Output:
{'142.250.186.142': {'nmap': {'command_line': 'nmap -oX - -p80 --script=http-enum 142.250.186.142', 'scaninfo': {'tcp': {'method': 'connect', 'services': '80'}}, 'scanstats': {'timestr': 'Tue Jul 12 07:38:13 2022', 'elapsed': '11.03', 'uphosts': '1', 'downhosts': '0', 'totalhosts': '1'}}, 'scan': {'142.250.186.142': {'hostnames': [{'name': 'fra24s07-in-f14.1e100.net', 'type': 'PTR'}], 'addresses': {'ipv4': '142.250.186.142'}, 'vendor': {}, 'status': {'state': 'up', 'reason': 'syn-ack'}, 'tcp': {80: {'state': 'open', 'reason': 'syn-ack', 'name': 'http', 'product': '', 'version': '', 'extrainfo': '', 'conf': '3', 'cpe': ''}}}}}}
When called in python, looks like it returns a dictionary with all the info. To get specific info you can access these attributes separately as shown in the docs. If instead, you want to run the same command on the command line from a python script you can use subprocess:
import subprocess
command = "nmap -p80 google.com --script=http-enum"
subprocess.Popen(command.split(' '))
I'm by far not a Python3 hero, but focussed on learning some new skills with it, thus any help would be appreciated. Working on a personal project that I want to throw on GitHub later on, I run into having a command outputting the following Python dictionary:
{'masscan': {'command_line': 'masscan -oX - 192.168.0.131/24 -p 22,80 --max-rate=1000', 'scanstats': {'timestr': '2022-03-26 10:00:07', 'elapsed': '12', 'uphosts': '2', 'downhosts': '0', 'totalhosts': '2'}}, 'scan': {'192.168.0.254': {'tcp': {80: {'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '64', 'endtime': '1648285195', 'services': []}, 22: {'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '64', 'endtime': '1648285195', 'services': []}}}}}
I then want to parse that to the following JSON format:
{
"data": [
{
"{#PORT}": 80,
"{#STATE}": "OPEN",
"{#ENDTIME}": "1648285195"
},
{
"{#PORT}": 22,
"{#STATE}": "OPEN",
"{#ENDTIME}": "1648285195"
}
]
}
What would be the most efficient way to parse through it? I don't want it to end up in a file but keep it within my code preferrably. Keeping in mind that there might be more ports than just port 22 and 80. The dictionary might be a lot longer, but following the same format.
Thanks!
this function will return exactly what you want (i suppose):
def parse_data(input):
data = []
for ip in input['scan'].keys():
for protocol in input['scan'][ip].keys():
for port in input['scan'][ip][protocol].keys():
port_data = {"{#PORT}": port, "{#STATE}": input['scan'][ip][protocol][port]['state'].upper(), "{#ENDTIME}": input['scan'][ip][protocol][port]['endtime']}
data.append(port_data)
return {'data': data}
function returns (ouput):
{
"data":[
{
"{#PORT}":80,
"{#STATE}":"OPEN",
"{#ENDTIME}":"1648285195"
},
{
"{#PORT}":22,
"{#STATE}":"OPEN",
"{#ENDTIME}":"1648285195"
}
]
}
don't know where 'Interface #2' in port '22' 'state' came from (in your desired result).
Possible solution is the following:
log_data = {'masscan': {'command_line': 'masscan -oX - 192.168.0.131/24 -p 22,80 --max-rate=1000', 'scanstats': {'timestr': '2022-03-26 10:00:07', 'elapsed': '12', 'uphosts': '2', 'downhosts': '0', 'totalhosts': '2'}}, 'scan': {'192.168.0.254': {'tcp': {80: {'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '64', 'endtime': '1648285195', 'services': []}, 22: {'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '64', 'endtime': '1648285195', 'services': []}}}}}
result = {"data": []}
for k, v in dct['scan'].items():
for tcp, tcp_data in v.items():
for port, port_data in tcp_data.items():
data = {"{#PORT}": port, "{#STATE}": port_data['state'], "{#ENDTIME}": port_data['endtime']}
result["data"].append(data)
print(result)
Prints
{'data': [
{'{#PORT}': 80, '{#STATE}': 'open', '{#ENDTIME}': '1648285195'},
{'{#PORT}': 22, '{#STATE}': 'open', '{#ENDTIME}': '1648285195'}]}
You could do a recursive search for the 'tcp' key and go from there. Something like this:
mydict = {'masscan': {'command_line': 'masscan -oX - 192.168.0.131/24 -p 22,80 --max-rate=1000', 'scanstats': {'timestr': '2022-03-26 10:00:07', 'elapsed': '12', 'uphosts': '2', 'downhosts': '0', 'totalhosts': '2'}},
'scan': {'192.168.0.254': {'tcp': {80: {'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '64', 'endtime': '1648285195', 'services': []}, 22: {'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '64', 'endtime': '1648285195', 'services': []}}}}}
def findkey(d, k):
if k in d:
return d[k]
for v in d.values():
if isinstance(v, dict):
if r := findkey(v, k):
return r
rdict = {'data': []}
for k, v in findkey(mydict, 'tcp').items():
rdict['data'].append(
{'{#PORT}': k, '{#STATE}': v['state'].upper(), '{#ENDTIME}': v['endtime']})
print(rdict)
Output:
{'data': [{'{#PORT}': 80, '{#STATE}': 'OPEN', '{#ENDTIME}': '1648285195'}, {'{#PORT}': 22, '{#STATE}': 'OPEN', '{#ENDTIME}': '1648285195'}]}
Here, this is value of my dictionary but I want to get only details like product and version of 443 and 80.
Is there any way or command with the help of which, we can gethis info?
Here is my dictionary value:
{'nmap': {'scanstats': {'timestr': 'Fri Apr 17 05:08:18 2015', 'uphosts': '1', 'downhosts': '0', 'totalhosts': '1', 'elapsed': '14.91'}, 'scaninfo': {'tcp': {'services': '80,443', 'method': 'connect'}}, 'command_line': 'nmap -oX - -p 80,443 -sV xxxx'}, 'scan': {'x.x.x.x': {'status': {'state': 'up', 'reason': 'syn-ack'}, 'hostname': 'xxxx', 'vendor': {}, 'addresses': {'ipv4': '0x.x.x'}, 'tcp': {'443': {'product': 'Apache Tomcat/Coyote JSP engine', 'name': 'http', 'extrainfo': '', 'reason': 'syn-ack', 'cpe': '', 'state': 'open', 'version': '1.1', 'conf': '10'}, '80': {'product': 'Apache Tomcat/Coyote JSP engine', 'name': 'http', 'extrainfo': '', 'reason': 'syn-ack', 'cpe': '', 'state': 'open', 'version': '1.1', 'conf': '0'}}}}}
So. I ran this command
scan=[v for k,v in x.iteritems() if 'scan' in k]
It gives me result below:
[{
'x.x.x.x': {
'status': {
'state': 'up',
'reason': 'syn-ack'
},
'hostname': 'xxxx',
'vendor': {},
'addresses': {
'ipv4': 'x.x.x.x'
},
'tcp': {
'443': {
'product': 'Apache Tomcat/Coyote JSP engine',
'name': 'http',
'extrainfo': '',
'reason': 'syn-ack',
'cpe': '',
'state': 'open',
'version': '1.1',
'conf': '10'
},
'80': {
'product': '',
'name': 'http',
'extrainfo': '',
'reason': 'conn-refused',
'cpe': '',
'state': 'closed',
'version': '',
'conf': '3'
}
}
}
}]
You can try the following:
>>> data = [{'x.x.x.x': {'status': {'state': 'up', 'reason': 'syn-ack'}, 'hostname': 'xxxx', 'vendor': {}, 'addresses': {'ipv4': 'x.x.x.x'}, 'tcp': {'443': {'product': 'Apache Tomcat/Coyote JSP engine', 'name': 'http', 'extrainfo': '', 'reason': 'syn-ack', 'cpe': '', 'state': 'open', 'version': '1.1', 'conf': '10'}, '80': {'product': '', 'name': 'http', 'extrainfo': '', 'reason': 'conn-refused', 'cpe': '', 'state': 'closed', 'version': '', 'conf': '3'}}}}]
>>> for i in data[0]['x.x.x.x']['tcp']:
... print i, data[0]['x.x.x.x']['tcp'][i]['product'], data[0]['x.x.x.x']['tcp'][i]['version']
...
443 Apache Tomcat/Coyote JSP engine 1.1
80
>>>
You could use method items (iteritems in Python 2) for extracting both port number and associated information:
In [4]: for port, info in data[0]['x.x.x.x']['tcp'].items():
...: print(port, info['product'], info['version'])
...:
443 Apache Tomcat/Coyote JSP engine 1.1
80
You can always use d.keys() to see what is in the dictionary keys to traverse it.
d = your dictionary
d['x.x.x.x']['tcp']['443']['product']
Out[109]: 'Apache Tomcat/Coyote JSP engine'
d['x.x.x.x']['tcp']['443']['version']
Out[110]: '1.1'
d['x.x.x.x']['tcp']['80']['product']
Out[109]: ''
d['x.x.x.x']['tcp']['80']['version']
Out[110]: ''
Your data is basically a tree, so it can be traversed recursively with a function like this:
def parse_data(output, keys_wanted, values_wanted, data):
for key, val in data.iteritems():
if key in keys_wanted:
output.update({key: {k: val[k] for k in values_wanted}})
if isinstance(val, dict):
parse_data(output, keys_wanted, values_wanted, val)
Use:
data = <your dict>
keys = ['443', '80']
vals = ['product', 'version']
out = {}
parse_data(out, keys, vals, data)
Output:
>>> print out
{'443': {'product': 'Apache Tomcat/Coyote JSP engine', 'version': '1.1'}, '80': {'product': '', 'version': ''}}
A benefit to this function is that it's general purpose -- if you want different keys and values just pass different lists in the parameters.
BTW, in your sample input, the dict is inside a list, but there's just one item, so I stripped off the list brackets for simplicity's sake. If your actual data is in a list with many other items, you'd of course want to call this function in an iteration loop.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Closed 8 years ago.
Improve this question
Hi I'm reposting this question but providing more information about what I'm trying to achieve. Its been driving me crazy for the last few days and I can't seem to make a progress. Basically, I have this data structure:
data_in =\
{'map': {'command_line': u'command goes here',
'scaninfo': {u'tcp': {'method': u'syn', 'services': u'80,443'}},
'stats': {'downhosts': u'0',
'elapsed': u'1.71',
'timestr': u'Thu Mar 20 18:18:09 2014',
'totalhosts': u'3',
'uphosts': u'3'}},
'scan': {u'2a00:2384:0:208f::13': {'addresses': {u'ipv6': u'2a00:2384:0:f467::13',
u'mac': u'00:gf:88:9:56:D5'},
'hostname': u'static.abc.com',
'status': {'reason': u'nd-response',
'state': u'up'},
u'tcp': {80: {'conf': u'3',
'cpe': '',
'extrainfo': '',
'name': u'http',
'product': '',
'reason': u'syn-ack',
'state': u'open',
'version': ''},
443: {'conf': u'3',
'cpe': '',
'extrainfo': '',
'name': u'https',
'product': '',
'reason': u'syn-ack',
'script': {u'ssl-cert': u'place holder'},
'state': u'open',
'version': ''}},
'vendor': {u'00:0C:29:7C:13:D3': u'VMware'}},
u'2a00:2384:0:208f::15': {'addresses': {u'ipv6': u'a848:2384:0:3456::15',
u'mac': u'00:gf:29:99:6D:96'},
'hostname': u'static.xyz.com',
'status': {'reason': u'nd-response',
'state': u'up'},
u'tcp': {80: {'conf': u'3',
'cpe': '',
'extrainfo': '',
'name': u'http',
'product': '',
'reason': u'syn-ack',
'state': u'open',
'version': ''},
443: {'conf': u'3',
'cpe': '',
'extrainfo': '',
'name': u'https',
'product': '',
'reason': u'syn-ack',
'script': {u'ssl-cert': u'place holder'},
'state': u'open',
'version': ''}},
'vendor': {u'00:0C:67:99:6f:96': u'VMware'}},
u'2a00:2384:0:208f::16': {'addresses': {u'ipv6': u'8938:8584:0:8685::16',
u'mac': u'00:54:29:fg:55:0F'},
'hostname': u'static.edf.com',
'status': {'reason': u'nd-response',
'state': u'up'},
u'tcp': {80: {'conf': u'3',
'cpe': '',
'extrainfo': '',
'name': u'http',
'product': '',
'reason': u'syn-ack',
'state': u'open',
'version': ''},
443: {'conf': u'3',
'cpe': '',
'extrainfo': '',
'name': u'https',
'product': '',
'reason': u'syn-ack',
'script': {u'ssl-cert': u'place holder'},
'state': u'open',
'version': ''}},
'vendor': {u'00:0C:55:AE:33:ff': u'VMware'}}}}
And need to create a simplified version of it that looks like this:
data_out =\
[{'address': u'2a00:2384:0:208f::13',
'hostname': u'static.bt.com',
'ports': [{80: {'reason': u'syn-ack', 'state': u'open'}},
{443: {'reason': u'syn-ack',
'ssl_cert': u'place holder',
'state': u'open'}}]}]
As per previous advice from #jonrsharpe I've created a helper function that enables me to find keys. This has proved helpful, but I still struggling to get the desired results.
def find_key(data, search_key, out=None):
"""Find all values from a nested dictionary for a given key."""
if out is None:
out = []
if isinstance(data, dict):
if search_key in data:
out.append(data[search_key])
for key in data:
find_key(data[key], search_key, out)
return out
Any help would be really appreciated here!
This isn't all that hard; you just have to go through and look at the data structure that leads to what you want - which is made much harder by poor formatting, so I re-indented your input and marked the keys (<==) and fields (!!!) you are seeking:
data_in = {
'map': {
'stats': {
'uphosts': u'3',
'timestr': u'Thu Mar 20 18:18:09 2014',
'downhosts': u'0',
'totalhosts': u'3',
'elapsed': u'1.71'
},
'scaninfo': {
u'tcp': {
'services': u'80,443',
'method': u'syn'
}
},
'command_line': u'command goes here'
},
'scan': { # <==
u'2a00:2384:0:208f::13': { # <== !!!
'status': {
'state': u'up',
'reason': u'nd-response'
},
'hostname': u'static.abc.com', # !!!
'vendor': {
u'00:0C:29:7C:13:D3': u'VMware'
},
'addresses': {
u'mac': u'00:gf:88:9:56:D5',
u'ipv6': u'2a00:2384:0:f467::13'
},
u'tcp': { # <==
80: { # <== !!!
'product': '',
'state': u'open', # !!!
'version': '',
'name': u'http',
'conf': u'3',
'extrainfo': '',
'reason': u'syn-ack', # !!!
'cpe': ''
},
443: { # <== !!!
'product': '',
'state': u'open', # !!!
'version': '',
'name': u'https',
'conf': u'3',
'script': { # <==
u'ssl-cert': u'place holder' # !!!
},
'extrainfo': '',
'reason': u'syn-ack', # !!!
'cpe': ''
}
}
},
u'2a00:2384:0:208f::15': {
'status': {
'state': u'up',
'reason': u'nd-response'
},
'hostname': u'static.xyz.com',
'vendor': {
u'00:0C:67:99:6f:96': u'VMware'
},
'addresses': {
u'mac': u'00:gf:29:99:6D:96',
u'ipv6': u'a848:2384:0:3456::15'
},
u'tcp': {
80: {
'product': '',
'state': u'open',
'version': '',
'name': u'http',
'conf': u'3',
'extrainfo': '',
'reason': u'syn-ack',
'cpe': ''
},
443: {
'product': '',
'state': u'open',
'version': '',
'name': u'https',
'conf': u'3',
'script': {
u'ssl-cert': u'place holder'
},
'extrainfo': '',
'reason': u'syn-ack',
'cpe': ''
}
}
},
u'2a00:2384:0:208f::16': {
'status': {
'state': u'up',
'reason': u'nd-response'
},
'hostname': u'static.edf.com',
'vendor': {
u'00:0C:55:AE:33:ff': u'VMware'
},
'addresses': {
u'mac': u'00:54:29:fg:55:0F',
u'ipv6': u'8938:8584:0:8685::16'
},
u'tcp': {
80: {
'product': '',
'state': u'open',
'version': '',
'name': u'http',
'conf': u'3',
'extrainfo': '',
'reason': u'syn-ack',
'cpe': ''
},
443: {
'product': '',
'state': u'open',
'version': '',
'name': u'https',
'conf': u'3',
'script': {
u'ssl-cert': u'place holder'
},
'extrainfo': '',
'reason': u'syn-ack',
'cpe': ''
}
}
}
}
}
and likewise for your desired output (with appropriate adjustments):
data_out = [
{
'address': u'2a00:2384:0:208f::13',
'hostname': u'static.bt.com',
'ports': {
80: {
'state': u'open',
'reason': u'syn-ack'
},
443: {
'ssl_cert': u'place holder',
'state': u'open',
'reason': u'syn-ack'
}
}
}
]
then the extraction becomes:
def remap_port(port, port_data):
result = {
"state": port_data["state"],
"reason": port_data["reason"]
}
try:
result["ssl_cert"] = port_data["script"]["ssl-cert"]
except KeyError:
pass
return port, result
def remap_scanned_address(address, address_data):
return {
"address": address,
"hostname": address_data["hostname"],
"ports": dict(remap_port(port, port_data) for port,port_data in address_data["tcp"].items())
}
def remap_scan_data(data_in):
return [remap_scanned_address(address, address_data) for address, address_data in data_in["scan"].items()]
data_out = remap_scan_data(data_in)
which results in the desired output,
[{'address': u'2a00:2384:0:208f::13',
'hostname': u'static.abc.com',
'ports': {80: {'reason': u'syn-ack', 'state': u'open'},
443: {'reason': u'syn-ack',
'ssl_cert': u'place holder',
'state': u'open'}}},
{'address': u'2a00:2384:0:208f::15',
'hostname': u'static.xyz.com',
'ports': {80: {'reason': u'syn-ack', 'state': u'open'},
443: {'reason': u'syn-ack',
'ssl_cert': u'place holder',
'state': u'open'}}},
{'address': u'2a00:2384:0:208f::16',
'hostname': u'static.edf.com',
'ports': {80: {'reason': u'syn-ack', 'state': u'open'},
443: {'reason': u'syn-ack',
'ssl_cert': u'place holder',
'state': u'open'}}}]
I'm using python to fetch issues from Jira with xml-rpc. It works well except it is missing the 'Resolution' field in the returned dictionary. For example 'Fixed', or 'WontFix' etc.
This is how I get the issue from Jira:
import xmlrpclib
s = xmlrpclib.ServerProxy('http://myjira.com/rpc/xmlrpc')
auth = s.jira1.login('user', 'pass')
issue = s.jira1.getIssue(auth, 'PROJ-28')
print issue.keys()
And this is the list of fields that I get back:
['status', 'project', 'attachmentNames', 'votes', 'updated',
'components', 'reporter', 'customFieldValues', 'created',
'fixVersions', 'summary', 'priority', 'assignee', 'key',
'affectsVersions', 'type', 'id', 'description']
The full content is:
{'affectsVersions': [{'archived': 'false',
'id': '11314',
'name': 'v3.09',
'released': 'false',
'sequence': '7'}],
'assignee': 'myuser',
'attachmentNames': '2011-08-17_attach.tar.gz',
'components': [],
'created': '2011-06-14 12:33:54.0',
'customFieldValues': [{'customfieldId': 'customfield_10040', 'values': ''},
{'customfieldId': 'customfield_10010',
'values': 'Normal'}],
'description': "Blah blah...\r\n",
'fixVersions': [],
'id': '28322',
'key': 'PROJ-28',
'priority': '3',
'project': 'PROJ',
'reporter': 'myuser',
'status': '1',
'summary': 'blah blah...',
'type': '1',
'updated': '2011-08-18 15:41:04.0',
'votes': '0'}
When I do:
resolutions = s.jira1.getResolutions(auth )
pprint.pprint(resolutions)
I get:
[{'description': 'A fix for this issue is checked into the tree and tested.',
'id': '1',
'name': 'Fixed'},
{'description': 'The problem described is an issue which will never be fixed.',
'id': '2',
'name': "Won't Fix"},
{'description': 'The problem is a duplicate of an existing issue.',
'id': '3',
'name': 'Duplicate'},
{'description': 'The problem is not completely described.',
'id': '4',
'name': 'Incomplete'},
{'description': 'All attempts at reproducing this issue failed, or not enough information was available to reproduce the issue. Reading the code produces no clues as to why this behavior would occur. If more information appears later, please reopen the issue.',
'id': '5',
'name': 'Cannot Reproduce'},
{'description': 'Code is checked in, and is, er, ready for build.',
'id': '6',
'name': 'Ready For Build'},
{'description': 'Invalid bug', 'id': '7', 'name': 'Invalid'}]
The Jira version is v4.1.1#522 and I using Python 2.7.
Any ideas why I don't get a field called 'resolution'?
Thanks!
The answer is that the getIssue method in JiraXmlRpcService.java calls makeIssueStruct with a RemoteIssue object. The RemoteIssue object contains the Resolution field, but makeIssueStruct copies only values that are set. So if Resolution is not set, it won't appear in the Hashtable there.