We Keep Coding

I'm trying to write a program that can scan IPs against OSINT databases, but I keep running into TypeError. I'm not sure where to go from here

So, someone started this project and it fell on my lap so I could fix it.
This is my first attempt at coding outside of school, so I'm not very experienced or good. Apologies if the solution was something obvious.
Basically this program is supposed to do a few things.
Accept one or more inputs of IP addresses
Scan the databases written into it
Return each database that flags the IP as malicious
Return any database that has an error related to the API key.
Return the location of the IP as reported by AbuseIPDB.(but I haven't even started that yet)
Right now, this is the error I'm getting.
Traceback (most recent call last):
File "script.py", line 119, in <module>
is_malicious, flagged_databases = check_ip_reputation(ip)
TypeError: cannot unpack non-iterable bool object
I have no idea how to correct that. I've rewritten a few lines to fix other errors but something new always comes up.
This is the code. Something to note, is that two databases are missing APIs. But those should return an error as mentioned above.
'''
import requests
def check_ip_reputation(ip_address):
# Set up a list to store the names of the databases that flag the IP as malicious
flagged_databases = []
# Set up the parameters for the AbuseIPDB request
params = {
'key': 'db327100238564236c6e25fe412ed23d80cfecab28691b0e672bd2a0798156250de5473bc648d255',
'ipAddress': ip_address
}
# Make the request to AbuseIPDB
try:
response = requests.get('https://api.abuseipdb.com/api/v2/check', params=params)
except requests.exceptions.RequestException as e:
print(f'Error making request to AbuseIPDB: {e}')
return False
# Extract the "abuseConfidenceScore" field from the response
abuse_score = response.json()['data']['abuseConfidenceScore']
# Set a threshold for the AbuseIPDB score
abuse_threshold = 50
# Check if the abuse score is above the threshold
if abuse_score >= abuse_threshold:
flagged_databases.append('AbuseIPDB')
# Set up the parameters for the VirusTotal request
params = {
'apikey': '7f21d9a126b73adf22ea100f883e38496f44412933a27cf1740858f3568be5e4',
'ip': ip_address
}
# Make the request to VirusTotal
try:
response = requests.get('https://www.virustotal.com/vtapi/v2/ip-address/report', params=params)
except requests.exceptions.RequestException as e:
print(f'Error making request to VirusTotal: {e}')
return False
# Extract the "response_code" field from the response
response_code = response.json()['response_code']
# Check if the response code indicates that the IP is listed
if response_code == 1:
flagged_databases.append('VirusTotal')
# Set up the parameters for the MXtoolbox request
params = {
'key': 'API_KEY',
'ip': ip_address
}
# Make the request to MXtoolbox
try:
response = requests.get('https://mxtoolbox.com/api/v1/lookup/blacklist/' + ip_address, params=params)
except requests.exceptions.RequestException as e:
print(f'Error making request to MXtoolbox: {e}')
return False
# Try to extract the "blacklist" field from the response
try:
blacklist = response.json()['blacklist']
except TypeError:
# If the response is a string, then the IP is not blacklisted
return False
# Check if the IP is listed in any of the blacklists
is_blacklisted = len(blacklist) > 0
# Return the result
return is_blacklisted
# Set up the parameters for the Talos request
params = {
'key': 'API_KEY',
'ip': ip_address
}
# Make the request to Talos
try:
response = requests.get('https://talosintelligence.com/documents/ip-blacklist', params=params)
except requests.exceptions.RequestException as e:
print(f'Error making request to Talos: {e}')
return False
# Check if the response code indicates that the IP is listed
if response.status_code == 200:
flagged_databases.append('Talos Intelligence')
##############################################################################
# Combine the results from all four databases
if(len(flagged_databases) > 0):
is_malicious = len(flagged_databases)
else:
is_malicious = 0
# Return the result
return is_malicious, flagged_databases;
##############################################################################
# Prompt the user for a list of IP addresses
ip_addresses_str = input("Enter a list of IP addresses separated by commas: ")
# Split the input string into a list of IP addresses
ip_addresses = ip_addresses_str.split(',')
# Strip any leading or trailing whitespace from the IP addresses
ip_addresses = [ip.strip() for ip in ip_addresses]
# Check the reputation of each IP address
for ip in ip_addresses:
is_malicious, flagged_databases = check_ip_reputation(ip)
if is_malicious:
print(f'{ip} has been flagged as malicious by the following databases: {", ".join(flagged_databases)}')
else:
print(f'{ip} has not been flagged as malicious by any of the OSINT databases.')
'''
Any help would be so, so appreciated.
Listed above, but I did try changing it so it could read strings and dictionary.

Here are some suggestions for making the code more robust.
I noticed that you were returning from the function in a few places, with different semantics for the result:
Errors were being returned as just a boolean
One of the checks, you were returning just a boolean rather than adding to the flagged databases
You had a return statement that matched the calling code, but you would not reach it because of the above return statements
The calling code expected a boolean and a list, but it would only ever get a boolean, which is why you got the error.
When you are querying multiple sources, and presuming you want to return all the information that is available rather than give up with one error, it may be good to put results from all of them into a data structure and then return what you have, including the errors. Let the calling code decide whether having errors is a problem since some of the results may be useful.
import requests
def check_ip_reputation(ip_address):
# Set up a list to store the names of the databases that flag the IP as malicious
databaseResults = {'Errors': [], 'ReportingMalicious': [], 'ReportingClean': []}
# Set up the parameters for the AbuseIPDB request
params = {
'key': 'db327100238564236c6e25fe412ed23d80cfecab28691b0e672bd2a0798156250de5473bc648d255',
'ipAddress': ip_address
}
# Make the request to AbuseIPDB
try:
response = requests.get('https://api.abuseipdb.com/api/v2/check', params=params)
except requests.exceptions.RequestException as e:
databaseResults['Errors'].append('AbuseIPDB')
# Extract the "abuseConfidenceScore" field from the response
abuse_score = response.json()['data']['abuseConfidenceScore']
# Set a threshold for the AbuseIPDB score
abuse_threshold = 50
# Check if the abuse score is above the threshold
if abuse_score >= abuse_threshold:
databaseResults['ReportingMalicious'].append('AbuseIPDB')
else:
databaseResults['ReportingClean'].append('AbuseIPDB')
# Set up the parameters for the VirusTotal request
params = {
'apikey': '7f21d9a126b73adf22ea100f883e38496f44412933a27cf1740858f3568be5e4',
'ip': ip_address
}
# Make the request to VirusTotal
try:
response = requests.get('https://www.virustotal.com/vtapi/v2/ip-address/report', params=params)
except requests.exceptions.RequestException as e:
databaseResults['Errors'].append('VirusTotal')
# Extract the "response_code" field from the response
response_code = response.json()['response_code']
# Check if the response code indicates that the IP is listed
if response_code == 1:
databaseResults['ReportingMalicious'].append('VirusTotal')
else:
databaseResults['ReportingClean'].append('VirusTotal')
# Set up the parameters for the MXtoolbox request
params = {
'key': 'API_KEY',
'ip': ip_address
}
# Make the request to MXtoolbox
try:
response = requests.get('https://mxtoolbox.com/api/v1/lookup/blacklist/' + ip_address, params=params)
except requests.exceptions.RequestException as e:
databaseResults['Errors'].append('MXtoolbox')
# Try to extract the "blacklist" field from the response
try:
blacklist = response.json()['blacklist']
is_blacklisted = len(blacklist) > 0
except TypeError:
is_blacklisted = False
# Return the result
if is_blacklisted:
databaseResults['ReportingMalicious'].append('MXtoolbox')
else:
databaseResults['ReportingClean'].append('MXtoolbox')
# Set up the parameters for the Talos request
params = {
'key': 'API_KEY',
'ip': ip_address
}
# Make the request to Talos
try:
response = requests.get('https://talosintelligence.com/documents/ip-blacklist', params=params)
except requests.exceptions.RequestException as e:
databaseResults['Errors'].append('TalosIntelligence')
# Check if the response code indicates that the IP is listed
if response.status_code == 200:
databaseResults['ReportingMalicious'].append('TalosIntelligence')
else:
databaseResults['ReportingClean'].append('TalosIntelligence')
##############################################################################
# Combine the results from all four databases
is_malicious = len(databaseResults['ReportingMalicious']) > 0
# Return the result
return is_malicious, databaseResults;
##############################################################################
# Prompt the user for a list of IP addresses
ip_addresses_str = input("Enter a list of IP addresses separated by commas: ")
# Split the input string into a list of IP addresses
ip_addresses = ip_addresses_str.split(',')
# Strip any leading or trailing whitespace from the IP addresses
ip_addresses = [ip.strip() for ip in ip_addresses]
# Check the reputation of each IP address
for ip in ip_addresses:
is_malicious, flagged_databases = check_ip_reputation(ip)
if is_malicious:
print(f'{ip} has been flagged as malicious by the following databases: ' + ", ".join([db for db in flagged_databases['ReportingMalicious']]) + '}')
else:
print(f'{ip} has not been flagged as malicious by any of the OSINT databases.')

LDAP: querying for all users in entire domain using sAMAccountName

I have modified this code python-paged-ldap-snippet.py from https://gist.github.com/mattfahrner/c228ead9c516fc322d3a
My problem is that when I change my SEARCHFILTER from '(&(objectCategory=person)(objectClass=user))' to '(&(objectCategory=person)(objectClass=user)(memberOf=CN=Users0,OU=Groups,DC=ad,DC=company,DC=com))'
it runs just fine.
If it is on SEARCHFILTER='(&(objectCategory=person)(objectClass=user))', I notice that the code is not entering the writeToFile function.
The objective of the code is to dump all the user information and parse the info into a file.
I tried running LDAPSEARCH against '(&(objectCategory=person)(objectClass=user))' and I manage to get the output .
Not sure what is wrong. Suggestions are greatly appreciated.
Thank you.
#!/usr/bin/python
import sys
import ldap
import os
LDAPSERVER='ldap://xxx.xxx.xxx.xxx:389'
BASEDN='dc=ad,dc=company,dc=com'
LDAPUSER = "CN=LDAPuser,OU=XXX,OU=Users,DC=ad,DC=company,DC=com"
LDAPPASSWORD = 'LDAPpassword'
PAGESIZE = 20000
ATTRLIST = ['sAMAccountName','uid']
SEARCHFILTER='(&(objectCategory=person)(objectClass=user))'
#SEARCHFILTER='(&(objectCategory=person)(objectClass=user)(memberOf=CN=Users0,OU=Groups,DC=ad,DC=company,DC=com))'
data = []
ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_ALLOW)
ldap.set_option(ldap.OPT_REFERRALS, 0)
l = ldap.initialize(LDAPSERVER)
l.protocol_version = 3 # Paged results only apply to LDAP v3
try:
l.simple_bind_s(LDAPUSER, LDAPPASSWORD)
print ' Login Done, Searching data'
except ldap.LDAPError as e:
exit('LDAP bind failed: %s' % e)
lc = ldap.controls.SimplePagedResultsControl(True,size=PAGESIZE,cookie='')
def writeToFile(data):
print ' Writing data to file'
#code to print all output into CVS file
while True:
try:
msgid = l.search_ext(BASEDN, ldap.SCOPE_SUBTREE, SEARCHFILTER, ATTRLIST, serverctrls=[lc])
except ldap.LDAPError as e:
sys.exit('LDAP search failed: %s' % e)
try:
rtype, rdata, rmsgid, serverctrls = l.result3(msgid)
except ldap.LDAPError as e:
sys.exit('Could not pull LDAP results: %s' % e)
for dn, attrs in rdata:
data.append(attrs)
pctrls = [
c for c in serverctrls if c.controlType == ldap.controls.SimplePagedResultsControl.controlType ]
if not pctrls:
print >> sys.stderr, 'Warning: Server ignores RFC 2696 control.'
break
cookie = pctrls[0].cookie
if not cookie:
writeToFile(data)
print 'Task Complete'
break
lc.controlValue = (PAGESIZE, cookie)

PAGESIZE = 20000
Lower your page size to a value <= 1000, since that's the max AD will give you at a time anyway. It's possible that it's waiting for 20000 records before requesting the next page and never getting it.

Reduce run time DNS look with socket - Python

I look to get Forward and Reverse look DNS look up for an address, i.e. a URL. I write below function with socket but time performance is about 4.5 second.
I would appreciate help and recommendation to improve time performance.
def DNS_lookups(address):
'''
Forward and reverse look DNS look up for address = URL
- forward: URL --> ip - check if is live
- reverse: ip --> domain name
'''
try:
ip_add = socket.gethostbyname(address)
try:
domain_name = socket.gethostbyaddr(ip_add)[0]
except socket.herror:
# print('oops')
return [ip_add, 'host not found']
except socket.gaierror:
# print('oops')
return ['False', 'NA']
#finally: # else:
# domain_name = socket.gethostbyaddr(socket.gethostbyname(address))[0]
return [ip_add, domain_name]

iot edge direct method handler in python

I have created a module for a Bacnet scan and it will respond with a list of devices and its address as a result. But I'm having trouble implementing a direct method handler in python. When i first tried implementing it myself i got this error. Which could mean I didn't successfully register the direct method callback. I have some references but it was from C# and azure docs is not helping me figure out the right method to register the callback. for IoTHubModuleClient there's a on_method_request_received and a receive_method_request. appreciate any help!
def iothub_client_scan_run():
try:
iot_client = iothub_client_init()
bacnet_scan_listener_thread = threading.Thread(target=device_method_listener, args=(iot_client,))
bacnet_scan_listener_thread.daemon = True
bacnet_scan_listener_thread.start()
while True:
time.sleep(1000)
def device_method_listener(iot_client):
while True:
# Receive the direct method request
method_request = iot_client.receive_method_request()
print (
"\nMethod callback called with:\nmethodName = {method_name}\npayload = {payload}".format(
method_name=method_request.name,
payload=method_request.payload
)
)
if method_request.name == "runBacnetScan":
response = bacnet_scan_device(method_request)
else:
response_payload = {"Response": "Direct method {} not defined".format(method_request.name)}
response_status = 404
# Send a method response indicating the method request was resolved
print('Sending method response')
iot_client.send_method_response(response)
print('Message sent!')
Edit:
Here is my route config

I was able to resolve my issue or at least find the root cause and it was my network configuration under the createOptions. It seems like there's an issue when I'm trying to do NetworkMode: host and connects to the IotModuleClient.connect_from_edge_environment via connect with connection string. I'm still trying to tweak the connection configuration but at least i know its not on the code.

async def method_request_handler(module_client):
while True:
method_request = await module_client.receive_method_request()
print (
"\nMethod callback called with:\nmethodName = {method_name}\npayload = {payload}".format(
method_name=method_request.name,
payload=method_request.payload
)
)
if method_request.name == "method1":
payload = {"result": True, "data": "some data"} # set response payload
status = 200 # set return status code
print("executed method1")
elif method_request.name == "method2":
payload = {"result": True, "data": 1234} # set response payload
status = 200 # set return status code
print("executed method2")
else:
payload = {"result": False, "data": "unknown method"} # set response payload
status = 400 # set return status code
print("executed unknown method: " + method_request.name)
# Send the response
method_response = MethodResponse.create_from_method_request(method_request, status, payload)
await module_client.send_method_response(method_response)
print('Message sent!')
def stdin_listener():
while True:
try:
selection = input("Press Q to quit\n")
if selection == "Q" or selection == "q":
print("Quitting...")
break
except:
time.sleep(10)
# Schedule task for C2D Listener
listeners = asyncio.gather(input1_listener(module_client), twin_patch_listener(module_client), method_request_handler(module_client))

Python: Bloomberg API is not authorized

I am trying to pull data from Bloomberg using Python API. API package comes with example codes and the programs that only requires local host work perfectly. However, the programs that uses other authorization ways are always stuck with the error:
Connecting to port 8194 on localhost
TokenGenerationFailure = {
reason = {
source = "apitkns (apiauth) on ebbdbp-ob-053"
category = "NO_AUTH"
errorCode = 12
description = "User not in emrs userid=NA\mds firm=22691"
subcategory = "INVALID_USER"
}
}
Failed to get token
No authorization
I saw one more person having similar problem but instead of solving it he chose to just use local host. I can't always use localhost because I will have to assist and troubleshoot for other users. So I need a hint how to overcome this error.
My question is how can I set the userid anything other than OS_LOGON which automatically uses the login credentials of my account so that I can use other users' name when needed? I tried to change OS_LOGON with the user name but it didn't work.
The full program I am trying to run is:
"""SnapshotRequestTemplateExample.py"""
from __future__ import print_function
from __future__ import absolute_import
import datetime
from optparse import OptionParser, OptionValueError
import blpapi
TOKEN_SUCCESS = blpapi.Name("TokenGenerationSuccess")
TOKEN_FAILURE = blpapi.Name("TokenGenerationFailure")
AUTHORIZATION_SUCCESS = blpapi.Name("AuthorizationSuccess")
TOKEN = blpapi.Name("token")
def authOptionCallback(_option, _opt, value, parser):
vals = value.split('=', 1)
if value == "user":
parser.values.auth = "AuthenticationType=OS_LOGON"
elif value == "none":
parser.values.auth = None
elif vals[0] == "app" and len(vals) == 2:
parser.values.auth = "AuthenticationMode=APPLICATION_ONLY;"\
"ApplicationAuthenticationType=APPNAME_AND_KEY;"\
"ApplicationName=" + vals[1]
elif vals[0] == "userapp" and len(vals) == 2:
parser.values.auth = "AuthenticationMode=USER_AND_APPLICATION;"\
"AuthenticationType=OS_LOGON;"\
"ApplicationAuthenticationType=APPNAME_AND_KEY;"\
"ApplicationName=" + vals[1]
elif vals[0] == "dir" and len(vals) == 2:
parser.values.auth = "AuthenticationType=DIRECTORY_SERVICE;"\
"DirSvcPropertyName=" + vals[1]
else:
raise OptionValueError("Invalid auth option '%s'" % value)
def parseCmdLine():
"""parse cli arguments"""
parser = OptionParser(description="Retrieve realtime data.")
parser.add_option("-a",
"--ip",
dest="hosts",
help="server name or IP (default: localhost)",
metavar="ipAddress",
action="append",
default=[])
parser.add_option("-p",
dest="port",
type="int",
help="server port (default: %default)",
metavar="tcpPort",
default=8194)
parser.add_option("--auth",
dest="auth",
help="authentication option: "
"user|none|app=<app>|userapp=<app>|dir=<property>"
" (default: %default)",
metavar="option",
action="callback",
callback=authOptionCallback,
type="string",
default="user")
(opts, _) = parser.parse_args()
if not opts.hosts:
opts.hosts = ["localhost"]
if not opts.topics:
opts.topics = ["/ticker/IBM US Equity"]
return opts
def authorize(authService, identity, session, cid):
"""authorize the session for identity via authService"""
tokenEventQueue = blpapi.EventQueue()
session.generateToken(eventQueue=tokenEventQueue)
# Process related response
ev = tokenEventQueue.nextEvent()
token = None
if ev.eventType() == blpapi.Event.TOKEN_STATUS or \
ev.eventType() == blpapi.Event.REQUEST_STATUS:
for msg in ev:
print(msg)
if msg.messageType() == TOKEN_SUCCESS:
token = msg.getElementAsString(TOKEN)
elif msg.messageType() == TOKEN_FAILURE:
break
if not token:
print("Failed to get token")
return False
# Create and fill the authorization request
authRequest = authService.createAuthorizationRequest()
authRequest.set(TOKEN, token)
# Send authorization request to "fill" the Identity
session.sendAuthorizationRequest(authRequest, identity, cid)
# Process related responses
startTime = datetime.datetime.today()
WAIT_TIME_SECONDS = 10
while True:
event = session.nextEvent(WAIT_TIME_SECONDS * 1000)
if event.eventType() == blpapi.Event.RESPONSE or \
event.eventType() == blpapi.Event.REQUEST_STATUS or \
event.eventType() == blpapi.Event.PARTIAL_RESPONSE:
for msg in event:
print(msg)
if msg.messageType() == AUTHORIZATION_SUCCESS:
return True
print("Authorization failed")
return False
endTime = datetime.datetime.today()
if endTime - startTime > datetime.timedelta(seconds=WAIT_TIME_SECONDS):
return False
def main():
"""main entry point"""
global options
options = parseCmdLine()
# Fill SessionOptions
sessionOptions = blpapi.SessionOptions()
for idx, host in enumerate(options.hosts):
sessionOptions.setServerAddress(host, options.port, idx)
sessionOptions.setAuthenticationOptions(options.auth)
sessionOptions.setAutoRestartOnDisconnection(True)
print("Connecting to port %d on %s" % (
options.port, ", ".join(options.hosts)))
session = blpapi.Session(sessionOptions)
if not session.start():
print("Failed to start session.")
return
subscriptionIdentity = None
if options.auth:
subscriptionIdentity = session.createIdentity()
isAuthorized = False
authServiceName = "//blp/apiauth"
if session.openService(authServiceName):
authService = session.getService(authServiceName)
isAuthorized = authorize(authService, subscriptionIdentity,
session, blpapi.CorrelationId("auth"))
if not isAuthorized:
print("No authorization")
return
else:
print("Not using authorization")
.
.
.
.
.
finally:
session.stop()
if __name__ == "__main__":
print("SnapshotRequestTemplateExample")
try:
main()
except KeyboardInterrupt:
print("Ctrl+C pressed. Stopping...")

This example is intended for Bloomberg's BPIPE product and as such includes the necessary authorization code. For this example, if you're connecting to the Desktop API (typically localhost:8194) you would want to pass an auth parameter of "none". Note that this example is for the mktdata snapshot functionality which isn't supported by Desktop API.
You state you're trying to troubleshoot on behalf of other users, presumably traders using BPIPE under their credentials. In this case you would need to create an Identity object to represent that user.
This would be done thusly:
# Create and fill the authorization request
authRequest = authService.createAuthorizationRequest()
authRequest.set("authId", STRING_CONTAINING_USERS_EMRS_LOGON)
authRequest.set("ipAddress", STRING_OF_IP_ADDRESS_WHERE_USER_IS_LOGGED_INTO_TERMINAL)
# Send authorization request to "fill" the Identity
session.sendAuthorizationRequest(authRequest, identity, cid)
Please be aware of potential licensing compliance issues when using this approach as this can have serious consequences. If in any doubt, approach your firm's market data team who will be able to ask their Bloomberg contacts.
Edit:
As asked in the comments, it's useful to elaborate on the other possible parameters for the AuthorizationRequest.
"uuid" + "ipAddress"; this would be the default method of authenticating users for Server API. On BPIPE this would require Bloomberg to explicitly enable it for you. The UUID is the unique integer identifier assigned to each Bloomberg Anywhere user. You can look this up in the terminal by running IAM
"emrsId" + "ipAddress"; "emrsId" is a deprecated alias for "authId". This shouldn't be used anymore.
"authId" + "ipAddress"; "authId" is the String defined in EMRS (the BPIPE Entitlements Management and Reporting System) or SAPE (the Server API's equivalent of EMRS) that represents each user. This would typically be that user's OS login details (e.g. DOMAIN/USERID) or Active Directory property (e.g. mail -> blah#blah.blah)
"authId" + "ipAddress" + "application"; "application" is the application name defined on EMRS/SAPE. This will check to see whether the user defined in authId is enabled for the named application on EMRS. Using one of these user+app style Identity objects in requests should record usage against both the user and application in the EMRS usage reports.
"token"; this is the preferred approach. Using the session.generateToken functionality (which can be seen in the original question's code snippet) will result in an alphanumeric string. You'd pass this as the only parameter into the Authorization request. Note that the token generation system is virtualization-aware; if it detects it's running in Citrix or a remote desktop it will report the IP address of the display machine (or one hop towards where the user actually is).