We Keep Coding

getting "errorMessage": "'BadRequestException' object has no attribute 'message'",

call to http()URl & download the file in S3 bucket. its working. then in 2nd part i am calling guardduty & give location of s3 file to create threat intel set. while running code i am getting below error:-
Response
{
"errorMessage": "'BadRequestException' object has no attribute 'message'",
"errorType": "AttributeError",
"requestId": "bec541eb-a315-4f65-9fa9-3f1139e31f86",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 38, in lambda_handler\n if \"name already exists\" in error.message:\n"
]
}
i want to create threat intel set using the file which is in S3--(downloaded from the URl)
code:-
import boto3
from datetime import datetime
import requests.packages.urllib3 as urllib3
def lambda_handler(event, context):
url='https://rules.emergingthreats.net/blockrules/compromised-ips.txt' # put your url here
bucket = 'awssaflitetifeeds-security' #your s3 bucket
key = 'GDfeeds/compromised-ips.csv' #your desired s3 path or filename
s3=boto3.client('s3')
http=urllib3.PoolManager()
s3.upload_fileobj(http.request('GET', url,preload_content=False), bucket, key)
#------------------------------------------------------------------
# Guard Duty
#------------------------------------------------------------------
location = "https://s3://awssaflitetifeeds-security/GDfeeds/compromised-ips.csv"
timeStamp = datetime.now()
name = "TF-%s"%timeStamp.strftime("%Y%m%d")
guardduty = boto3.client('guardduty')
response = guardduty.list_detectors()
if len(response['DetectorIds']) == 0:
raise Exception('Failed to read GuardDuty info. Please check if the service is activated')
detectorId = response['DetectorIds'][0]
try:
response = guardduty.create_threat_intel_set(
Activate=True,
DetectorId=detectorId,
Format='FIRE_EYE',
Location=location,
Name=name
)
except Exception as error:
if "name already exists" in error.message:
found = False
response = guardduty.list_threat_intel_sets(DetectorId=detectorId)
for setId in response['ThreatIntelSetIds']:
response = guardduty.get_threat_intel_set(DetectorId=detectorId, ThreatIntelSetId=setId)
if (name == response['Name']):
found = True
response = guardduty.update_threat_intel_set(
Activate=True,
DetectorId=detectorId,
Location=location,
Name=name,
ThreatIntelSetId=setId
)
break
if not found:
raise
#-------------------------------------------------------------------
# Update result data
#------------------------------------------------------------------
result = {
'statusCode': '200',
'body': {'message': "You requested: %s day(s) of /view/iocs indicators in CSV"%environ['DAYS_REQUESTED']}
}
except Exception as error:
logging.getLogger().error(str(error))
responseStatus = 'FAILED'
reason = error.message
result = {
'statusCode': '500',
'body': {'message': error.message}
}
finally:
#------------------------------------------------------------------
# Send Result
#------------------------------------------------------------------
if 'ResponseURL' in event:
send_response(event, context, responseStatus, responseData, event['LogicalResourceId'], reason)

The reason you are getting that error message is because the exception being returned from guardduty.create_threat_intel_set does not have the message attribute directly on the exception. I think you want either error.response['Message'] or error.response['Error']['Message'] for this exception case.
A couple of other suggestions:
you should replace the except Exception which is matching the exception showing an already-existing name with something more targeted. I'd recommend looking at what exceptions the guardduty client can throw for the particular operation and catch just the one you care about.
it is likely better to check that error.response['Error']['Code'] is exactly the error you want rather than doing a partial string match.

LDAP: querying for all users in entire domain using sAMAccountName

I have modified this code python-paged-ldap-snippet.py from https://gist.github.com/mattfahrner/c228ead9c516fc322d3a
My problem is that when I change my SEARCHFILTER from '(&(objectCategory=person)(objectClass=user))' to '(&(objectCategory=person)(objectClass=user)(memberOf=CN=Users0,OU=Groups,DC=ad,DC=company,DC=com))'
it runs just fine.
If it is on SEARCHFILTER='(&(objectCategory=person)(objectClass=user))', I notice that the code is not entering the writeToFile function.
The objective of the code is to dump all the user information and parse the info into a file.
I tried running LDAPSEARCH against '(&(objectCategory=person)(objectClass=user))' and I manage to get the output .
Not sure what is wrong. Suggestions are greatly appreciated.
Thank you.
#!/usr/bin/python
import sys
import ldap
import os
LDAPSERVER='ldap://xxx.xxx.xxx.xxx:389'
BASEDN='dc=ad,dc=company,dc=com'
LDAPUSER = "CN=LDAPuser,OU=XXX,OU=Users,DC=ad,DC=company,DC=com"
LDAPPASSWORD = 'LDAPpassword'
PAGESIZE = 20000
ATTRLIST = ['sAMAccountName','uid']
SEARCHFILTER='(&(objectCategory=person)(objectClass=user))'
#SEARCHFILTER='(&(objectCategory=person)(objectClass=user)(memberOf=CN=Users0,OU=Groups,DC=ad,DC=company,DC=com))'
data = []
ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_ALLOW)
ldap.set_option(ldap.OPT_REFERRALS, 0)
l = ldap.initialize(LDAPSERVER)
l.protocol_version = 3 # Paged results only apply to LDAP v3
try:
l.simple_bind_s(LDAPUSER, LDAPPASSWORD)
print ' Login Done, Searching data'
except ldap.LDAPError as e:
exit('LDAP bind failed: %s' % e)
lc = ldap.controls.SimplePagedResultsControl(True,size=PAGESIZE,cookie='')
def writeToFile(data):
print ' Writing data to file'
#code to print all output into CVS file
while True:
try:
msgid = l.search_ext(BASEDN, ldap.SCOPE_SUBTREE, SEARCHFILTER, ATTRLIST, serverctrls=[lc])
except ldap.LDAPError as e:
sys.exit('LDAP search failed: %s' % e)
try:
rtype, rdata, rmsgid, serverctrls = l.result3(msgid)
except ldap.LDAPError as e:
sys.exit('Could not pull LDAP results: %s' % e)
for dn, attrs in rdata:
data.append(attrs)
pctrls = [
c for c in serverctrls if c.controlType == ldap.controls.SimplePagedResultsControl.controlType ]
if not pctrls:
print >> sys.stderr, 'Warning: Server ignores RFC 2696 control.'
break
cookie = pctrls[0].cookie
if not cookie:
writeToFile(data)
print 'Task Complete'
break
lc.controlValue = (PAGESIZE, cookie)

PAGESIZE = 20000
Lower your page size to a value <= 1000, since that's the max AD will give you at a time anyway. It's possible that it's waiting for 20000 records before requesting the next page and never getting it.

Mocking requests.post [duplicate]

This question already has answers here:
How can I mock requests and the response?
(20 answers)
Closed 2 years ago.
This is my first time writing unit tests, apologies for the annoyances inevitably present, despite my best efforts. I am trying to mock requests.post but my test function is not having the desired effect, to induce a 404 status code so that I can test error handling.
mymodule.py
def scrape(data):
logger.debug(f'\nBeginning scrape function')
result = {}
exceptions = {}
for id, receipts in data.items():
logger.debug(f'Looking up Id # {id} and receipts: \n{receipts}')
dispositions = []
for receipt in receipts:
logger.debug(f'The length of receipts is:' + str(len(receipts)))
attempts = 1
while attempts < 6:
logger.debug(
f'This is attempt number {attempts} to search for {receipt}')
payload = {'receipt': 'receipt',
'button': 'CHECK+STATUS', }
try:
NOW = datetime.today().strftime('%c')
logger.debug(NOW)
logger.debug(f'Making post request for: {receipt}')
response = requests.post(URL, data=payload, headers=HEADERS, timeout=10)
except Exception as e:
logger.debug(f'There was an exception: {e}')
exceptions[id] = receipt + f': {e}'
time.sleep(3)
attempts += 1
else:
logger.debug(f'It worked {response.status_code}')
attempts = 6
disp = parse(response)
dispositions.append(f'{receipt}: {disp}')
result[id] = dispositions
logger.debug(f'Here is the result: {result}')
return result
test_mymodule.py
def test_scrape(self):
print(f'\ntest_scrape running')
# mock a 404 in scrape() here
with patch("mymodule.requests") as patched_post:
# mock a request response
patched_post.return_value.status_code = 404
print('404 mocked')
# verify the function returns nothing due to 404
result = scrape(test_data)
print(f'\n{result}')
mock_requests.post.assert_called_once()
self.assertEqual(result, {})

def test_scrape(self):
print(f'\ntest_scrape running')
# mock a 404 in scrape() here
with patch("mymodule.requests") as patched_post:
# mock a request response
patched_post.return_value.status_code = 404
print('404 mocked')
# verify the function returns nothing due to 404
result = scrape(test_data)
print(f'\n{result}')
mock_requests.post.assert_called_once()
self.assertEqual(result, {})

How to test next part of code when the request object has been mocked?

I have some a Python function written this way:
def verify_secondary_hostnames(self, publishedName, hostnames):
cname = self.get_cname(publishedName)
edge_ip = requests.get('http://{}'.format(cname)).headers['X-Edge-IP']
for hostname in hostnames:
headers = {"host": hostname}
data = requests.head("http://{}".format(edge_ip), headers=headers)
data.raise_for_status()
The unit test that I have tried to write for it looks like this:
#patch("requests.get")
#patch("requests.head")
#patch("FrontEnd.FrontEnd.get_cname")
def test_verify_secondary_hostnames(self, mock_get_cname, mock_head, mock_get):
mock_get_cname.return_value = "hulahoop"
mock_get.return_value.headers = {"X-Edge-IP": "hulahoop"}
self.fe.verify_secondary_hostnames('publishedName', ['hostname'])
self.assertEqual(mock_head.call_args,
call('http://hulahoop', headers={'host': 'hostname'}))
In this test the last part raise_for_status which actually does this:
def raise_for_status(self):
"""Raises stored :class:`HTTPError`, if one occurred."""
http_error_msg = ''
if 400 <= self.status_code < 500:
http_error_msg = '%s Client Error: %s' % (self.status_code, self.reason)
elif 500 <= self.status_code < 600:
http_error_msg = '%s Server Error: %s' % (self.status_code, self.reason)
if http_error_msg:
raise HTTPError(http_error_msg, response=self)
I can't do it here since I have already mocked the requests.head. How can I write my test to cover for that as well ? What I mean by "cover" here is that since the object is a mock object, How can I make it raise_for_status ? Do I have to create a mock method to do that or emulate a raise in my test ?
Edit: This is the latest test I have with the problems outlined:
#patch("requests.get")
#patch("requests.head")
#patch("FrontEnd.FrontEnd.get_cname")
#patch("requests.models.Response.raise_for_status")
def test_verify_secondary_hostnames(self, mock_raise,
mock_get_cname, mock_head, mock_get):
mock_get_cname.return_value = "hulahoop"
mock_get.return_value.headers = {"X-Edge-IP": "hulahoop"}
mock_head.return_value.status_code = 200
self.fe.verify_secondary_hostnames('publishedName', ['hostname'])
self.assertEqual(mock_head.call_args,
call('http://hulahoop', headers={'host': 'hostname'}))
mock_head.raise_for_status.called <=== False here
mock_head.reset_mock()
mock_head.return_value.status_code = 400
self.fe.verify_secondary_hostnames('publishedName', ['hostname'])
mock_head.raise_for_status.called <=== Still false here.

You can try something like this in your unit test:
mock_data = mock.MagicMock()
mock_data.status_code = 200 # TODO (Low Kian Seong): Test raise_for_status
mock_head.return_value = mock_data
...
mock_data.raise_for_status.assert_called_once_with()

How to extract exception message from AWS DynamoDB python exception?

I have the following block of Python code talking to DynamoDB on AWS:
try:
response = conn.batch_write_item(batch_list)
except Exception ,e:
try:
mess = e.message
except:
mess = "NOMESS"
try:
earg0 = e.args[0]
except:
earg0 = "NOEARG0"
try:
stre = str(e)
except:
stre = "NOSTRE"
print "mess = '%s'" % mess
print "earg0 = '%s'" % earg0
print "stre = '%s'" % stre
What I get is this:
mess = ''
earg0 = 'NOEARG0'
stre = 'DynamoDBValidationError: 400 Bad Request {'message': 'Item size has exceeded the maximum allowed size', '__type': 'com.amazon.coral.validate#ValidationException'}'
What I need to somehow reliably extract the message string such as 'Item size has exceeded the maximum allowed size' from e. How can I do it?

I'm assuming you're using boto to access DynamoDB.
Here is the JSONResponseError (supersuperclass of DynamoDBValidationError) __init__ method:
self.status = status
self.reason = reason
self.body = body
if self.body:
self.error_message = self.body.get('message', None)
self.error_code = self.body.get('__type', None)
if self.error_code:
self.error_code = self.error_code.split('#')[-1]
Wild guess: I would go with e.error_message to get 'Item size has exceeded ...'.
You can also print all attributes (and their values) of e:
for attr in dir(e):
print "e[%r] = '''%s'''" % (attr, getattr(e, attr))

Take e.body, You will get the error as a dictionary.
example:
{u'message': u'The conditional request failed', u'__type': u'com.amazonaws.dynamodb.v20120810#ConditionalCheckFailedException'}
From this easily you will get message.