We Keep Coding

Iterate over JSON file using python and post request

My Json is like this
[
{
"name":"abc"
"emai":"hotmail"
},
{
"name":"def"
"emai":"gmail"
}
]
I have to read one record at time and post request
in 1st post request with data
'{
"name":"abc"
"emai":"hotmail"
}'
in 2nd post request with data
'{
"name":"def"
"emai":"gmail"
}'
i tried with this code,
import json
import requests
from requests.structures import CaseInsensitiveDict
url = "https://testurl.com"
headers = CaseInsensitiveDict()
headers["accept"] = "text/plain"
headers["username"] = "username"
headers["password"] = "password"
headers["Content-Type"] = "application/json"
with open("data.json", encoding='utf-8', errors='ignore') as json_data:
for i in json_data:
data = json.load(i, strict=False)
idata = json.dumps(data) ## converting to string
fData = idata.replace("'", '"') # replacing ' with "
output = "'"+fData+"'" ## placing ' before and after json string which is mandatory in my case
print(fData)
# resp = requests.post(url, headers=headers, data=fData)
# print(f"status_code: : {resp.status_code}, response : {resp.json}")
I'm getting error 'str' object has no attribute error, can you please suggest me what i have to change ?

load all json, then loop through it
import json
import requests
from requests.structures import CaseInsensitiveDict
url = "https://testurl.com"
headers = CaseInsensitiveDict()
headers["accept"] = "text/plain"
headers["username"] = "username"
headers["password"] = "password"
headers["Content-Type"] = "application/json"
with open("data.json", encoding='utf-8', errors='ignore') as json_string:
json_data = json.load(json_string.read(), strict=False)
for i in json_data:
data = i
idata = json.dumps(data) ## converting to string
fData = idata.replace("'", '"') # replacing ' with "
output = "'"+fData+"'" ## placing ' before and after json string which is mandatory in my case
print(fData)
# resp = requests.post(url, headers=headers, data=fData)
# print(f"status_code: : {resp.status_code}, response : {resp.json}")

I guess you want to convert your JSON data to python structure first and then process it, something like:
import json
with open("data.json", encoding='utf-8', errors='ignore') as json_data:
data = json.loads(json_data.read())
for obj in data:
resp = requests.post(url, headers=headers, data=obj)
print(f"status_code: : {resp.status_code}, response : {resp.json}")

Downloading custom Nessus scan report using Nessus API

I have python code that successfully downloads a Nessus scan report in csv format, but I need to add some additional fields to the downloaded report. I include parameters in the request payload to include some fields, but the scan that is downloaded does not include those fields.
I've tried changing the value of the reportedContents params to actual Boolean types with the True keyword.
Also, I changed the format to pdf and it exports a PDF file that is just a title page and a page with a blank table of contents.
The downloaded csv file has data in it, but only includes the default headers (i.e.):
Plugin ID,CVE,CVSS v2.0 Base Score,Risk,Host,Protocol,Port,Name,Synopsis,Description,Solution,See Also,Plugin Output
The raw output of the POST request looks like:
POST https://localhost:8834/scans/<scan_id>/export
X-ApiKeys: accessKey=accessKey;secretKey=secretKey
Content-Type: application/x-www-form-urlencoded
Content-Length: 122
format=csv&reportContents.vulnerabilitySections.exploitable_with=true&reportContents.vulnerabilitySections.references=true
def download_scan(scan_num):
# Post an export request
headers = {
'X-ApiKeys': 'accessKey=accessKey;secretKey=secretKey',
'Content-Type': 'application/x-www-form-urlencoded'
}
data = {
'format': 'csv',
'reportContents.vulnerabilitySections.exploitable_with': 'true',
'reportContents.vulnerabilitySections.references': 'true'
}
res = requests.post(url + '/scans/{id_num}/export'.format(id_num = scan_num), data=data, verify=False, headers=headers)
if res.status_code == 200:
export = json.loads(res.text)
file_id = export.get('file')
# Continually check the scan status until the status is ready
while True:
# Check file status
res = requests.get(url + '/scans/{id_num}/export/{file_num}/status'.format(id_num = scan_num, file_num = file_id), verify=False, headers=headers)
if res.status_code == 200:
status = json.loads(res.text)['status']
if status == 'ready':
break
# Download the scan
res = requests.get(url + '/scans/{scan_num}/export/{file_num}/download'.format(scan_num = scan_num, file_num = file_id), verify=False, headers=headers)
# If the scan is successfully downloaded, get the attachment file
if res.status_code == 200:
attachment = res.content
print("Scan downloaded!!!")
else:
raise Exception("Download request failed with status code: " + str(res))
return attachment
def main():
# Download the scan based on the scan_id. I have a helper function that returns the id that I am omitting here
try:
scan = download_scan(scan_id)
except Exception as e:
print(e)
quit()
with open("scan.csv", "wb") as f:
f.write(scan)
f.close()
if __name__ == "__main__":
main()

I'm having the exact same issue but with PowerShell. Neither my additional columns nor filters appear to be working. Was wondering if you'd had any joy getting this to work?
If I change the scan_id I get the correct different results, which suggests it is receiving the JSON but ignoring the columns and filters.
My JSON is as follows...
{
"scan_id": 3416,
"format": "csv",
"reportContents.vulnerabilitySections.cvss3_base_score": true,
"filters": {
"filter.0.quality": "gt",
"filter.0.filter": "cvss2_base_score",
"filter.0.value": "6.9",
"filter.1.quality": "neq",
"filter.1.filter": "cvss2_base_score",
"filter.1.value": ""
}
}

I managed to fix it, my problem was that I was using Python's requests module and it's data={} keyword, which defaults to header content-type: application-x-www-form-urlencoded, it generates reports with strictly 13 fields regardless of your payload.
To make it actually consider your payload, use the header "content-type": "application/json", in your code implicitly and json={} in your payload instead of data={}.
WILL NOT WORK:
requests.post(
nessus_url + f"/scans/{scan_id}/export",
data={
"format": "csv",
"template_id": "",
"reportContents": {
"csvColumns": {
"id": True,
"cve": True,
"cvss": True,
**other_columns,
}
}
},
verify=False,
headers={
"X-ApiKeys": f"accessKey={credentials['access_key']}; secretKey={credentials['secret_key']}",
},
)
WILL WORK:
requests.post(
nessus_url + f"/scans/{scan_id}/export",
json={
"format": "csv",
"template_id": "",
"reportContents": {
"csvColumns": {
"id": True,
"cve": True,
"cvss": True,
**other_columns
}
}
},
verify=False,
headers={
"X-ApiKeys": f"accessKey={credentials['access_key']}; secretKey={credentials['secret_key']}",
"content-type": "application/json",
},
)

How to pass query parameters value from the csv file to restapi using python

I've below codes to collect the json output from my storage device by using restAPI.
The code is working fine when I pass 2 query parameters (portId and hostGroupName) manually, but currently I want to pass all value from csv file to the query parameters and combine all output under the same json structure. Is is possible to do that? Thanks
import json
from pathlib import Path
import requests
path = Path(r"C:\Python\Code")
url = "http://192.168.1.30:23451/ConfigurationManager/v1/objects/storages/800000010888/host-wwns"
pwd = "129yZm1vhjqidGFzc3dvcmRAZW50MQ=="
headers = {
"Accept": "application/json",
"Content-Type": "application/json",
"Authorization": "Basic " + pwd
}
params = {
"portId": "CL3-C",
"hostGroupName": "expdb1a"
##### Expected to take value from list.csv file and pass it to below query parameters ######
"portId": "portID from list.csv",
"hostGroupName": "hostGroupName from list.csv"
##########################################################################################
}
response = requests.get(url, headers=headers, params=params)
stg = response.json()
json_data_w = json.dumps(stg, indent=2)
Path(path/"output.json").write_text(json_data_w)
list.csv
portId,hostGroupName
CL3-C,expdb1a
CL4-C,expdb1a
CL3-A,expdb2a
CL4-A,expdb2a
CL3-E,expapp1a
CL4-E,expapp1a
CL3-J,expapp2a
CL4-J,expapp2a
Example of output.json I pass query parameters manually:
{
"data" : [ {
"hostWwnId" : "CL3-C,10,c050760996aa003a",
"portId" : "CL3-C",
"hostGroupNumber" : 10,
"hostGroupName" : "expdb1a",
"hostWwn" : "1090384996aa003a",
"wwnNickname" : "expdb1a_v1"
}, {
"hostWwnId" : "CL3-C,10,c050760996aa003b",
"portId" : "CL3-C",
"hostGroupNumber" : 10,
"hostGroupName" : "expdb1a",
"hostWwn" : "1090384996aa003b",
"wwnNickname" : "expdb1a_v2"
} ]
}

maybe this:
with open('list.csv') as f:
lines = f.read().splitlines() #list of portId,hostGroupName
json_list = []
for l in lines[1:]: #start at 1 to skip headers
# print(l.split(',')[0],l.split(',')[1] )
params = {
"portId": l.split(',')[0],
"hostGroupName": l.split(',')[1]
}
response = requests.get(url, headers=headers, params=params)
stg = response.json()
json_data_w = json.dumps(stg, indent=2)
json_list.append(json_data_w)
Path(path/"output.json").write_text(json_list) #this writes a list of your json outputs

how to pass variables in the email body from lambda aws -python

I am trying to send email from lambda by passing variable in the html body inside payload(in send_email()).I tried using +str(latesturgency)+ and also using {latestimpact} .This doesn.t work.I am new to lambda and python.
How can I solve this?
import json
import logging
import re
import http.client
import mimetypes
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
def send_email(email,latestdescription,latestservice,latestimpact,latesturgency):
conn = http.client.HTTPSConnection("mail.us-east-1.aws.cloud.bmw")
payload = {
"from":"xxx#yyy.com",
"to": "xxx#yyy.com",
"subject": "Test mail","textbody": "body",
"htmlbody": "<h3>Test body!Please create a +str(latesturgency)+ priority ticket to {latestservice} , with below message:{latestdescription}</h3>"
}
print(payload)
headers = {
'Content-Type': 'application/json'
}
data=json.dumps(payload)
print(data)
conn.request("POST", "", data, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
print("done")
def lambda_handler(event, context):
empidemployee= event['currentIntent']["slots"]["empidemployee"]
latestdescription= event['currentIntent']["slots"]["latestdescription"]
latestservice= event['currentIntent']["slots"]["latestservice"]
latestimpact= event['currentIntent']["slots"]["latestimpact"]
latesturgency= event['currentIntent']["slots"]["latesturgency"]
email=event['currentIntent']["slots"]["email"]
send_email(email,latestdescription,latestservice,latestimpact,latesturgency)

You can do this using the format function fro your string, I have added this below.
import json
import logging
import re
import http.client
import mimetypes
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
def send_email(email,latestdescription,latestservice,latestimpact,latesturgency):
conn = http.client.HTTPSConnection("mail.us-east-1.aws.cloud.bmw")
payload = {
"from":"xxx#yyy.com",
"to": "xxx#yyy.com",
"subject": "Test mail","textbody": "body",
"htmlbody": "<h3>Test body!Please create a {} priority ticket to {}, with below message:{}</h3>".format(latesturgency, latestservice, latestdescription)
}
print(payload)
headers = {
'Content-Type': 'application/json'
}
data=json.dumps(payload)
print(data)
conn.request("POST", "", data, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))
print("done")
def lambda_handler(event, context):
empidemployee= event['currentIntent']["slots"]["empidemployee"]
latestdescription= event['currentIntent']["slots"]["latestdescription"]
latestservice= event['currentIntent']["slots"]["latestservice"]
latestimpact= event['currentIntent']["slots"]["latestimpact"]
latesturgency= event['currentIntent']["slots"]["latesturgency"]
email=event['currentIntent']["slots"]["email"]
send_email(email,latestdescription,latestservice,latestimpact,latesturgency)
Use the {} notation to indicate where a variable should be replace then use .format() after your string passing in the variables in the ordering you will use them in.
In your case it is now
"<h3>Test body!Please create a {} priority ticket to {}, with below message:{}</h3>".format(latesturgency, latestservice, latestdescription)
For more information about the format function take a read in the documentation.

Python requests how to use cursor to get every page

I'm trying to get every page and write it to a JSON. This is what I have and it writes fine to a JSON but it only writes the first 100 and doesn't cursor to each page. This is the api https://developers.gfycat.com/api/#user-feeds
import requests
import json
url = "https://api.gfycat.com/v1/users/brandonedora/gfycats"
params = {"count": "100", "cursor": "YnJhbmRvbmVkb3JhfDEsdmlnb3JvdXNkZWZpYW50ZG9iZXJtYW5waW5zY2hlcixicmFuZG9uZWRvcmEsMTU4NDYxNDQ5Mg=="}
r = requests.get(url, params=params)
text_json = json.loads(r.content)
json.dump(text_json, open("testdump.json", "w"), indent = 2)

import requests
import json
params = {
"count": "100",
"cursor": "YnJhbmRvbmVkb3JhfDEsdmlnb3JvdXNkZWZpYW50ZG9iZXJtYW5waW5zY2hlcixicmFuZG9uZWRvcmEsMTU4NDYxNDQ5Mg=="
}
def main(url):
with requests.Session() as req:
allin = []
for _ in range(1, 4):
r = req.get(url, params=params).json()
params['cursor'] = r['cursor']
print(params)
allin.append(r)
goal = json.dumps(allin, indent=4)
with open("data.txt", 'w') as f:
f.write(goal)
main("https://api.gfycat.com/v1/users/brandonedora/gfycats")