In my python script, I'm trying to loop through a text file containing domain names, and fill them to my JSON request body. The correct format required for the API call is
payload = {
"threatInfo": {
"threatEntries": [
{"url": "http://malware.wicar.org/"}, {"url": "http://urltocheck2.org"},
]
}
}
The variable I'm using to replicate this is called mystring
domain_list_formatted = []
for item in domain_list:
domain_list_formatted.append("""{"url": """ + '"{}"'.format(item) + "},")
domain_list_formatted_tuple= tuple(domain_list_formatted)
mystring = ' '.join(map(str, (domain_list_formatted_tuple)))
Printing mystring gets me the results I need to pass to the payload variable
{"url": "http://malware.wicar.org/"},
{"url": "http://www.urltocheck2.org/"},
However, I want to loop this, so I add the following loop
for item in domain_list_formatted_tuple:
printcorrectly = ' '.join(map(str, (domain_list_formatted_tuple)))
payload["threatInfo"]["threatEntries"] = [printcorrectly]
And this is the result:
['{"url": "http://malware.wicar.org/"}, {"url": "http://www.urltocheck2.org/"}']
The single quotes on the outside of the bracket completely throw it off. How is the for loop modifying or encoding the payload in a way that's creating this issue? Your help would be greatly appreciated.
Your code:
for item in domain_list_formatted_tuple:
printcorrectly = ' '.join(map(str, (domain_list_formatted_tuple)))
payload["threatInfo"]["threatEntries"] = [printcorrectly]
should probably be:
for item in domain_list_formatted_tuple:
printcorrectly = ' '.join(map(str, (domain_list_formatted_tuple)))
payload["threatInfo"]["threatEntries"] = printcorrectly
without the brackets around printcorrectly
If you have :
a = ['xxx']
print(a)
You will get output ['xxx'] with brackets and quotation marks.
Related
I try to pass a variable to nested JSON in Python script.
Script as below,
import requests, request
group = request.form['grp']
zon = request.form['zone']
load = { "extra_vars": {
"g_name": "' +str(group)+ '",
"z_name": "' +str(zon)+ '"
}
}
----
--
-
However when i post the value to the API, it seem i post word '+str(group)+' and '+str(zon)+' instead the actual value that assign under declared variable.
Since i'm very new in Python programming, does passing value to nested JSON is allow in Python?
Try the following:
group = request.form['grp']
zon = request.form['zone']
load = { "extra_vars": {
"g_name": f"{group}",
"z_name": f"{zon}"
}
}
You can pass variables into a string using f-strings and brackets around your variable (note {group}):
>>> group = "my_group"
>>> {"g_name": f"'{group}'"}
{'g_name': "'my_group'"}
Or doing simple string concatenation also, which is what you almost done in your code (but just did not properly close the ' character using "'":
>>> "'" + str(group) + "'"
"'my_group'"
All in all here's your code adapted:
load = { "extra_vars": {
"g_name": f"'{group}'",
"z_name": f"'{zon}'"
}
}
I'm trying to call a function by passing a url to it but I can't quite figure how it can be done for my use case.
On the last line in the code below, there is what I would like to do. I know it's terrible to enter dictionary key in a function like that, but I don't know how to properly pass the def key of the urls dictionary to the get_page() function. Thanks for your guidance.
This is my code:
...
urls = {
"abc": abc_url,
"def": def_url,
}
if site in urls:
print("site: " + site, "url: " + urls[site])
products = get_index_data(get_page(urls[site]))
for link in products:
time.sleep(15)
data = get_detail_data(get_page(link))
print(data)
write_csv(data, link)
def get_page(urls[site]):
...code continues...
First of all
As python is an interpreted language (which reads code line by line and does not compile); hence function implementation should be before the function call.
Now coming to your question, when calling get_page(urls[site])) it passes the value from dictionary.
So your function should have a parameter like def get_page(website_url):
To summarise -
...
urls = {
"abc": abc_url,
"def": def_url,
}
def get_page(website_url):
...function code...
if site in urls:
print("site: " + site, "url: " + urls[site])
products = get_index_data(get_page(urls[site]))
for link in products:
time.sleep(15)
data = get_detail_data(get_page(link))
print(data)
write_csv(data, link)
...code continues...
Note: You can get both keys, values from a dictionary like
if website_name, website_url in urls.items():
Yes, yet another. I can't figure out what the issue is. I'm trying to iterate over a list that is a subsection of JSON output from an API call.
This is the section of JSON that I'm working with:
[
{
"created_at": "2017-02-22 17:20:29 UTC",
"description": "",
"id": 1,
"label": "FOO",
"name": "FOO",
"title": "FOO",
"updated_at": "2018-12-04 16:37:09 UTC"
}
]
The code that I'm running that retrieves this and displays it:
#!/usr/bin/python
import json
import sys
try:
import requests
except ImportError:
print "Please install the python-requests module."
sys.exit(-1)
SAT_API = 'https://satellite6.example.com/api/v2/'
USERNAME = "admin"
PASSWORD = "password"
SSL_VERIFY = False # Ignore SSL for now
def get_json(url):
# Performs a GET using the passed URL location
r = requests.get(url, auth=(USERNAME, PASSWORD), verify=SSL_VERIFY)
return r.json()
def get_results(url):
jsn = get_json(url)
if jsn.get('error'):
print "Error: " + jsn['error']['message']
else:
if jsn.get('results'):
return jsn['results']
elif 'results' not in jsn:
return jsn
else:
print "No results found"
return None
def display_all_results(url):
results = get_results(url)
if results:
return json.dumps(results, indent=4, sort_keys=True)
def main():
orgs = display_all_results(KATELLO_API + "organizations/")
for org in orgs:
print org
if __name__ == "__main__":
main()
I appear to be missing a concept because when I print org I get each character per line such as
[
{
"
c
r
e
a
t
e
d
_
a
t
"
It does this through to the final ]
I've also tried to print org['name'] which throws the TypeError: list indices must be integers, not str Python error. This makes me think that org is being seen as a list rather than a dictionary which I thought it would be due to the [{...}] format.
What concept am I missing?
EDIT: An explanation for why I'm not getting this: I'm working with a script in the Red Hat Satellite API Guide which I'm using to base another script on. I'm basically learning as I go.
display_all_results is returning a string since you are doing json.dumps in json.dumps(results, indent=4, sort_keys=True), which converts the dictionary to a string (you are getting that dictionary from r.json() in get_json function)
You then end up iterating over the characters of that string in main, and you see one character per line
Instead just return results from display_all_results and the code will work as intended
def display_all_results(url):
#results is already a dictionary, just return it
results = get_results(url)
if results:
return results
Orgs is a result of json.dump which produces a string. So instead of this code:
for org in orgs:
print(org)
replace it with simply:
#for org in orgs:
print(orgs)
I need to parse some load balancer configuration section. It's seemingly simple (at least for a human).
Config consists of several objects with their content in curly braces like so:
ltm rule ssl-header-insert {
when HTTP_REQUEST {
HTTP::header insert "X-SSL-Connection" "yes"
}
}
ltm rule some_redirect {
priority 1
when HTTP_REQUEST {
if { (not [class match [IP::remote_addr] equals addresses_group ]) }
{
HTTP::redirect "http://some.page.example.com"
TCP::close
event disable all
}
}
The contents of each section/object is a TCL code so there will be nested curly braces. What I want to achieve is to parse this in pairs as: object identifier (after ltm rule keywords) and it's contents (tcl code within braces) as it is.
I've looked around some examples and experimented a lot, but it's really giving me a hard time. I did some debugging within pyparsing (which is a bit confusing to me too) and I think that I'm failing to detect closing braces somehow, but can't figure that out.
What I came up with so far:
from pyparsing import *
import json
list_sample = """ltm rule ssl-header-insert {
when HTTP_REQUEST {
HTTP::header insert "X-SSL-Connection" "yes"
}
}
ltm rule some_redirect {
priority 1
when HTTP_REQUEST {
if { (not [class match [IP::remote_addr] equals addresses_group ]) }
{
HTTP::redirect "http://some.page.example.com"
TCP::close
event disable all
}
}
}
ltm rule http_header_replace {
when HTTP_REQUEST {
HTTP::header replace Host some.host.example.com
}
}"""
ParserElement.defaultWhitespaceChars=(" \t")
NL = LineEnd()
END = StringEnd()
LBRACE, RBRACE = map(Suppress, '{}')
ANY_HEADER = Suppress("ltm rule ") + Word(alphas, alphanums + "_-")
END_MARK = Literal("ltm rule")
CONTENT_LINE = (~ANY_HEADER + (NotAny(RBRACE + FollowedBy(END_MARK)) + ~END + restOfLine) | (~ANY_HEADER + NotAny(RBRACE + FollowedBy(END)) + ~END + restOfLine)) | (~RBRACE + ~END + restOfLine)
ANY_HEADER.setName("HEADER").setDebug()
LBRACE.setName("LBRACE").setDebug()
RBRACE.setName("RBRACE").setDebug()
CONTENT_LINE.setName("LINE").setDebug()
template_defn = ZeroOrMore((ANY_HEADER + LBRACE +
Group(ZeroOrMore(CONTENT_LINE)) +
RBRACE))
template_defn.ignore(NL)
results = template_defn.parseString(list_sample).asList()
print("Raw print:")
print(results)
print("----------------------------------------------")
print("JSON pretty dump:")
print json.dumps(results, indent=2)
I see in the debug that some of the matches work but in the end it fails with an empty list as a result.
On a sidenote - my CONTENT_LINE part of the grammar is probably overly complicated in general, but I didn't find any simpler way to cover it so far.
The next thing would be to figure out how to preserve new lines and tabs in content part, since I need that to be unchanged in the output. But looks like I have to use ignore() function - which is skipping new lines - to parse the multiline text in the first place, so that's another challenge.
I'd be grateful for someone to help me find out what the issues are. Or maybe I should take some other approach?
I think nestedExpr('{', '}') will help. That will take care of the nested '{}'s, and wrapping in originalTextFor will preserve newlines and spaces.
import pyparsing as pp
LTM, RULE = map(pp.Keyword, "ltm rule".split())
ident = pp.Word(pp.alphas, pp.alphanums+'-_')
ltm_rule_expr = pp.Group(LTM + RULE
+ ident('name')
+ pp.originalTextFor(pp.nestedExpr('{', '}'))('body'))
Using your sample string (after adding missing trailing '}'):
for rule, _, _ in ltm_rule_expr.scanString(sample):
print(rule[0].name, rule[0].body.splitlines()[0:2])
gives
ssl-header-insert ['{', ' when HTTP_REQUEST {']
some_redirect ['{', ' priority 1']
dump() is also a good way to list out the contents of a returned ParseResults:
for rule, _, _ in ltm_rule_expr.scanString(sample):
print(rule[0].dump())
print()
['ltm', 'rule', 'ssl-header-insert', '{\n when HTTP_REQUEST {\n HTTP::header insert "X-SSL-Connection" "yes"\n}\n}']
- body: '{\n when HTTP_REQUEST {\n HTTP::header insert "X-SSL-Connection" "yes"\n}\n}'
- name: 'ssl-header-insert'
['ltm', 'rule', 'some_redirect', '{\n priority 1\n\nwhen HTTP_REQUEST {\n\n if { (not [class match [IP::remote_addr] equals addresses_group ]) }\n {\n HTTP::redirect "http://some.page.example.com"\n TCP::close\n event disable all\n }\n}}']
- body: '{\n priority 1\n\nwhen HTTP_REQUEST {\n\n if { (not [class match [IP::remote_addr] equals addresses_group ]) }\n {\n HTTP::redirect "http://some.page.example.com"\n TCP::close\n event disable all\n }\n}}'
- name: 'some_redirect'
Note that I broke up 'ltm' and 'rule' into separate keyword expressions. This guards against the case where a developer may have written valid code as ltm rule blah, with > 1 space between "ltm" and "rule". This kind of thing happens all the time, you never know where whitespace will crop up.
I want to extract the 'avail' value from the JSON output that look like this.
{
"result": {
"code": 100,
"message": "Command Successful"
},
"domains": {
"yolotaxpayers.com": {
"avail": false,
"tld": "com",
"price": "49.95",
"premium": false,
"backorder": true
}
}
}
The problem is that the ['avail'] value is under ["domains"]["domain_name"] and I can't figure out how to get the domain name.
You have my spider below. The first part works fine, but not the second one.
import scrapy
import json
from whois.items import WhoisItem
class whoislistSpider(scrapy.Spider):
name = "whois_list"
start_urls = []
f = open('test.txt', 'r')
global lines
lines = f.read().splitlines()
f.close()
def __init__(self):
for line in lines:
self.start_urls.append('http://www.example.com/api/domain/check/%s/com' % line)
def parse(self, response):
for line in lines:
jsonresponse = json.loads(response.body_as_unicode())
item = WhoisItem()
domain_name = list(jsonresponse['domains'].keys())[0]
item["avail"] = jsonresponse["domains"][domain_name]["avail"]
item["domain"] = domain_name
yield item
Thank you in advance for your replies.
Currently, it tries to get the value by the "('%s.com' % line)" key.
You need to do the string formatting correctly:
domain_name = "%s.com" % line.strip()
item["avail"] = jsonresponse["domains"][domain_name]["avail"]
Assuming you are only expecting one result per response:
domain_name = list(jsonresponse['domains'].keys())[0]
item["avail"] = jsonresponse["domains"][domain_name]["avail"]
This will work even if there is a mismatch between the domain in the file "test.txt" and the domain in the result.
To get the domain name from above json response you can use list comprehension , e.g:
domain_name = [x for x in jsonresponse.values()[0].keys()]
To get the "avail" value use same method, e.g:
avail = [x["avail"] for x in jsonresponse.values()[0].values() if "avail" in x]
to get the values in string format you should call it by index 0 e.g:
domain_name[0] and avail[0] because list comprehension results stored in list type variable.
More info on list comprehension