def enumerate_all_config_objects(baseDN):
url = 'https://www.AwebsiteThatIWontProvide.com'
payload={"ObjectDN":baseDN,"Pattern":"*aws*","Pattern":"*jkb*"}
r = requests.post(url, verify='/PathToKeyForVerification/', headers=headers,data=json.dumps(payload))
response = r.json()
status = r.status_code
print " "
print "Returned status code:", status
print " "
return response ['Objects'][0]['GUID']
Output:
Returned status code: 200
{11871545-8c5b-4c3c-9609-7372fae1add5}
Process finished with exit code 0
I am trying to return ONLY the "GUID" information from a json request. This works (the 1187154...), as I enter values into the index between ['objects'] and ['guid'], each value is successfully produced from the list. My problem is, even though I am printing out the actual response to verify the output is correct, the final script should not require anything being dumped to a CSV file. I have to perform everything in memory. The next function that I need to create will use the returned GUID values and query the server with those values.
How do I get the items in the list to display from the output of enumerate_all_config_objects? I would like to print them to troubleshoot initially. Then I will comment out this feature and have the second function pull each value from that list and use it.
Two problems:
Print out the list which will always have an unknown number of entries.
Create another function to reference / use the values from that list.
The list is populated correctly, I've verified this. I just don't know how to access it or print it.
If I understood correctly, you are looking to do something like:
def enumerate_all_config_objects(baseDN):
url = 'https://www.AwebsiteThatIWontProvide.com'
payload = {"ObjectDN": baseDN, "Pattern": "*aws*", "Pattern": "*jkb*"}
r = requests.post(url, verify='/PathToKeyForVerification/', headers=headers,data=json.dumps(payload))
response = r.json()
status = r.status_code
return map(lambda x: x["GUID"] , response['Objects'])
def use_guids(guid_list):
#do_stuff, for example, to show the guids:
for guid in guid_list:
print(guid)
use_guids(enumerate_all_config_objects(baseDN=<replaceWithYourParameter>))
Edit : To clear out the questions from your comment, I decided to mock the call to the API which you said already works
def enumerate_all_config_objects():
foo = {"GUID" : 1}
bar = {"GUID" : 2}
baz = {"GUID" : 3}
response = {"Objects": [foo, bar, baz] }
mapped = list(map(lambda x: x["GUID"] , response['Objects']))
return map(lambda x: x["GUID"] , response['Objects'])
def use_guids(guid_list):
#do_stuff, for example, to show the guids:
for guid in guid_list:
print(guid)
use_guids(enumerate_all_config_objects())
prints out
1
2
3
When you want to use the value computed from a function, you need to use the return keyword.
For example return map(lambda x: x["GUID"] , response['Objects']), in this new example would return a map object containing [1, 2, 3]. This return value can then be used such as in my first example by passing it to another function.
In the example just above, the list is passed to the use_guids function, which prints the contents of the list.
Edit 2 : If you really insist on calling a function that handles one GUID, you can do that in this way:
def enumerate_all_config_objects(baseDN):
url = 'https://www.AwebsiteThatIWontProvide.com'
payload = {"ObjectDN": baseDN, "Pattern": "*aws*", "Pattern": "*jkb*"}
r = requests.post(url, verify='/PathToKeyForVerification/', headers=headers,data=json.dumps(payload))
response = r.json()
status = r.status_code
for obj in response['Objects']:
use_guid(obj["GUID"])
def use_guid(guid):
print(guid)
# Do some other stuff.
# ...
enumerate_all_config_objects(baseDN=<replaceWithYourParameter>)
Related
The code below errors out when trying to execute this line
"RptTime = TimeTable[0].xpath('//text()')"
Not sure why I see TimeTable has a value in my variable window, but the HtmlElement "TimeTable[0]" has no value and the "content.cssselect" at time of assignment returns value. Why then would I get an error "list index out of range". This tells me that the element is empty. I am trying to get the Year Month value in that field.
import pandas as pd
from datetime import datetime
from lxml import html
import requests
def http_request_get(url, session=None, payload=None, parse=True):
""" Sends a GET HTTP request to a website and returns its HTML content and full url address. """
if payload is None:
payload = {}
if session:
content = session.get(url, params=payload, verify=False, headers={"content-type":"text"})
else:
content = requests.get(url, params=payload, verify=False, headers={"content-type":"text"})
content.raise_for_status() # Raise HTTPError for bad requests (4xx or 5xx)
if parse:
return html.fromstring(content.text), content.url
else:
return content.text, content.url
def get_html(link):
"""
Returns a html.
"""
page_parsed, _ = http_request_get(url=link, payload={'t': ''}, parse=True)
return page_parsed
cmslinks=[
'https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/MCRAdvPartDEnrolData/Monthly-Contract-and-Enrollment-Summary-Report?items_per_page=100&items_per_page_options%5B5%5D=5%20per%20page&items_per_page_options%5B10%5D=10%20per%20page&items_per_page_options%5B25%5D=25%20per%20page&items_per_page_options%5B50%5D=50%20per%20page&items_per_page_options%5B100%5D=100%20per%20page&combine=&page=0',
'https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/MCRAdvPartDEnrolData/Monthly-Contract-and-Enrollment-Summary-Report?items_per_page=100&items_per_page_options%5B5%5D=5%20per%20page&items_per_page_options%5B10%5D=10%20per%20page&items_per_page_options%5B25%5D=25%20per%20page&items_per_page_options%5B50%5D=50%20per%20page&items_per_page_options%5B100%5D=100%20per%20page&combine=&page=1']
df=pd.DataFrame()
df2=pd.DataFrame()
for cmslink in cmslinks:
print(cmslink)
content, _ = http_request_get(url=cmslink,payload={'t':''},parse=True)
linkTable = content.cssselect('td[headers="view-dlf-1-title-table-column"]')[0]
TimeTable = content.cssselect('td[headers="view-dlf-2-report-period-table-column"]')[0]
headers = linkTable[0].xpath("//a[contains(text(),'Contract Summary') or contains(text(),'Monthly Enrollment by CPSC')]/#href")
RptTime = TimeTable.xpath('//text()')
dfl = pd.DataFrame(headers,columns= ['links'])
dft = pd.DataFrame(RptTime,columns= ['ReportTime'])
df=df.append(dfl)
df2=df.append(dft)
Error
src\lxml\etree.pyx in lxml.etree._Element.__getitem__()
IndexError: list index out of range
Look carefully at your last line. df = df.append(df1). Your explanation and code is quite unclear due to indenting and error traceback however this is obviously not what you intended.
df.append(df1) is a procedure rather than strictly a function, it does not return anything. You simply write the line and it does its magic similar to print("hi") rather than this_is_wrong = print("hi").
What would end up happening is you overwrite df will null which should be causing some major errors if you ever use that variable again. However, this is not the cause of your problem I thought it my duty to tell you anyway.
Could you please tell us exactly what is returned by the css... function. Although you said it returned something you only store the [0] index of the return value. Meaning that if it is ["","something"] hypothetically, the value stored would be null.
It is quite likely that the problem you are having is that you indexed [0] twice, when you probably only meant to do it once.
I am making a small project. The project is in the python and I am now stuck in a problem and i.e.
I have a list of URLs with their response code like 200, 300, 400, 403.
Okay let's make it more clear
http://aa.domain.com 200
http://bb.domain.com 302
http://cc.domain.com 400
http://dd.domain.com 403
http://ee.domain.com 403
Now what I exactly want is I want to seperate the URLs with their status code.
Like a "Making a Seperate list of 400 and 403 URLs".
How can I do it with python? As a newbie, I can't. Could You?
EDIT:- I have tried this
try:
req = requests.get(xxx) #xxx is a list of subdomains
responsecode = xxx , req.status_code, "\n"
print responsecode
except requests.exceptions.RequestException as e:
print "Not full fill request"
I only can print the response code. And successfully print the response code as like I said above
You can use split function to split the request in to two (request code & URL) & inserted into dictionary till the size of request list .
I am also attach the screenshot of the output .
dictionary = {}
request = ["http://aa.domain.com 200", "http://bb.domain.com 302", "http://cc.domain.com 400", "http://dd.domain.com 403", "http://ee.domain.com 403"]
for i in range(0, len(request)) :
word = request[i].split(" ")[1];
dictionary[word] = request[i].split(" ")[0];
print(dictionary)
Well for the query you asked we can solve it by using dictionary in python for desired output. Checkout the code snippet below :
dict = {}
req = ["http://aa.domain.com 200", "http://bb.domain.com 302", "http://cc.domain.com 400", "http://dd.domain.com 403", "http://ee.domain.com 403"]
for i in range(0, len(req)) :
term = req[i].split(" ")
while True :
if term[1] not in dict.keys():
dict.setdefault(term[1], [])
dict[term[1]].append(term[0])
break
else :
dict[term[1]].append(term[0])
break
print(dict)
It gives the desired output as :
{'200': ['http://aa.domain.com'], '400': ['http://cc.domain.com'], '403': ['http://dd.domain.com', 'http://ee.domain.com'], '302': ['http://bb.domain.com']}
Let me know if you are unable to understand it.
I am getting JIRA data using the following python code,
how do I store the response for more than one key (my example shows only one KEY but in general I get lot of data) and print only the values corresponding to total,key, customfield_12830, summary
import requests
import json
import logging
import datetime
import base64
import urllib
serverURL = 'https://jira-stability-tools.company.com/jira'
user = 'username'
password = 'password'
query = 'project = PROJECTNAME AND "Build Info" ~ BUILDNAME AND assignee=ASSIGNEENAME'
jql = '/rest/api/2/search?jql=%s' % urllib.quote(query)
response = requests.get(serverURL + jql,verify=False,auth=(user, password))
print response.json()
response.json() OUTPUT:-
http://pastebin.com/h8R4QMgB
From the the link you pasted to pastebin and from the json that I saw, its a you issues as list containing key, fields(which holds custom fields), self, id, expand.
You can simply iterate through this response and extract values for keys you want. You can go like.
data = response.json()
issues = data.get('issues', list())
x = list()
for issue in issues:
temp = {
'key': issue['key'],
'customfield': issue['fields']['customfield_12830'],
'total': issue['fields']['progress']['total']
}
x.append(temp)
print(x)
x is list of dictionaries containing the data for fields you mentioned. Let me know if I have been unclear somewhere or what I have given is not what you are looking for.
PS: It is always advisable to use dict.get('keyname', None) to get values as you can always put a default value if key is not found. For this solution I didn't do it as I just wanted to provide approach.
Update: In the comments you(OP) mentioned that it gives attributerror.Try this code
data = response.json()
issues = data.get('issues', list())
x = list()
for issue in issues:
temp = dict()
key = issue.get('key', None)
if key:
temp['key'] = key
fields = issue.get('fields', None)
if fields:
customfield = fields.get('customfield_12830', None)
temp['customfield'] = customfield
progress = fields.get('progress', None)
if progress:
total = progress.get('total', None)
temp['total'] = total
x.append(temp)
print(x)
I have a function which require to pass some ids as comma seperated to the url string.
My code:
def shipment(self, orderItemIds):
url = "https://api.flipkart.net/sellers/orders/shipments"
payload = {'orderItemsIds':orderItemIds}
return self.session.get(url, data=payload)
I need to pass id1, id2, id3 so that i get a link as:
https://api.flipkart.net/sellers/orders/shipments?orderItemsIds={id1,id2...}
I tried to pass it as a string. and as a list too. But it didn't worked.
oid = 'id1,id2,id3' # or
oid = ['id1',id2'','id3']
How to do it?
I passed an id as oiids = '230501592'.
On running response.url, it gives me:
https://api.flipkart.net/sellers/orders/shipments?orderItemsIds
It shows that parameter values are not passing in the url string.
I believe what you want is this (where orderItemIds is a list of strings):
def shipment(self, orderItemIds):
url = "https://api.flipkart.net/sellers/orders/shipments"
params = {'orderItemsIds': ','.join(orderItemIds)}
return self.session.get(url, params=params)
The differences from the version you have in the question are that orderItemIds is assumed to be a list of strings and is being joined with commas, and the data keyword is replaced with params, which is the right choice for a GET request.
You can easily send get parameters in python requests library-
data = {
'orderItemsIds': ['id1', 'id2', 'id3']
}
response = requests.get(url, data = data)
I am trying to update an atomic count counter with Python Boto 2.3.0, but can find no documentation for the operation.
It seems there is no direct interface, so I tried to go to "raw" updates using the layer1 interface, but I was unable to complete even a simple update.
I tried the following variations but all with no luck
dynoConn.update_item(INFLUENCER_DATA_TABLE,
{'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"},
{'new': {'Value': {'N':"1"}, 'Action': "ADD"}})
dynoConn.update_item('influencer_data',
{'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"},
{'new': {'S' :'hello'}})
dynoConn.update_item("influencer_data",
{"HashKeyElement": "9f08b4f5-d25a-4950-a948-0381c34aed1c"},
{"AttributesToPut" : {"new": {"S" :"hello"}}})
They all produce the same error:
File "/usr/local/lib/python2.6/dist-packages/boto-2.3.0-py2.6.egg/boto/dynamodb/layer1.py", line 164, in _retry_handler
data)
boto.exception.DynamoDBResponseError: DynamoDBResponseError: 400 Bad Request
{u'Message': u'Expected null', u'__type': u'com.amazon.coral.service#SerializationException'}
I also investigated the API docs here but they were pretty spartan.
I have done a lot of searching and fiddling, and the only thing I have left is to use the PHP API and dive into the code to find where it "formats" the JSON body, but that is a bit of a pain. Please save me from that pain!
Sorry, I misunderstood what you were looking for. You can accomplish this via layer2 although there is a small bug that needs to be addressed. Here's some Layer2 code:
>>> import boto
>>> c = boto.connect_dynamodb()
>>> t = c.get_table('counter')
>>> item = t.get_item('counter')
>>> item
{u'id': 'counter', u'n': 1}
>>> item.add_attribute('n', 20)
>>> item.save()
{u'ConsumedCapacityUnits': 1.0}
>>> item # Here's the bug, local Item is not updated
{u'id': 'counter', u'n': 1}
>>> item = t.get_item('counter') # Refetch item just to verify change occurred
>>> item
{u'id': 'counter', u'n': 21}
This results in the same over-the-wire request as you are performing in your Layer1 code, as shown by the following debug output.
2012-04-27 04:17:59,170 foo [DEBUG]:StringToSign:
POST
/
host:dynamodb.us-east-1.amazonaws.com
x-amz-date:Fri, 27 Apr 2012 11:17:59 GMT
x-amz-security- token:<removed> ==
x-amz-target:DynamoDB_20111205.UpdateItem
{"AttributeUpdates": {"n": {"Action": "ADD", "Value": {"N": "20"}}}, "TableName": "counter", "Key": {"HashKeyElement": {"S": "counter"}}}
If you want to avoid the initial GetItem call, you could do this instead:
>>> import boto
>>> c = boto.connect_dynamodb()
>>> t = c.get_table('counter')
>>> item = t.new_item('counter')
>>> item.add_attribute('n', 20)
>>> item.save()
{u'ConsumedCapacityUnits': 1.0}
Which will update the item if it already exists or create it if it doesn't yet exist.
For those looking for the answer I have found it.
First IMPORTANT NOTE, I am currently unaware of what is going on BUT for the moment, to get a layer1 instance I have had to do the following:
import boto
AWS_ACCESS_KEY=XXXXX
AWS_SECRET_KEY=YYYYY
dynoConn = boto.connect_dynamodb(AWS_ACCESS_KEY, AWS_SECRET_KEY)
dynoConnLayer1 = boto.dynamodb.layer1.Layer1(AWS_ACCESS_KEY, AWS_SECRET_KEY)
Essentially instantiating a layer2 FIRST and THEN a layer 1.
Maybe Im doing something stupid but at this point Im just happy to have it working....
I'll sort the details later. THEN...to actually do the atomic update call:
dynoConnLayer1.update_item("influencer_data",
{"HashKeyElement":{"S":"9f08b4f5-d25a-4950-a948-0381c34aed1c"}},
{"direct_influence":
{"Action":"ADD","Value":{"N":"20"}}
}
);
Note in the example above Dynamo will ADD 20 to what ever the current value is and this operation will be atomic meaning other operations happening at the "same time" will be correctly "scheduled" to happen after the new value has been established as +20 OR before this operation is executed. Either way the desired effect will be accomplished.
Be certain to do this on the instance of the layer1 connection as the layer2 will throw errors given it expects a different set of parameter types.
Thats all there is to it!!!! Just so folks know, I figured this out using the PHP SDK. Takes a very short time to install and set up AND THEN when you do a call, the debug data will actually show you the format of the HTTP request body so you will be able to copy/model your layer1 parameters after the example. Here is the code I used to do the atomic update in PHP:
<?php
// Instantiate the class
$dynamodb = new AmazonDynamoDB();
$update_response = $dynamodb->update_item(array(
'TableName' => 'influencer_data',
'Key' => array(
'HashKeyElement' => array(
AmazonDynamoDB::TYPE_STRING=> '9f08b4f5-d25a-4950-a948-0381c34aed1c'
)
),
'AttributeUpdates' => array(
'direct_influence' => array(
'Action' => AmazonDynamoDB::ACTION_ADD,
'Value' => array(
AmazonDynamoDB::TYPE_NUMBER => '20'
)
)
)
));
// status code 200 indicates success
print_r($update_response);
?>
Hopefully this will help other up until the Boto layer2 interface catches up...or someone simply figures out how to do it in level2 :-)
I'm not sure this is truly an atomic counter, since when you increment the value of 1, another call call could increment the number by 1, so that when you "get" the value, it is not the value that you would expect.
For instance, putting the code by garnaat, which is marked as the accepted answer, I see that when you put it in a thread, it does not work:
class ThreadClass(threading.Thread):
def run(self):
conn = boto.dynamodb.connect_to_region(aws_access_key_id=os.environ['AWS_ACCESS_KEY'], aws_secret_access_key=os.environ['AWS_SECRET_KEY'], region_name='us-east-1')
t = conn.get_table('zoo_keeper_ids')
item = t.new_item('counter')
item.add_attribute('n', 1)
r = item.save() #- Item has been atomically updated!
# Uh-Oh! The value may have changed by the time "get_item" is called!
item = t.get_item('counter')
self.counter = item['n']
logging.critical('Thread has counter: ' + str(self.counter))
tcount = 3
threads = []
for i in range(tcount):
threads.append(ThreadClass())
# Start running the threads:
for t in threads:
t.start()
# Wait for all threads to complete:
for t in threads:
t.join()
#- Now verify all threads have unique numbers:
results = set()
for t in threads:
results.add(t.counter)
print len(results)
print tcount
if len(results) != tcount:
print '***Error: All threads do not have unique values!'
else:
print 'Success! All threads have unique values!'
Note: If you want this to truly work, change the code to this:
def run(self):
conn = boto.dynamodb.connect_to_region(aws_access_key_id=os.environ['AWS_ACCESS_KEY'], aws_secret_access_key=os.environ['AWS_SECRET_KEY'], region_name='us-east-1')
t = conn.get_table('zoo_keeper_ids')
item = t.new_item('counter')
item.add_attribute('n', 1)
r = item.save(return_values='ALL_NEW') #- Item has been atomically updated, and you have the correct value without having to do a "get"!
self.counter = str(r['Attributes']['n'])
logging.critical('Thread has counter: ' + str(self.counter))
Hope this helps!
There is no high-level function in DynamoDB for atomic counters. However, you can implement an atomic counter using the conditional write feature. For example, let's say you a table with an string hash key called like this.
>>> import boto
>>> c = boto.connect_dynamodb()
>>> schema = s.create_schema('id', 's')
>>> counter_table = c.create_table('counter', schema, 5, 5)
You now write an item to that table that includes an attribute called 'n' whose value is zero.
>>> n = 0
>>> item = counter_table.new_item('counter', {'n': n})
>>> item.put()
Now, if I want to update the value of my counter, I would perform a conditional write operation that will bump the value of 'n' to 1 iff it's current value agrees with my idea of it's current value.
>>> n += 1
>>> item['n'] = n
>>> item.put(expected_value={'n': n-1})
This will set the value of 'n' in the item to 1 but only if the current value in the DynamoDB is zero. If the value was already incremented by someone else, the write would fail and I would then need to increment by local counter and try again.
This is kind of complicated but all of this could be wrapped up in some code to make it much simpler to use. I did a similar thing for SimpleDB that you can find here:
http://www.elastician.com/2010/02/stupid-boto-tricks-2-reliable-counters.html
I should probably try to update that example to use DynamoDB
You want to increment a value in dynamodb then you can achieve this by using:
import boto3
import json
import decimal
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
if o % 1 > 0:
return float(o)
else:
return int(o)
return super(DecimalEncoder, self).default(o)
ddb = boto3.resource('dynamodb')
def get_counter():
table = ddb.Table(TableName)
try:
response = table.update_item(
Key={
'haskey' : 'counterName'
},
UpdateExpression="set currentValue = currentValue + :val",
ExpressionAttributeValues={
':val': decimal.Decimal(1)
},
ReturnValues="UPDATED_NEW"
)
print("UpdateItem succeeded:")
except Exception as e:
raise e
print(response["Attributes"]["currentValue" ])
This implementetion needs a extra counter table that will just keep the last used value for you.