We Keep Coding

boto3 script does not tag ec2 resource and neither returns any error

I am using boto3 to tag EC2, and the script was working fine but out of blue somehow its no more tagging it. The peculiar thing is that its not returning any error. I mess the ARN up with dummy digits or no ARN at all just few digits, Still no error. Any Ideas what caused this?
# client = session.resource('ec2')
# This is the client that is passed into the function
def tag_ec2(client, cache, logger):
try:
resource_id = cache['ARN'].split(':')[-1]
response = client.Tag(
resource_id,
cache['key'],
cache['value']
)
print(response)
except Exception as e:
logger.error(f"Exception occurred while tagging {cache['resource']}: {e}.")
return str(e)
return None
This is the function that I use. No matter what ARN I give a real one or a fake one, It never throws any error or tag the instance with the real one either.
I've tried it on other EC2 resources like subnet and it works fine with them.

The documentation says, client.Tag represents a tag, you can think of it as accessing one of the tags in the instance as an object. It is not actually meant to 'create' tags. In that sense, it does make sense that it never returns an error b/c it may actually be your intention that you will first create the object and then do something with it (see resource's available actions here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#networkinterface )
You can use instance resource's create_tags method as follows (see ref here):
# client = session.resource('ec2')
instance = client.Instance(resource_id)
response= instance.create_tags(Tags=[{'Key': cache['key'], 'Value': cache['Value']}])
and check results with
instance.tags
There are other ways you can use service resource. Actually, it is probably confusing for people to see you are referring to ec2 service resource as client, so let's use correct terminology as follows:
resource = session.resource('ec2')
tags = [{'Key': cache['key'], 'Value': cache['Value']}]
response = resource.create_tags(Resources=[resource_id], Tags=tags)
Note that when you directly use Service Resource, you need to specify both Tags and resource id(s) as explained here.

python active directory module pyad max returned group members

How can I return all group members if it is over 1500, I'm using below code
Using execute_query member attribute in the group is empty
import pyad.adquery
q = pyad.adquery.ADQuery()
q.execute_query(
attributes = ["CN", "sAMAccountName", "sAMAccountType", "name", "title", "distinguishedName", "mail", "managedBy", "member", "memberOf"][::-1],
where_clause = "{0} = '{1}'".format(query_attribute,query_value),
base_dn = "DC=domain,DC=company,DC=net",
) --> Using execute_query member attribute is empty
result = list(q.get_results())[0]['member'] --> result is None
using pyad.from_cn only first 1500 users returned
f = pyad.pyad.from_cn('Group Name')
f.get_attribute('member') or f.get_members() --> both return only 1500 Users

This limit doesn't come from pyad, but AD itself. Active Directory will only give you 1500 rows from a multi-value attribute. To get the rest, you have to specifically ask for more.
I haven't tried this in Python, but to ask for the next 1500, you should be able to do something like this:
f.get_attribute('member;range=1500-*')
Try that and see if it works.
Looking at the Pyad source code, that might not actually work (because of the hasattr check, which might not remove the "range" part when checking if the attribute is valid). There is also an issue logged for this, which hasn't been replied to. And since the project is no longer maintained, it's unlikely to get fixed unless you fork it and fix it yourself (it should be as easy as removing that hasattr check).
But if that does happen to work, you will have to put that into a loop and keep going until you get an error, which means there are no more results. I have an example in C# here. You can translate the logic in the loop to Python.

'tags' parameter at Python Softlayer API call SoftLayer.VSManager.list_instances() not working as expected

I am implementing a cloud bursting system with Softlayer instances and Slurm. But I got a problem with Python Softlayer API.
When I try to get a list of some specific instances with the API call SoftLayer.VSManager.list_instances() I use the parameter 'tags', since I tagged the instances to classify them. But it does not work as expected.
It is supposed to find instances whose 'tagReferences' field matches with the value of the parameter 'tags' you passed in the API call.
However, I get a list with all the nodes whose 'tagReferences' field is not empty. Whatever is the value I pass as 'tags' parameter.
I have the following nodes:
hostname: 'node000' tags: 'slurm, node'
hostname: 'node005' tags: 'test'
I run this script:
import os
import SoftLayer
os.environ["SL_USERNAME"] = "***"
os.environ["SL_API_KEY"] = "******"
client = SoftLayer.Client()
mgr = SoftLayer.VSManager(client)
for vsi in mgr.list_instances(tags = 'slurm'):
print vsi['hostname']
This is the output I get:
node000
node005
I tried passing different values as 'tags' parameter (see below), but I always get the same result shown above, even with the last one.
Set of values passed as 'tags' parameter:
slurm, node
slurm
node
test
random
Did I miss anything?
I wrote a ticket to Softlayer support team but they believe my script should work and they assured me that the tags feature does work. Even they told me explicitly to come here to ask because they have no idea of what is happening.

According the documentation of the method that you are using, you need to send a list of tags, so change the string by a list like this:
client = SoftLayer.Client()
mgr = SoftLayer.VSManager(client)
for vsi in mgr.list_instances(tags = ['mytag']):
print (vsi['hostname'])
Regards

Suds write request missing datatype?

I am trying to communicate to a webservice using Suds, reading from the service works fine, however writing throws an error.
suds.WebFault: Server raised fault: 'The formatter threw an exception
while trying to deserialize the message: There was an error while
trying to deserialize parameter http://tempuri.org/:tagValues. The
InnerException message was 'Element Value from namespace
http://schemas.datacontract.org/2004/07/NOV.Api.Messages cannot have
child contents to be deserialized as an object. Please use XmlNode[]
to deserialize this pattern of XML.'. Please see InnerException for
more details.'
The XML produces does not seem to add the neccessary xsi:type="xsd:int"
Produced:
<ns1:TagValue>
<ns1:Quality>
<ns1:Id>1</ns1:Id>
<ns1:QualityData>Quality</ns1:QualityData>
</ns1:Quality>
<ns1:TagID>
<ns1:Id>0</ns1:Id>
<ns1:TagID>BitDepth</ns1:TagID>
</ns1:TagID>
<ns1:Value>23</ns1:Value>
</ns1:TagValue>
Expected:
<ns1:TagValue>
<ns1:Quality>
<ns1:Id>1</ns1:Id>
<ns1:QualityData>Quality</ns1:QualityData>
</ns1:Quality>
<ns1:TagID>
<ns1:Id>0</ns1:Id>
<ns1:TagID>BitDepth</ns1:TagID>
</ns1:TagID>
<ns1:Value xsi:type="xsd:int">23</ns1:Value>
</ns1:TagValue>
After searching around i figured to try the ImportDoctor to see if i could get in the xsi:type
I added
schema_url = 'http://schemas.xmlsoap.org/soap/encoding/'
schema_import = Import(schema_url)
schema_doctor = ImportDoctor(schema_import)
and doctor=schema_doctor in the Client ctor
This now gave me an additional prefix and a much extended list of Types
Prefixes (4)
ns0 = "http://schemas.datacontract.org/2004/07/NOV.Api.Messages"
ns1 = "http://schemas.microsoft.com/2003/10/Serialization/"
ns2 = "http://schemas.xmlsoap.org/soap/encoding/"
ns3 = "http://tempuri.org/"
I now have a ns2:int
I used the factory to create an object of type ns2:int, setting its value to 23
When sending this, i get the following XML:
<ns1:TagValue>
<ns1:Quality>
<ns1:Id>1</ns1:Id>
<ns1:QualityData>Quality</ns1:QualityData>
</ns1:Quality>
<ns1:TagID>
<ns1:Id>0</ns1:Id>
<ns1:TagID>BitDepth</ns1:TagID>
</ns1:TagID>
<ns1:Value xsi:type="ns2:int">23</ns1:Value>
</ns1:TagValue>
I now get the following exception when trying to send it:
suds.WebFault: Server raised fault: 'The formatter threw an exception
while trying to deserialize the message: There was an error while
trying to deserialize parameter http://tempuri.org/:tagValues. The
InnerException message was 'Error in line 1 position 651. Element
'http://schemas.datacontract.org/2004/07/NOV.Api.Messages:Value'
contains data from a type that maps to the name 'http://schemas.xm
lsoap.org/soap/encoding/:int'. The deserializer has no knowledge of
any type that maps to this name. Consider using a DataContractResolver
or add the type corresponding to 'int' to the list of known types -
for example, by using the KnownTypeAttribute attribute or by adding it
to the list of known types passed to DataContractSerializer.'. Please
see InnerException for more details.'
Seems slightly closer, but seems like there is some mess with namespaces?
Full XML produced:
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns3="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns0="http://tempuri.org/" xmlns:ns1="http://schemas.datacontract.org/2004/07/NOV.Api.Messages" xmlns:ns2="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<ns3:Body>
<ns0:WriteRealtimeValues>
<ns0:tagValues>
<ns1:TagValue>
<ns1:Quality>
<ns1:Id>1</ns1:Id>
<ns1:QualityData>Quality</ns1:QualityData>
</ns1:Quality>
<ns1:TagID>
<ns1:Id>0</ns1:Id>
<ns1:TagID>BitDepth</ns1:TagID>
</ns1:TagID>
<ns1:Value xsi:type="ns2:int">23</ns1:Value>
</ns1:TagValue>
</ns0:tagValues>
</ns0:WriteRealtimeValues>
</ns3:Body>
</SOAP-ENV:Envelope>
As reference, I create the client using the following code
credentials = dict(username='%s' % (username), password='%s' % password)
url= "http://%s:%s/TagValueWriteService?wsdl" % (ip,port)
self.transport = HttpAuthenticated(**credentials)
suds.client.Client.__init__(self,url, transport=self.transport, cache=None,doctor=schema_doctor)
There seem to be several similar issues here on stackoverflow, most of them mentioning the ImportDoctor in a similar manner as i tried. I am lacking some of the fundamental understanding of SOAP i suspect...

I managed to solve it, using the answer from Adding xsi:type and envelope namespace when using SUDS ( https://stackoverflow.com/a/10977734/696768 )
I am not sure this is the only possible solution, and to me it seems more of a hack than anything else, however it will work fine for my current scenario.
The solution i used, is making a plugin for the client, looking for the particular element that i need to be xsi:type="xsd:int", then adding these attributes to those elements.
The code i ended up using for reference (from the aforementioned stackoverflow question with minor adjustments):
from suds.plugin import MessagePlugin
from suds.sax.attribute import Attribute
class SoapFixer(MessagePlugin):
def marshalled(self, context):
# Alter the envelope so that the xsd namespace is allowed
context.envelope.nsprefixes['xsd'] = 'http://www.w3.org/2001/XMLSchema'
# Go through every node in the document and apply the fix function to patch up incompatible XML.
context.envelope.walk(self.fix_any_type_string)
def fix_any_type_string(self, element):
"""Used as a filter function with walk in order to fix errors.
If the element has a certain name, give it a xsi:type=xsd:int. Note that the nsprefix xsd must also
be added in to make this work."""
# Fix elements which have these names
fix_names = ['Value', 'anotherelementname']
if element.name in fix_names:
element.attributes.append(Attribute('xsi:type', 'xsd:int'))
plugin=SoapFixer()
Then I added plugins=[plugin] to the client ctor.
Example:
client = suds.client.Client("http://127.0.0.1:8099/TagValueWriteService?wsdl",plugins=[plugin])

This is not an 'answer' because this question is client-side. But I'm putting this here for the search engines for now.
The problem is the request message is a complex type.
My solution was on the server side. My service now accepts untyped elements in the request.
The server-side parsing of the request body must know about the Request schema. Once that happens, the server can typecheck and parse the request without the elements being typed by the client.
Specifically, My error came from a service implemented with Python ZSI module, and Zope.
Any cannot parse untyped element
Here, I got the hint on complex request objects:
http://pypi.python.org/pypi/z3c.soap/ (see ValidateEmailRequest)
Here, I got a crash course in ZSI:
Are there any working examples of Zolera SOAP Infrastructure (ZSI)?
And decent ZSI docs here: http://pywebsvcs.sourceforge.net/zsi.html#SECTION0071100000000000000000
To make ZSI happy, you just have to create a class that represents the Request message, and add a typecode to it. This is why you see a lot of services to "operation foo" and "fooRequest" and "fooResponse", so they can type the request and response objects as xml complex types.
for the example above, I would import something like this into the namespace where the soap request body is being parsed. You can get much more complicated, but this is really all that's necessary:
import ZSI
class WriteRealTimeValuesRequest(object):
tagValues = array() #of TagValue
WriteRealTimeValuesRequest.typecode = ZSI.TC.Struct(WriteRealTimeValuesRequest,
(ZSI.TC.Array("TagValue",
TagValue.typecode,
"tagValues"
),
),
"WriteRealTimeValuesRequest")
"Whats a tag value?"
class TagValue(object):
Quality = Quality
TagId = TagId
Value = Value
TagValue.typecode = ZSI.TC.Struct(TagValue,
(Quality.typecode,
TagId.typecode,
Value.typecode),
"TagValue")
What's a Quality?
class Quality(object):
Id = 0
QualityData = "I'm a string"
Quality.typecode = ZSI.TC.Struct(Quality,
(ZSI.TC.Integer("Id"), #this is the secret sauce
ZSI.TC.String("QualityData") #and here
),
"Quality")
And so on, until you've drilled down all the way to all the primitive types.

SQLAlchemy DetachedInstanceError with regular attribute (not a relation)

I just started using SQLAlchemy and get a DetachedInstanceError and can't find much information on this anywhere. I am using the instance outside a session, so it is natural that SQLAlchemy is unable to load any relations if they are not already loaded, however, the attribute I am accessing is not a relation, in fact this object has no relations at all. I found solutions such as eager loading, but I can't apply to this because this is not a relation. I even tried "touching" this attribute before closing the session, but it still doesn't prevent the exception. What could be causing this exception for a non-relational property even after it has been successfully accessed once before? Any help in debugging this issue is appreciated. I will meanwhile try to get a reproducible stand-alone scenario and update here.
Update: This is the actual exception message with a few stacks:
File "/home/hari/bin/lib/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/orm/attributes.py", line 159, in __get__
return self.impl.get(instance_state(instance), instance_dict(instance))
File "/home/hari/bin/lib/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/orm/attributes.py", line 377, in get
value = callable_(passive=passive)
File "/home/hari/bin/lib/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/orm/state.py", line 280, in __call__
self.manager.deferred_scalar_loader(self, toload)
File "/home/hari/bin/lib/python2.6/site-packages/SQLAlchemy-0.6.1-py2.6.egg/sqlalchemy/orm/mapper.py", line 2323, in _load_scalar_attributes
(state_str(state)))
DetachedInstanceError: Instance <ReportingJob at 0xa41cd8c> is not bound to a Session; attribute refresh operation cannot proceed
The partial model looks like this:
metadata = MetaData()
ModelBase = declarative_base(metadata=metadata)
class ReportingJob(ModelBase):
__tablename__ = 'reporting_job'
job_id = Column(BigInteger, Sequence('job_id_sequence'), primary_key=True)
client_id = Column(BigInteger, nullable=True)
And the field client_id is what is causing this exception with a usage like the below:
Query:
jobs = session \
.query(ReportingJob) \
.filter(ReportingJob.job_id == job_id) \
.all()
if jobs:
# FIXME(Hari): Workaround for the attribute getting lazy-loaded.
jobs[0].client_id
return jobs[0]
This is what triggers the exception later out of the session scope:
msg = msg + ", client_id: %s" % job.client_id

I found the root cause while trying to narrow down the code that caused the exception. I placed the same attribute access code at different places after session close and found that it definitely doesn't cause any issue immediately after the close of query session. It turns out the problem starts appearing after closing a fresh session that is opened to update the object. Once I understood that the state of the object is unusable after a session close, I was able to find this thread that discussed this same issue. Two solutions that come out of the thread are:
Keep a session open (which is obvious)
Specify expire_on_commit=False to sessionmaker().
The 3rd option is to manually set expire_on_commit to False on the session once it is created, something like: session.expire_on_commit = False. I verified that this solves my issue.

We were getting similar errors, even with expire_on_commit set to False. In the end it was actually caused by having two sessionmakers that were both getting used to make sessions in different requests. I don't really understand what was going on, but if you see this exception with expire_on_commit=False, make sure you don't have two sessionmakers initialized.

I had a similar problem with the DetachedInstanceError: Instance <> is not bound to a Session;
The situation was quite simple, I pass the session and the record to be updated to my function and it would merge the record and commit it to the database. In the first sample I would get the error, as I was lazy and thought that I could just return the merged object so my operating record would be updated (ie its is_modified value would be false). It did return the updated record and is_modified was now false but subsequent uses threw the error. I think this was compounded because of related child records but not entirely sure of that.
def EditStaff(self, session, record):
try:
r = session.merge(record)
session.commit()
return r
except:
return False
After much googling and reading about sessions etc, I realized that since I had captured the instance r before the commit and returned it, when that same record was sent back to this function for another edit/commit it had lost its session.
So to fix this I just query the database for the record just updated and return it to keep it in session and mark its is_modified value back to false.
def EditStaff(self, session, record):
try:
session.merge(record)
session.commit()
r = self.GetStaff(session, record)
return r
except:
return False
Setting the expire_on_commit=False also avoided the error as mentioned above, but I don't think it actually addresses the error, and could lead to many other issues IMO.

To throw my cause & solution into the ring, I use flask and flask-sqlalchemy to manage all my session stuff. This is fine when I'm doing things via the site, but when doing things via command line and scripts, you have to ensure that anything that's doing flask-y things has to do it with the flask context.
So, in my situation, I needed to get things from a database (using flask-sqlalchemy), then render them to templates (using flask's render_template), then email them (using flask-mail).
In code, what I'd done was something like,
def render_obj(db_obj):
with app.app_context():
return render_template('template_for_my_db_obj.html', db_obj=db_obj
def get_renders():
my_db_objs = MyDbObj.query.all()
renders = []
for day, _db_objs in itertools.groupby(my_db_objs, MyDbObj.get_date):
renders.extend(list(map(render_obj, _db_obj)))
return renders
def email_report():
renders = get_renders()
report = '\n'.join(renders)
with app.app_context():
mail.send(Message('Subject', ['me#me.com'], html=report))
(this is basically pseudocode, I was doing other things in the grouping section)
And when I was running, I'd get through the first _db_obj, but then I'd get the error on any run after.
The culprit? with app.app_context().
Basically it does a few things when you come out of that context, including kinda freshening up the db connections. One of the things that comes from that is getting rid of the last session that was around, which was the session that all the my_db_objs were associated with.
There's a few different options for solutions, but I went with a variant of,
def render_obj(db_obj):
return render_template('template_for_my_db_obj.html', db_obj=db_obj
def get_renders():
my_db_objs = MyDbObj.query.all()
renders = []
for day, _db_objs in itertools.groupby(my_db_objs, MyDbObj.get_date):
renders.extend(list(map(render_obj, _db_obj)))
return renders
def email_report():
with app.app_context():
renders = get_renders()
report = '\n'.join(renders)
mail.send(Message('Subject', ['me#me.com'], html=report))
Only 1 with app.app_context() which wraps them all. The main thing you need to do (if you've a setup like mine) is ensure any dB object you're using to be "inside" any app_context you're using. If you do what I did in the first iteration, all your dB objects will lose their session, ending in DetachedInstanceError like me.

My solution was a simple oversight;
I created an object, added and ,committed it to the db and after that I tried to access on of the original object attributes without refreshing session session.refresh(object)
user = UserFactory()
session.add(user)
session.commit()
# missing session.refresh(user) and causing the problem
print(user.name)

As for me (newbie), I made a mistake on the indent and close the session inside my loop, in which I loop each row, do some operation and commit each time.
So for those newbie like me, check your code before setting things like expire_on_commit=False, it may lead your to another trap.

My solution to this error was also a simple oversight, which I don't think any of the other answers cover.
My function is fetching object x, modifying it, then returning the original x, because I would like the older version.
Before committing and returning x, I was calling expunge_all, but it was "too late", as the object was already marked dirty.
The solution was simply to expunge the object as early as possible.
# pseudo code
x = session.fetch_x()
# adding the following line fixed it
session.expunge(x)
y = session.update(x)
return x

I have a similar problem in my current project and this fix works for me. Please check in your DB relationship for options lazy=True and change it to lazy='dynamic'.