We Keep Coding

Groovy Script AEM Asset Reference Update

So I'm working with AEM and am attempting to create a script that grabs all pages under a specific path and updates the image reference on the page from a list of assets under a curtain path.
Both of my select query's aren't returning the specific pages and assets I need.
I'm also getting an error that my queries are searching over 100000 Nodes
How can i resolve this error and query my resources better?
import com.day.cq.wcm.api.Page
import javax.jcr.query.Query
import javax.jcr.query.QueryManager
import org.apache.sling.api.resource.ModifiableValueMap
import groovy.transform.Field
static void main(String[] args)
{
String[] assetNodes
String[] pageNodes
String pagePath ="/content/we-retail/us/en"
String pageResourceType = "weretail/components/structure/page"
String assetPath ="/content/dam/microsoft/internal/en"
String assetQuery = "b1048291-23fa-422a-a7c4-9ea4bae0effc"
boolean isAsset = true;
pageNodes = GetResourcePath(pagePath, pageResourceType);
assetNodes = GetRosourceAsset(assetPath, assetQuery);
InputAssetsOnPage(pageNodes,assetNodes);
}
//Find the Node paths for all Pages to modify
//Narrow down to image component
def GetResourcePath(String rootPath,String queryParam)
{
int i = 0
def String[] allNodes = new String[500]
Page page = getPage(rootPath)
def queryManager = session.workspace.queryManager;
def param= queryParam;
def statement = 'select * from nt:base where jcr:path like \''+page.path+'/%\' and sling:resourceType = \'' + param + '\'';
Query query=queryManager.createQuery(statement, 'sql');
final def result = query.execute()
println "Total pages found = " + result.nodes.size();
NodeIterator nodeIterator = result.getNodes();
while(nodeIterator.hasNext())
{
def hitNode = nodeIterator.nextNode();
allNodes[i] = hitNode.getPath();
i++;
}
println allNodes
return allNodes;
}
//Find all assets paths to add to page
def GetRosourceAsset(String rootPath,String queryParam)
{
int i = 0
def String[] allNodes = new String[500]
Page page = getPage(rootPath)
def queryManager = session.workspace.queryManager;
def param= queryParam;
def statement = 'select * from nt:base where jcr:path like \''+rootPath+'/%\'';
Query query=queryManager.createQuery(statement, 'sql');
final def result = query.execute()
println "Total Assets found = " + result.nodes.size();
NodeIterator nodeIterator = result.getNodes();
while(nodeIterator.hasNext())
{
def hitNode = nodeIterator.nextNode();
allNodes[i] = hitNode.getPath();
i++;
}
println allNodes
return allNodes;
}
//Modify image component property with unique asset path
void InputAssetsOnPage(String[]pageRefrence, String[]assetRefrences)
{
String[] nodes= pageRefrence;
String[] assetNodes = assetRefrences;
nodes.eachWithIndex { self,i->
javax.jcr.Node node=getNode(nodes[i])
ModifiableValueMap mVMap=resourceResolver.resolve(node.path).adaptTo(ModifiableValueMap.class);
mVMap.put("fileRefrence", assetNodes[i]);
println "Property modified to "+node.path;
println "Dry Run "+data.dryRun;
if(!data.dryRun) {
session.save();
}
}
}
//Save session

For mass-updates it is very likely to target too many nodes. You have to try some approaches, to either get just under the limit - or change you approach.
First select pages from cq:PageContent, instead from nt:base. The query-indexes are organised by jcr:primaryType. The nt:base index contains everything. So there are much more nodes.
Second use SQL-2 and ISDESCENDANTNODE() instead of a like operator. I don't expect the like-operator to be so specific. But, if you query almost all pages anyway - it won't help much.
Third iterate over parts of you page-tree. Then the remaining subtree is much small, can can be queried.
For a mass-update, which probably touches more than 15% of your pages, then just iterate over all pages. Don't forget to commit from time to time. (e.g. every 100 changes)
Code sample to iterate a page-tree
def rootPage = getPage("/content/uac/glooly/es")
rootPage.recurse() { page ->
println(page.path)
fixAssetsOnPage(page)
}
You can iterate over the first n levels over the content tree, and then deep dive with a query. So you could have less than 100.000 image components
Code sample to iterate over first 2 levels, and then query:
def rootPage = getPage("/content/we-retail/us/en")
rootPage.iterator().each { firstLevelPage ->
firstLevelPage.iterator().each { secondLevelPage ->
println(secondLevelPage.path)
queryAndFixAssets(secondLevelPage)
save()
}
}
Fourth create an oak:index for you query. Especially if you could use it for later too. But in your case there is an existing index for sling:resourceType, which should be could enough. You have just too many hits.
PS: You probably don't need to query pages. Just query the image components in a small enough subtree. But we can't see that from you code sample.

Return all alarms from AWS instead of only 50 using describe_alarms on Python

How can I print all the alarms names, instead of only 50, when using the function describe_alarms?
Code, using Python:
conn = boto.connect_cloudwatch()
alarms = conn.describe_alarms()
for item in alarms:
print item.name
Thanks.

Even though I am a bit late to the party, here is my solution (in Java). You have to get the next token and keep on asking for the Result in a loop until there is no next token, so it behaves like a pagination on a website
String nextToken = null;
List<MetricAlarm> metricAlarms = new ArrayList<>();
for (int i = 0; i < 100; i++) {
DescribeAlarmsRequest describeAlarmsRequest = new DescribeAlarmsRequest();
describeAlarmsRequest.setNextToken(nextToken);
describeAlarmsRequest.setMaxRecords(100);
DescribeAlarmsResult describeAlarmsResult = getClient().describeAlarms(describeAlarmsRequest);
List<MetricAlarm> metricAlarmsTmp = describeAlarmsResult.getMetricAlarms();
metricAlarms.addAll(metricAlarmsTmp);
nextToken = describeAlarmsResult.getNextToken();
logger.info("nextToken: {}", nextToken);
if (nextToken == null) {
break;
}
}
logger.info("metricAlarms size: {}", metricAlarms.size());
Of course there is room for improvement, e.g. create a while loop instead of a for loop.
UPDATE:
Here my refined version
String nextToken = null;
List<MetricAlarm> metricAlarms = new ArrayList<>();
while (nextToken != null || metricAlarms.size() == 0) {
DescribeAlarmsRequest describeAlarmsRequest = new DescribeAlarmsRequest().withNextToken(nextToken).withMaxRecords(100); // create the request
DescribeAlarmsResult describeAlarmsResult = getClient().describeAlarms(describeAlarmsRequest); // get the result
metricAlarms.addAll(describeAlarmsResult.getMetricAlarms()); // add new alarms to our list
nextToken = describeAlarmsResult.getNextToken(); // check if we have a nextToken
if (nextToken == null && cachedMetricAlarms.size() == 0) { // if we have no alarm in AWS we get inside the loop but we would never exit -> intercept that
break;
}
}
logger.info("metricAlarms size: {}", metricAlarms.size());

By default it returns 50. If you want more, set max_records=value and try.
Due to underlying AWS API implementation restriction, it will return a maximum of 100 alarms. Don't know if it is fixed now.
conn.describe_alarms(max_records=100)
Help on method describe_alarms in module boto.ec2.cloudwatch:
describe_alarms(self, action_prefix=None, alarm_name_prefix=None,
alarm_names=None, max_records=None, state_value=None, next_token=None)
:type max_records: int
:param max_records: The maximum number of alarm descriptions
to retrieve.

Here's a complete example of how to paginate through the records in order to guarantee that you retrieve all of the records rather then being limited by the max records on the Cloudwatch Alarms API:
alarmMaxRecords = 10
response = client.describe_alarms(
AlarmNamePrefix=prefix,
MaxRecords=alarmMaxRecords
)
alarmsItems = []
while response:
alarmsItems += response['MetricAlarms']
response = client.describe_alarms(AlarmNamePrefix=prefix, MaxRecords=alarmMaxRecords, NextToken=response['NextToken']) if 'NextToken' in response else None
for alarm in alarmsItems:
# Do something with the alarm
print(response['MetricAlarms'])
The above will retrieve 10 records at a time, but can be anything up to 100.
Or more simply using the paginate method provided by boto3:
import boto3
# Create CloudWatch client
cloudwatch = boto3.client('cloudwatch')
# List alarms of insufficient data through the pagination interface
paginator = cloudwatch.get_paginator('describe_alarms')
for response in paginator.paginate(AlarmNamePrefix=prefix, MaxRecords=alarmMaxRecords):
# Do something with the alarm
print(response['MetricAlarms'])

Looping over Protocol Buffers attributes in Python

I would like help with recursively looping over all attributes/sub objects contained in a protocol buffers message, assuming that we do not know the names of them, or how many there are.
As an example, take the following .proto file from the tutorial on the google website:
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
and to use it...:
person = tutorial.Person()
person.id = 1234
person.name = "John Doe"
person.email = "jdoe#example.com"
phone = person.phone.add()
phone.number = "555-4321"
phone.type = tutorial.Person.HOME
Given Person, How do I then access both the name of the attribute and its value for each element: person.id, person.name, person.email, person.phone.number, person.phone.type?
I have tried the following, however it doesn't seem to recurs into person.phone.number or person.phone.type.
object_of_interest = Person
while( hasattr(object_of_interest, "_fields") ):
for obj in object_of_interest._fields:
# Do_something_with_object(obj) # eg print obj.name
object_of_interest = obj
I have tried using obj.DESCRIPTOR.fields_by_name.keys to access the sub elements, but these are the string representations of the sub objects, not the objects themselves.
obj.name gives me the attribute of the name, but im not sure how to actually get the value of that attribute, eg obj.name may give me 'name', but how do i get 'john doe' out of it?

I'm not super familiar with protobufs, so there may well be an easier way or api for this kind of thing. However, below shows an example of how you could iterate/introspect and objects fields and print them out. Hopefully enough to get you going in the right direction at least...
import addressbook_pb2 as addressbook
person = addressbook.Person(id=1234, name="John Doe", email="foo#example.com")
person.phone.add(number="1234567890")
def dump_object(obj):
for descriptor in obj.DESCRIPTOR.fields:
value = getattr(obj, descriptor.name)
if descriptor.type == descriptor.TYPE_MESSAGE:
if descriptor.label == descriptor.LABEL_REPEATED:
map(dump_object, value)
else:
dump_object(value)
elif descriptor.type == descriptor.TYPE_ENUM:
enum_name = descriptor.enum_type.values[value].name
print "%s: %s" % (descriptor.full_name, enum_name)
else:
print "%s: %s" % (descriptor.full_name, value)
dump_object(person)
which outputs
tutorial.Person.name: John Doe
tutorial.Person.id: 1234
tutorial.Person.email: foo#example.com
tutorial.Person.PhoneNumber.number: 1234567890
tutorial.Person.PhoneNumber.type: HOME

while loop for deleting data in datastore

I have attempted to clean up and revise code in an answer here for my needs where I only want to delete from the Model Reservations for data records prior to the date expressed in the get as yy,mm,dd.
If I am correctly anticipating the action of cleanTable/2012/10/5 against the routing ('/cleanTable/([\d]+)/([\d]+)/([\d]+)', CleanTable) then my code would only delete at most 50 (10*nlimit) data records.
Btw, the author of the original code (who likely no longer subscribes to SO), claimed his main trick for accomplishing this code was "to include redirect in html instead of using self.redirect".
I am unfamiliar with raise Exception and the like, but my instinct would be to add a raise Exception or raise StopIteration to the for loop after it is made into a while loop. But it is not clear to me whether raising an StopIteration exception actually causes iteration to stop or if more is needed. Also, I don't know how to revise so the html ends smoothly upon early exit.
class CleanTable(BaseHandler):
def get(self, yy,mm,dd):
nlimit=5
iyy=int(yy)
imm=int(mm)
idd=int(dd)
param=date(iyy,imm,idd)
q=Reservations.all(keys_only=True)
q.filter("date < ", dt(iyy,imm,idd))
results = q.fetch(nlimit)
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write("""
<html>
<meta HTTP-EQUIV="REFRESH" content="url=http://yourapp.appspot.com/cleanTable">
<body>""")
try:
for i in range(10):
db.delete(results)
results = q.fetch(nlimit, len(results))
for r in results:
logging.info("r.name: %s" % r.name)
self.response.out.write("<p> "+str(nlimit)+" removed</p>")
self.response.out.write("""
</body>
</html>""")
except Exception, inst:
logging.info("inst: %s" % inst)
self.response.out.write(str(inst))

This is not the best approach to clean your models. A better approach would be to get all the keys of your entities and create Task Queues. Each queue will get a batch of keys for the entities that need to be modified.
Another approach would also be to create a cron job that will query for the x number of oldest modified entities, fix them and then store them back.
Finally, if your number of entities is so huge, you could also consider the use of Backends.
Hope this helps.

Here is my update routine and it has converted 500.000 entities. Be sure to run it on a backend instance (You can target a Queue to a backend instance). Notice that I am using a cursor, thats the only way you can consistently iterate through data (Never use offset!).
Queue queue = QueueFactory.getQueue("grinderQueue");
queue.add(TaskOptions.Builder.withPayload(new DeferredTask() { //lets generate
private static final long serialVersionUID = 1L;
#Override
public void run() {
String cursor = null;
boolean done = false;
Date now = new Date(1346763868L * 1000L); // 09/04/2012
while(!done) {
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Query query = new Query("Venue");
query.setFilter(new FilterPredicate("timeOfLastUpdate", Query.FilterOperator.LESS_THAN,now));
PreparedQuery pq = datastore.prepare(query);
FetchOptions fetchOptions = FetchOptions.Builder.withLimit(1000);
if(cursor != null)
fetchOptions.startCursor(Cursor.fromWebSafeString(cursor));
QueryResultList<Entity> results = pq.asQueryResultList(fetchOptions);
List<Entity> updates = new ArrayList<Entity>();
List<Entity> oldVenueUpdates = new ArrayList<Entity>();
int tuples = 0;
for(Entity en : results) {
tuples++;
try {
if(en.getProperty(Venue.VENUE_KEY) == null)
continue;
Entity newVenue = new Entity("CPVenue",(String)en.getProperty(Venue.VENUE_KEY));
newVenue.setPropertiesFrom(en);
newVenue.removeProperty("timeOfLastVenueScoreCalculation");
newVenue.removeProperty("actionsSinceLastVenueScoreCalculation");
newVenue.removeProperty("venueImageUrl");
newVenue.removeProperty("foursquareId");
newVenue.setProperty("geoCell", GeoCellCalculator.calcCellId(Double.valueOf((String)en.getProperty("lng")), Double.valueOf((String)en.getProperty("lat")),8));
newVenue.setProperty(Venue.TIME_SINCE_LAST_UPDATE, new Date());
updates.add(newVenue);
Venue v = new Venue(newVenue);
//Set timestamp on Venue
en.setProperty("timeOfLastUpdate", now);
oldVenueUpdates.add(en);
}catch(Exception e) {
logger.log(Level.WARNING,"",e);
}
}
done = tuples == 0;
tuples = 0;
if(results.getCursor() != null)
cursor = results.getCursor().toWebSafeString();
else
done = true;
System.out.println("Venue Conversion LOOP updates.. " + updates.size() + " cursor " + cursor);
datastore.put(updates);
datastore.put(oldVenueUpdates);
}
System.out.println("Venue Conversion DONE");
}}));

Google app engine: filter by ID

I am lost somehow, I want to do something like below which filter by the ID.
id = 1000
query = Customers.all()
query.filter('ID =', id)
or
query = db.GqlQuery("select * from Customers where ID = %s" % id)
What is the correct method to filter by ID?

both are correct and even Customers.gql("WHERE ID = :1", id);
Edit: If ID is the automatically created id property you should use Customers.get_by_id()

you need to use Customers.get_by_id(id)

I had this same problem, and it turned out that I was just working too hard. The answer lies in getObjectById(). If this works for you, please go to my very-similar S.O. question and give Gordon's answer a vote-up, since he's the one who showed me this.
Player result = null;
if (playerKey == null)
{
log.log(Level.WARNING, "Tried to find player with null key.");
}
else
{
PersistenceManager pm = assassin.PMF.get().getPersistenceManager();
try {
result = (Player) pm.getObjectById(Player.class, playerKey);
} catch (javax.jdo.JDOObjectNotFoundException notFound) {
// Player not found; we will return null.
result = null;
}
pm.close();
}
return result;