We Keep Coding

AWS Lambda: Input values needs to be concatenated with error message

The query returns the expected result, the request in this forum is regarding the customized error message. Incase of failure the exception needs to stored along with the input values.
This is my first lambda code, please let me know any additional details.
Input
{"EventID":"1246", "DataflowID": "011010"}
Lambda Code (Nodejs), this was nodejs but suggestion in python are also appreciated.
var AWS = require('aws-sdk');
var mydocumentClient = new AWS.DynamoDB.DocumentClient();
exports.handler = function (event, context, callback) {
var params = {
TableName: 'TransactionLog',
KeyConditionExpression : 'EventID = :EventID and Status = :Status',
FilterExpression : 'EventID in (:EventID) and Status in (:Status)',
ExpressionAttributeValues: {":EventID": event.WorkflowDetail.EventID,":Status": "progress"},
ProjectionExpression: "EventID,DataflowID,Status"
};
mydocumentClient.scan(params, function (err, data){
if (err) {
callback(err, null);
}else{
callback(null, data);
}
}
)
}
Actual Example Error Message: Resource not found.
Expected Error Message: Resource not found "EventID":"1246", "DataflowID": "011010"
I tried using different options but no luck. please advise.
console.log(element.Title.S + " (" + element.Subtitle.S + ")");

Declare all the required events inside the parmas. I'm not sure this is the best solution but it meets my requirement. Thanks!
var params = {
...,
....,
Input: "{\"column1\" : \"" + 'type' + "\", \"column2\" : \"" + 'time' +\"}"
};
console.log(params.Input, err);
useful thread: Can execute a step function from a lambda but when I try to pass a value it fails saying input {}

Run cell programatically in JupyterLab

I want to write a code in a Jupyerlab notebook that runs other cells without the user doing anything. I found the solution:
from IPython.display import Javascript
Javascript('JupyterLab.notebook.execute_cells_below()')
But it doesn't seem to work in JupyterLab, it throws the error:
Javascript Error: notebook is not defined
Is there any similar way to do it using JupyterLab?

Maybe you can set a shortcut for 'run all cells above' and execute the following lines:
import keyboard
keyboard.press_and_release('your_shorcut_here')
Let's say that you've defined your shortcut as shift+s. Just put it in the code above as strig:
keyboard.press_and_release('shift+s')
For the latest jupyter notebook, (version 5) you can go to the 'help' tab in the top of the notebook and then select the option 'edit keyboard shortcuts' and add in your own customized shortcut.
Probably there is the same option for jupyter lab.

I found out how to execute cells programmatically when writing a JupyterLab extension. This is not exactly what you asked for but might help others searching for this topic. The code for JupyterLab seems to be more complex than for JupyterNotebook.
JupyterLab comes with the method
NotebookActions.run(notebook, sessionContext);
which executes the currently selected cells. There are a few other methods (e.g. "runAllBelow"), too. Also see
https://github.com/jupyterlab/jupyterlab/blob/master/packages/notebook/src/actions.tsx
Some example code (also see my JupyterLab extension for treezjs):
var { NotebookActions } = require("#jupyterlab/notebook");
var app = ... //injected with activate method of extension
var pythonCode = 'print("Hello World")';
this.__notebookPanel = this.__getFirstVisibleNotebookPanel(app);
executePythonCodeWithCell(pythonCode)
async executePythonCodeWithCell(pythonCode){
var self=this;
var NotebookActions = this.__dependencies['NotebookActions'];
return new Promise(async (resolve, reject) => {
if(self.__notebookPanel){
var notebook = self.__notebookPanel.content;
var notebookModel = notebook.model;
var sessionContext = self.__notebookPanel.sessionContext;
var options = { };
var cellModel = notebookModel.contentFactory.createCell('code',options);
cellModel.value.text = pythonCode;
const activeCellIndexBackup = notebook.activeCellIndex;
var newCellIndex = notebookModel.cells.length;
notebookModel.cells.insert(newCellIndex, cellModel);
notebook.activeCellIndex = newCellIndex;
var cell = notebook.activeCell;
try{
await NotebookActions.run(notebook, sessionContext);
} catch(error){
reject(error);
}
var htmlArray = [];
for(var output of cell.outputArea.node.children){
htmlArray.push(output.innerHTML);
}
await NotebookActions.deleteCells(notebook);
notebook.activeCellIndex = activeCellIndexBackup;
resolve(htmlArray);
}
});
}
__getFirstVisibleNotebookPanel(app){
var mainWidgets = app.shell.widgets('main');
var widget = mainWidgets.next();
while(widget){
var type = widget.constructor.name;
if(type == 'NotebookPanel'){ //other wigets might be of type DocumentWidget
if (widget.isVisible){
return widget;
}
}
widget = mainWidgets.next();
}
return null;
}
Part of my old code for JupyterNotebook:
this.__notebook = Jupyter.notebook;
async executePythonCodeWithCell(pythonCode){
var self=this;
return new Promise(function(resolve, reject) {
var cell = self.__notebook.insert_cell_below();
cell.element[0].style.display = 'none';
cell.set_text(pythonCode);
cell.events.on('finished_execute.CodeCell', (event, data) => self.__codeCellExecutionFinished(cell, data.cell, resolve));
try{
cell.execute();
} catch(error){
reject(error);
}
});
}
Related:
JupyterLab: Run all cells below
https://github.com/jupyterlab/jupyterlab/issues/6563
https://github.com/CDAT/jupyter-vcdat/blob/master/src/CellUtilities.ts
https://github.com/stefaneidelloth/treezjs/blob/master/jupyter_lab_extension/jupyterLabTerminal.js

Making 700 API calls together and merging the data using AWS Lambda

I have a task wherein I have to make 700 REST API calls. I have used a loop, but the run time of the loop is more than the timeout of AWS Lambda service. Is there any way I can make the calls concurrently and merge the results. I am using node JS, but a solution in python is also welcome
Here is a sample of the code I am running now:
HitBTCData = {}
for item in exchanges:
itemlist = item.split('-')
response = requests.get('https://min-
api.cryptocompare.com/data/histominute?
fsym='+itemlist[0]+'&tsym='+itemlist[1]+'&limit=2000&aggregate=1
&e=HitBTC').json()
if itemlist[0]+itemlist[1] not in HitBTCData:
HitBTCData[itemlist[0]+itemlist[1]] = []
HitBTCData[itemlist[0]+itemlist[1]].append(response['Data'])
while response['Type'] != 1:
time = response['TimeFrom']
response = requests.get('https://min-
api.cryptocompare.com/data/histominute?
fsym='+itemlist[0]+'&tsym='+itemlist[1]+
'&limit=2000&aggregate=1&toTs='+str(time)+'&e=HitBTC').json()
if response['Type'] != 1:
HitBTCData[itemlist[0]+itemlist[1]]
.append(response['Data'])
HitBTCData[itemlist[0]+itemlist[1]] =
HitBTCData[itemlist[0]+itemlist[1]][::-1]

Use a loop in Node.js and the calls will run concurrently, but to make sure all of your calls are processed before exiting the lambda, you need to have some form of counter to make sure all of the queries have ended.
example:
var request = require('request');
var results = [];
var runningQueries = 0;
exchanges.forEach(function (item) {
++runningQueries;
callAPI (item, function (err, data) {
if (err) {
results.push(err)
--runningQueries;
} else {
results.push(data)
--runningQueries;
}
if (runningQueries == 0) {
// all call results have been collated, so do something
}
})
})
You'll need to build your own callAPI function. Also, I'm being a bit lazy with what I'm pushing to the array here, but it's just to give you a framework.

You could very easily do this with Promise.all():
const urls = [...];
const requestFunction = async (url) => {
// all your ifs go here
let data = await request(url);
// transform/sanitize your data
return data;
}
const allResults = await Promise.all(urls.map(requestFunction));
// you now have an array of results
The only downside to Promise.all() is that if any of the promises rejects, everything fails. So I would suggest catching errors and resolving with null or something similar.
You could do a for loop to generate your array of URLs, new Array(700).map(...), or since it looks like you already have some kind of array, just .map() that directly and do more transforms inside the request function.

Need to convert python dict in proto message format for gRPC

I have a format like this and want to convert my python dict to this proto format
proto code:
message template{
message NicSettings{
string network_adapter = 1;
string network_name = 2;
}
message NetConfig{
bool keep_mac_address =1;
bool same_as_source = 2;
repeated NicSettings nic_settings = 3;
}
NetworkConfig network_config = 3;
}
python dict:
template:
{ keepMacAddress: true,
sameAsSource : false,
nicSettings: [ {networkAdapter: "ethernet0",
networkName: "Calpal1"
} ,
{networkAdapter: "ethernet1",
networkName: "Calpal2"
}
]
}
How do I convert this to a proto message to pass it to gRPC.

It's not entirely clear that your .proto is correct (or are there typos in it?) but this looks like it should be something along the lines of
my_template_message = my_message_module_pb2.template(
keep_mac_address=my_dictionary['keepMacAddress'],
same_as_source=my_dictionary['sameAsSource'],
nic_settings=tuple(
my_message_module_pb2.template.NicSettings(
network_adapter=my_subdictionary['networkAdapter'],
network_name=my_subdictionary['networkName'])
for my_subdictionary in my_dictionary['nicSettings']
)
)
. It's also a little odd in your .proto content that you've nested two message definitions inside another message definition - that doesn't look necessary at all.

extract data from gmail add to spreadsheet- Google apps script

I have searched, copied and modified code, and tried to break down what others have done and I still can't get this right.
I have email receipts for an ecommerce webiste, where I am trying to harvest particular details from each email and save to a spreadsheet with a script.
Here is the entire script as I have now.
function menu(e) {
var ui = SpreadsheetApp.getUi();
ui.createMenu('programs')
.addItem('parse mail', 'grabreceipt')
.addToUi();
}
function grabreceipt() {
var ss = SpreadsheetApp.getActiveSheet();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getSheetByName("Sheet1");
var threads = GmailApp.search("(subject:order receipt) and (after:2016/12/01)");
var a=[];
for (var i = 0; i<threads.length; i++)
{
var messages = threads[i].getMessages();
for (var j=0; j<messages.length; j++)
{
var messages = GmailApp.getMessagesForThread(threads[i]);
for (var j = 0; j < messages.length; j++) {
a[j]=parseMail(messages[j].getPlainBody());
}
}
var nextRow=s.getDataRange().getLastRow()+1;
var numRows=a.length;
var numCols=a[0].length;
s.getRange(nextRow,1,numRows,numCols).setValues(a);
}
function parseMail(body) {
var a=[];
var keystr="Order #,Subtotal:,Shipping:,Total:";
var keys=keystr.split(",");
var i,p,r;
for (i in keys) {
//p=keys[i]+(/-?\d+(,\d+)*(\.\d+(e\d+)?)?/);
p=keys[i]+"[\r\n]*([^\r^\n]*)[\r\n]";
//p=keys[i]+"[\$]?[\d]+[\.]?[\d]+$";
r=new RegExp(p,"m");
try {a[i]=body.match(p)[1];}
catch (err) {a[i]="no match";}
}
return a;
}
}
So the email data to pluck from comes as text only like this:
Order #89076
(body content, omitted)
Subtotal: $528.31
Shipping: $42.66 via Priority Mail®
Payment Method: Check Payment- Money order
Total: $570.97
Note: mywebsite order 456. Customer asked about this and that... etc.
The original code regex was designed to grab content, following the keystr values which were easily found on their own line. So this made sense:
p=keys[i]+"[\r\n]*([^\r^\n]*)[\r\n]";
This works okay, but results where the lines include more data that follows as in line Shipping: $42.66 via Priority Mail®.
My data is more blended, where I only wish to take numbers, or numbers and decimals. So I have this instead which validates on regex101.com
p=keys[i]+"[\$]?[\d]+[\.]?\d+$";
The expression only, [\$]?[\d]+[.]?\d+$ works great but I still get "no match" for each row.
Additionally, within this search there are 22 threads returned, and it populates 39 rows in the spreadsheet. I can not figure out why 39?

The reason for your regex not working like it should is because you are not escaping the "\" in the string you use to create the regex
So a regex like this
"\s?\$?(\d+\.?\d+)"
needs to be escaped like so:
"\\s?\\$?(\\d+\\.?\\d+)"
The below code is just modified from your parseEmail() to work as a snippet here. If you copy this to your app script code delete document.getElementById() lines.
Your can try your example in the snippet below it will only give you the numbers.
function parseMail(body) {
if(body == "" || body == undefined){
var body = document.getElementById("input").value
}
var a=[];
var keystr="Order #,Subtotal:,Shipping:,Total:";
var keys=keystr.split(",");
var i,p,r;
for (i in keys) {
p=keys[i]+"\\s?\\$?(\\d+\\.?\\d+)";
r=new RegExp(p,"m");
try {a[i]=body.match(p)[1];}
catch (err) {a[i]="no match";}
}
document.getElementById("output").innerHTML = a.join(";")
return a;
}
<textarea id ="input"></textarea>
<div id= "output"></div>
<input type = "button" value = "Parse" onclick = "parseMail()">
Hope that helps