We Keep Coding

Creating Custom varbinds dictionary using pysnmp

I am using sendVarbinds from pysnmp to send traps with custom MIBs
e.g.
def sendTrap(self, trapName, trapObjects):
mibViewController = view.MibViewController(self.snmpEngine.getMibBuilder())
ntfOrg = ntforg.NotificationOriginator()
ntfOrg.snmpContext = self.snmpContext
ntfOrg.sendVarBinds(
self.snmpEngine,
'my-notification', # notification targets
None, '', # contextEngineId, contextName
rfc1902.NotificationType(
trapName,
objects=trapObjects # <-- how to construct this?
).resolveWithMib(mibViewController)
)
The NotificationType requires a dictionary of varbinds , having object key value pair .
I would like to populate the object dictionary key and value from a json object .Right now I am doing it like this for each trap.
def function1():
trapName = rfc1902.ObjectIdentity('MY-MIB', 'ServerPart')
trapObjects = {
('MY-MIB', 'ServerIP'): event.get_ServerIP(),
('MY-MIB', 'ServerPartName'): event.get_ServerPartName(),
('MY-MIB', 'ServerState'): event.get_ServerState()
}
def function2():
trapName = rfc1902.ObjectIdentity('MY-MIB', 'ServerMemory')
trapObjects = {
('MY-MIB', 'ServerIP'): event.get_ServerIP(),
('MY-MIB', 'ServerMemoryAv'): event.get_ServerMemoryAv()
}
But I want to have a single function to create the traps without checking the varbinds and fill them using the JSON object? Say I run a loop on dictionary items().
My flow is like this.
Client --- event JSON object ----> Trap daemon ---- traps -----> Manager
This is an example of the JSON object which comes from client
my_event('ServerPart', '10.22.1.1', '44', '1', host, port)
my_event('ServerMemory', '10.22.1.1', '100MB', host, port)
Is there any example which can be followed?

Each notification type requires its own set of managed objects (trapOpjects) to be included with the notification. I suppose the mapping of event name -> notification type is 1-to-1 and you can re-constract event field getter from field name.
Then what's remaining is to map event and its parameters to notification type and managed objects:
eventParamsToManagedObjectsMap = {
'ServerPart': { # <- notification type
'ServerIP': 'get_ServerIP', # <- trap objects
# e.g. to populate ServerIP,
# call get_serverIP()
# ...
}
...
}
Finally, to build such map you could try something along these lines:
notificationType = 'ServerMemory' # on input you should have all
# event or notification types
trapObjectIdentity = rfc1902.ObjectIdentity('MY-MIB', notificationType)
trapObjectIdentiry.resolveWithMib(mibViewController)
trapObjectNames = trapObjectIdentity.getMibNode().getObjects()
eventParamsToManagedObjectsMap = defaultdict(dict)
for mibName, mibObjectName in trapObjectNames:
eventParamsToManagedObjectsMap[notificationType][mibObjectName] = 'get_' + mibObjectName
Once you have this map, you should be able to build notifications out of events uniformly within a single function.

I guess you can try something like this (untested):
def get_notification(trapName, **kwargs):
return rfc1902.NotificationType(
trapName,
objects={('MY-MIB', k): v for k, v in kwargs.items()}
}
Assuming json_data is already a Python dict (if it is a string call json.loads(json_data) in order to parse it) you can call it like this:
ntfOrg.sendVarBinds(
self.snmpEngine,
'my-notification', # notification targets
None, '', # contextEngineId, contextName
get_notification(**json_data).resolveWithMib(mibViewController)
)

How do I pass data from one operator to another

I made a custom airflow operator, this operator takes an input and the output of this operator is on XCOM.
What I want to achieve is to call the operator with some defined input, parse the output as Python callable inside the Branch Operator and then pass the parsed output to another task that calls the same operator tree:
CustomOperator_Task1 = CustomOperator(
data={
'type': 'custom',
'date': '2017-11-12'
},
task_id='CustomOperator_Task1',
dag=dag)
data = {}
def checkOutput(**kwargs):
result = kwargs['ti'].xcom_pull(task_ids='CustomOperator_Task1')
if result.success = True:
data = result.data
return "CustomOperator_Task2"
return "Failure"
BranchOperator_Task = BranchPythonOperator(
task_id='BranchOperator_Task ',
dag=dag,
python_callable=checkOutput,
provide_context=True,
trigger_rule="all_done")
CustomOperator_Task2 = CustomOperator(
data= data,
task_id='CustomOperator_Task2',
dag=dag)
CustomOperator_Task1 >> BranchOperator_Task >> CustomOperator_Task2
In task CustomOperator_Task2 I would want to pass the parsed data from BranchOperator_Task. Right now it is always empty {}
What is the best way to do that?

I see your issue now. Setting the data variable like you are won't work because of how Airflow works. An entirely different process will be running the next task, so it won't have the context of what data was set to.
Instead, BranchOperator_Task has to push the parsed output into another XCom so CustomOperator_Task2 can explicitly fetch it.
def checkOutput(**kwargs):
ti = kwargs['ti']
result = ti.xcom_pull(task_ids='CustomOperator_Task1')
if result.success:
ti.xcom_push(key='data', value=data)
return "CustomOperator_Task2"
return "Failure"
BranchOperator_Task = BranchPythonOperator(
...)
CustomOperator_Task2 = CustomOperator(
data_xcom_task_id=BranchOperator_Task.task_id,
data_xcom_key='data',
task_id='CustomOperator_Task2',
dag=dag)
Then your operator might look something like this.
class CustomOperator(BaseOperator):
#apply_defaults
def __init__(self, data_xcom_task_id, data_xcom_key, *args, **kwargs):
self.data_xcom_task_id = data_xcom_task_id
self.data_xcom_key = data_xcom_key
def execute(self, context):
data = context['ti'].xcom_pull(task_ids=self.data_xcom_task_id, key=self.data_xcom_key)
...
Parameters may not be required if you just want to hardcode them. It depends on your use case.

As your comment suggests, the return value from your custom operator is None, therefore your xcom_pull should expect to be empty.
Please use xcom_push explicitly, as the default behavior of airflow could change over time.

Python Django equals in function arguments. Setting the variable dynamically

I think this is called "Positional arguments" or the opposite or keyword arguments. I have written a script in Python Django and rest framework that has to take in parameters from the user and feed it to the Amazon API. Here is the code;
page = request.query_params.get('page');
search = request.query_params.get('search');
search_field = request.query_params.get('search_field');
search_index = request.query_params.get('search_index');
response = amazon.ItemSearch(Keywords=search, SearchIndex=search_index, ResponseGroup="ItemAttributes,Offers,Images,EditorialReview,Reviews")
In this code the text Keywords=search varies. That is. It could be like this Actor=search, Title=search, or Publisher=search I am not sure how to make the Keywords part dynamic such that it changes to the user's input such as Actor, Title, or Publisher

I think you should build up a dictionary of your kwargs and then use the ** syntax to expand it for the function call. I'm going give an example that presumes the search_field query param changes the search:
search = request.query_params.get('search');
search_field = request.query_params.get('search_field');
search_index = request.query_params.get('search_index');
kwargs = {
'SearchIndex': search_index,
'ResponseGroup': 'ItemAttributes,Offers,Images,EditorialReview,Reviews'
}
# Presuming search_field is the key, i.e. "Keywords", or "Actor"
kwargs[search_field] = search
### kwargs['Keywords'] = search
response = amazon.ItemSearch(**kwargs)

I have used a dictionary instead. This is how I have done it;
page = request.query_params.get('page');
search = request.query_params.get('search');
search_field = request.query_params.get('search_field');
search_index = request.query_params.get('search_index');
dict = {'SearchIndex':search_index, 'ResponseGroup': 'ItemAttributes,Offers,Images,EditorialReview,Reviews'}
dict[search_field] = search;
response = amazon.ItemSearch(**dict)
I just passed a dictionary with my dynamic variable as a key.

How to set group = true in couchdb

I am trying to use map/reduce to find the duplication of the data in couchDB
the map function is like this:
function(doc) {
if(doc.coordinates) {
emit({
twitter_id: doc.id_str,
text: doc.text,
coordinates: doc.coordinates
},1)};
}
}
and the reduce function is:
function(keys,values,rereduce){return sum(values)}
I want to find the sum of the data in the same key, but it just add everything together and I get the result:
<Row key=None, value=1035>
Is that a problem of group? How can I set it to true?

Assuming you're using the couchdb package from pypi, you'll need to pass a dictionary with all of the options you require to the view.
for example:
import couchdb
# the design doc and view name of the view you want to use
ddoc = "my_design_document"
view_name = "my_view"
#your server
server = couchdb.server("http://localhost:5984")
db = server["aCouchDatabase"]
#naming convention when passing a ddoc and view to the view method
view_string = ddoc +"/" + view_name
#query options
view_options = {"reduce": True,
"group" : True,
"group_level" : 2}
#call the view
results = db.view(view_string, view_options)
for row in results:
#do something
pass

Dealing with a high volume of query params in python

We have an api that consumes a around 50 to 100 query params. Currently the handler takes all of the params and sets them as attributes in a Meta object. Something like this
meta = Meta()
meta.param1 = param.get('param1', 'somedefault')
meta.param2 = param.get('param2', 'someotherdefault')
and so on. My question is, is there a better way to handle this than just a loooong list of assigns in the handler? My current idea is to just break it out into a helper function.
meta = self.get_meta(param)
Any other ideas?
(updated my example)

PARAMETERS = [
'param1',
'param2',
# ...
]
meta = Meta()
for name in PARAMETERS:
setattr(meta, name, param[name])
Based on your comment...
DEFAULTS = {
'param1': 1,
'param2': 'something',
}
meta = Meta()
for name, value in DEFAULTS.items():
setattr(meta, name, param.get(name, value))

This seems like something you should do with a mapping instead. Unless you need to filter the parameters, this sounds like a bad idea.
So you'll have something like this:
class Meta(IterableUserDict):
pass
meta = Meta()
meta.update(param)

Expanding on whats been said including default values
PARAMETERS = [
('param1', "default"),
('param2', "default2"),
# ...
]
meta = Meta()
for name, default in PARAMETERS:
setattr(meta, name, param.get(name, default))