We Keep Coding

Why isn't my Operator Extra Link showing in Airflow?

Using this as a reference: https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html
I can not get links to show in the UI. I have tried adding the link within the operator itself and building the separate extra_link.py file to add it and the link doesn't show up when looking at the task in graph or grid view. Here is my code for creating it in the operator:
class upstream_link(BaseOperatorLink):
"""Create a link to the upstream task"""
name = "Test Link"
def get_link(self, operator, *, ti_key):
return "https://www.google.com"
# Defining the plugin class
class AirflowExtraLinkPlugin(AirflowPlugin):
name = "integration_links"
operator_extra_links = [
upstream_link(),
]
class BaseOperator(BaseOperator, SkipMixin, ABC):
""" Base Operator for all integrations """
operator_extra_links = (upstream_link(),)
This is a custom BaseOperator class used by a few operators in my deployment. I don’t know if the inheritance is causing the issue or not. Any help would be greatly appreciated.
Also, the goal is to have this on mapped tasks, this does work with mapped tasks right?
Edit: Here is the code I used when i tried the stand alone file approach in the plugins folder:
from airflow.models.baseoperator import BaseOperatorLink
from plugins.operators.integrations.base_operator import BaseOperator
from airflow.plugins_manager import AirflowPlugin
class upstream_link(BaseOperatorLink):
"""Create a link to the upstream task"""
name = "Upstream Data"
operators = [BaseOperator]
def get_link(self, operator, *, ti_key):
return "https://www.google.com"
# Defining the plugin class
class AirflowExtraLinkPlugin(AirflowPlugin):
name = "extra_link_plugin"
operator_extra_links = [
upstream_link(),
]

The custom plugins should be defined in the plugins folder (by default $AIRFLOW_HOME/plugins) to be processed by the plugin manager.
Try to create a new script in the plugins folder, and move AirflowExtraLinkPlugin class to this script, it should work.

The issue turned out to be the inheritance. Attaching the extra link does not carry through to the children as it seems that Airflow is just looking for that specific operator name. Extra Links also do not seem to work with mapped tasks.

Keyword 'Capture Page Screenshot' could not be run on failure: Cannot access execution context

How do I access execution context in my program to capture screenshot ?
The following program will fail since contain text does not exist.
from ExtendedSelenium2Library import ExtendedSelenium2Library
import logging
class firsttest():
def googleit(self):
self.use_url = 'https://google.ca'
self.use_browser = 'chrome'
s2l = ExtendedSelenium2Library()
s2l.open_browser(self.use_url, self.use_browser)
s2l.maximize_browser_window()
try:
# Should fail
s2l.page_should_contain('this text does not exist on page')
except:
logger.debug('failed')
runit = firsttest()
runit.googleit()
When I run this program get warning
WARNING - Keyword 'Capture Page Screenshot' could not be run on failure: Cannot access execution context

You have to use robot to execute the test, you can't just instantiate classes and expect them to work. They are designed to work only when run by robot.
If you need to write tests in python, there's no need to use ExtendedSeleniumLilbrary, you can just call the selenium API directly from python.

The problem likely stems from the fact that you did not write your python Library in the correct format for Robot Framework.
Here is the correct format for writing Python code in Robot Framework:
from robot.libraries.BuiltIn import BuiltIn
class ClickAnElement(object):
def __init__(self):
self.selenium_lib = BuiltIn().get_library_instance('ExtendedSelenium2Library')
def click_an_element(self, locator):
BuiltIn().click_element(locator)
How this works (I believe) is in Robot Framework you call this library in your *** Settings *** section with Library ClickAnElement.py. That activates the __init__ function. Then you can call the keywords like you would a keyword from Selenium2Library. So, if I were to re-write your posted code in the correct format, it would look as follows:
from robot.libraries.BuiltIn import BuiltIn
import logging
class FirstTest():
def __init__(self):
self.selenium_lib = BuiltIn().get_library_instance('ExtendedSelenium2Library')
def google_it(self):
self.use_url = 'https://google.ca'
self.use_browser = 'chrome'
s2l = ExtendedSelenium2Library()
s2l.open_browser(self.use_url, self.use_browser)
s2l.maximize_browser_window()
try:
# Should fail
s2l.page_should_contain('this text does not exist on page')
except:
logger.debug('failed')
Then, my .robot file would look as such:
*** Settings ***
Library FirstTest
*** Test Cases ***
Test Google It
Google It
You were writing a Python file to work outside of Robot Framework. If you want it to work inside of Robot Framework, you need to use the correct Library format.
Mind you, I'm only formatting your code, not testing it. I can't, since I don't have your application to test it on.

How do I access variables/objects in my whole application?

I'm developping a PyQT application. This application is able to load some data from a database, and then do various analysis on these data. All of this works. But as the analyses can be quite complicated, and as a will not be the only user, I had to develop a system with users defined script.
Basically, there's a text editor where the user can program his own small python script (with functions). Then the user can save the script or execute it, by loading the file as a module (within the application).
Here, there a simplified version of my application.
The core of the application is in My_apps.py
and the plugins are here in the same folder i.e. Plugin_A.py
this is the code of My_apps.py:
import sys,os
class Analysis(object):
def __init__(self):
print "I'm the core of the application, I do some analysis etc..."
def Analyze_Stuff(self):
self.Amplitudes_1=[1,2,3,1,2,3]
class Plugins(object):
def __init__(self):
newpath = "C:\Users\Antoine.Valera.NEUROSECRETION\Desktop\Model" #where the file is
sys.path.append(newpath)
Plugin_List=[]
for module in os.listdir(newpath):
if os.path.splitext(module)[1] == ".py":
module=module.replace(".py","")
Plugin_List.append(module)
for plugin in Plugin_List:
a=__import__(plugin)
setattr(self,plugin,a)
def Execute_a_Plugin(self):
Plugins.Plugin_A.External_Function(self)
if __name__ == "__main__":
Analysis=Analysis()
Plugins=Plugins()
Plugins.Execute_a_Plugin()
and here is an example of the code of Plugin_A.py
def External_Function(self):
Analysis.Analyze_Stuff()
print Analysis.Amplitudes_1
why do I get :
Traceback (most recent call last):
File "C:\Users\Antoine.Valera.NEUROSECRETION\Desktop\Model\My_Apps.py", line 46, in <module>
Plugins.Execute_a_Plugin()
File "C:\Users\Antoine.Valera.NEUROSECRETION\Desktop\Model\My_Apps.py", line 37, in Execute_a_Plugin
Plugins.Plugin_A.External_Function(self)
File "C:\Users\Antoine.Valera.NEUROSECRETION\Desktop\Model\Plugin_A.py", line 8, in External_Function
Analysis.Analyze_Stuff()
NameError: global name 'Analysis' is not defined
but if I add the 2 lines following lines instead of Plugins.Execute_a_Plugin()
Analysis.Analyze_Stuff()
print Analysis.Amplitudes_1
then, it works.
How could I indicate to every dynamically loaded plugins that he has to use the variables/objects already existing in Analysis? Why can't I print Analysis.Amplitudes_1 from within the plugin?
Thank you!!

The error message seems perfectly clear: the name "Analysis" doesn't exist in the namespace of the Plugin_A module you imported, and so External_Function cannot access it.
When you import the Plugin_A module, it doesn't get access to the names in the namespace of the importing module, My_apps. So it cannot "see" the instance of the Analysis class that you created there.
A simple solution to this is to change the signature of External_Function (and other related functions), so that it can take an instance of the Analysis class:
Plugin_A.py:
def External_Function(self, analysis):
analysis.Analyze_Stuff()
print analysis.Amplitudes_1
My_apps.py:
...
def Execute_a_Plugin(self):
plugins.Plugin_A.External_Function(self, analysis)
if __name__ == "__main__":
analysis = Analysis()
plugins = Plugins()
plugins.Execute_a_Plugin()
Note that I have altered the naming so that the instance names don't shadow the class names.

You have to import your module. Add the following on top of Plugin_A.py
from My_apps import Analysis
A = Analysis()
A.Analyze_Stuff()
print A.Amplitudes_1

bottle framework with multiple files

I've read the Bottle Documentation but I can't find the example of how to use Bottle with multiple files. Below is the way I did and it's working but I'm not sure whether this is the proper way to go (I saw merge() and mount() in API but not sure if they are related to this). Please give me the comments.
all.py (This is the main file for running)
#! /usr/bin/python
from bottle import route, run
import hello1
import hello2 # if I have 10 files, it will be 10 imports
run(host='localhost', port=8080, debug=True)
hello1.py
#! /usr/bin/python
from bottle import route, run
#route('/hello1')
def hello1():
return "Hello world no.1"
hello2.py
#! /usr/bin/python
from bottle import route, run
#route('/hello2')
def hello2():
return "Hello world no.2"

I've wanted to use a single bottle server to serve a suite of micro-applications and for a decent separation of concerns, have wanted to do what you've been looking for.
Here's how I resolved my task:
rootApp.py (Your main file)
from bottle import Bottle
from clientApp import clientApp
rootApp = Bottle()
#rootApp.route('/')
def rootIndex():
return 'Application Suite Home Page'
if __name__ == '__main__':
rootApp.merge(clientApp)
rootApp.run(debug=True)
clientApp.py (The new app needing to be merged into the suite)
from bottle import Bottle
clientApp = Bottle()
#clientApp.route('/clientApp')
def clientAppIndex():
return 'Client App HomePage'
I am not sure if this is the best way to do it, but it seems to work without complaints and saves the hassle of having to share ports between applications that could otherwise have mutual knowledge. The approach really stems out of a design preference but I'd be grateful if someone could demonstrate how/if the AppStack could be used to get the same result.

If you split your code into 10 Python modules, you’re going to do 10 imports. You can iterate with __import__:
for i in range(1, 11):
__import__('hello%d' % i)
but this doesn’t strike me as a good idea. Why would you need 10 modules with a micro-framework?

Not sure about the BEST way, but indeed mount() might be used and looks good for me (tested with python 3.6 + Bottle 0.12.13).
First of all, building one of your "sub apps" in a app1.py file:
from bottle import Bottle
server1 = Bottle()
#server1.route('/')
def root():
return 'Hello from sub app 1'
Then you use it in your main app:
from bottle import Bottle
from app1 import server1
mainApp = Bottle()
if __name__ == "__main__":
mainApp.mount('/appli1', server1)
mainApp.run()
Lets go:
Hit your server address: http://myServer/appli1
Let me know if you need a fully functional example.

Why would you want to have one module per view? The views are usually grouped in some logical manner, e.g.:
/, /post/:id, /tags/, /tag/:tag in blog.py,
/admin, /admin/newpost, /admin/editpost/:id in admin.py,
and so forth.
You should also read the chapter Becoming Big from the Flask documentation. For a medium-sized app, you'll probably want to create a package with a layout similar to this:
/yourapplication
/runserver.py
/yourapplication
/__init__.py
/views.py
/static
/style.css
/templates
layout.html
index.html
login.html
For even larger apps, split views into a sub-package.

web.py / pythonpath confusion

Im playing around with web.py as a lightweight web framework. Im having problems when i attempt to move the actual implementation of my page into a separate file instead of the root file. As a demonstration, My core.py file looks like this:
import web, sys, os
sys.path.append(os.path.abspath(os.path.dirname(__file__)))
urls = (
'/', 'index'
)
app = web.application(urls, globals())
render = web.template.render('templates/')
if __name__ == "__main__":
app.run()
ive moved my implementation into a file called index.py at the same level as core.py. My implementation looks like this:
class index:
def GET(self):
return "Hello world"
however, whenever i run my application, i get an error:
<type 'exceptions.KeyError'> at /
can anybody tell me what is going on?

According to http://webpy.org/tutorial3.en#urlhandling, web.py does a lookup for the classes you specified in your urls in the global namespace.
In your core.py there is no class named index (after you moved it), that's what causes this keyerror. In my test I could fix that by importing the index class in core.py.
from index import index
(I haven't used web.py before, so please correct me if I'm wrong)

You can add dots to crawl into modules. So say you have a folder controllers with a file named file.py and you wanted to access the controller named index:
from controllers import *
urls = (
'/', 'controllers.file.index'
)

I'm guessing the bug is in your template. I hit this error when if forgot a ':' on an if statement in my template.