We Keep Coding

How can I get the url of the host there I want do build docker

I need to build docker conters using DockerOperator on a remote host via Airflow
determining my ockerOperator
task1 = DockerOperator(
task_id="task1",
image=image,
docker_url=DOCKER_URL,
auto_remove=True,
api_version="auto",
force_pull=True
}
I Have a remote server like this: user#12.345.67.890
which format is correct for the docker_url parameter?

programatically set connections / variables in airflow

Is there a way to set connections / variables programtically in airflow? I am aware this is defeating the very purpose of not exposing these details in the code, but to debug it would really help me big time if I could do something like the following pseudo code:
# pseudo code
from airflow import connections
connections.add({name:'...',
user:'...'})

Connection is DB entity and you can create it. See below
from airflow import settings
from airflow.models import Connection
conn = Connection(
conn_id=conn_id,
conn_type=conn_type,
host=host,
login=login,
password=password,
port=port
)
session = settings.Session()
session.add(conn)
session.commit()
As for variables - just use the API. See example below
from airflow.models import Variable
Variable.set("my_key", "my_value")
A good blog post on this topic can be found here.

how to pass redshift console output to logger?

I would like to connect to a redshift server using psycopg2 and execute some SQL statements. I am scheduling this on an airflow server and running via PythonOperator. I would like the redshift server console output (STOUT) to be passed back to the LoggingConnection, however it only seems to pass the query executed.
The purpose of this is to be able to debug upstream issues and log ETL metadata (in context of the job). Is this possible??
Instantiate a logger object and initialize conn with the logger. This works, but provides no information besides the query executed.
import pandas as pd
import logging
import psycopg2 as pg
import sys
from psycopg2.extras import LoggingConnection
conn = pg.connect(<.....>)
conn.autocommit = True
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.addHandler(logging.StreamHandler(sys.stdout))
conn.initialize(logger)
cur.execute("INSERT INTO tableA <select...>")
# Example of what I want to Log:
# [2019-07-15 15:59:51] 2000 rows inserted... in 2 s 318 ms (execution: 2 s 279 ms, fetching: 39 ms)
I want the logger to be passed all output from the server console, emulating what one would see if they were to run it locally.

Is there a way to create/modify connections through Airflow API

Going through Admin -> Connections, we have the ability to create/modify a connection's params, but I'm wondering if I can do the same through API so I can programmatically set the connections
airflow.models.Connection seems like it only deals with actually connecting to the instance instead of saving it to the list. It seems like a function that should have been implemented, but I'm not sure where I can find the docs for this specific function.

Connection is actually a model which you can use to query and insert a new connection
from airflow import settings
from airflow.models import Connection
conn = Connection(
conn_id=conn_id,
conn_type=conn_type,
host=host,
login=login,
password=password,
port=port
) #create a connection object
session = settings.Session() # get the session
session.add(conn)
session.commit() # it will insert the connection object programmatically.

You can also add, delete, and list connections from the Airflow CLI if you need to do it outside of Python/Airflow code, via bash, in a Dockerfile, etc.
airflow connections --add ...
Usage:
airflow connections [-h] [-l] [-a] [-d] [--conn_id CONN_ID]
[--conn_uri CONN_URI] [--conn_extra CONN_EXTRA]
[--conn_type CONN_TYPE] [--conn_host CONN_HOST]
[--conn_login CONN_LOGIN] [--conn_password CONN_PASSWORD]
[--conn_schema CONN_SCHEMA] [--conn_port CONN_PORT]
https://airflow.apache.org/cli.html#connections
It doesn't look like the CLI currently supports modifying an existing connection, but there is a Jira issue for it with an active open PR on GitHub.
AIRFLOW-2840 - cli option to update existing connection
https://github.com/apache/incubator-airflow/pull/3684

First check if connection exists, after create new Connection using from airflow.models import Connection :
import logging
from airflow import settings
from airflow.models import Connection
def create_conn(conn_id, conn_type, host, login, pwd, port, desc):
conn = Connection(conn_id=conn_id,
conn_type=conn_type,
host=host,
login=login,
password=pwd,
port=port,
description=desc)
session = settings.Session()
conn_name = session.query(Connection).filter(Connection.conn_id == conn.conn_id).first()
if str(conn_name) == str(conn.conn_id):
logging.warning(f"Connection {conn.conn_id} already exists")
return None
session.add(conn)
session.commit()
logging.info(Connection.log_info(conn))
logging.info(f'Connection {conn_id} is created')
return conn

You can populate connections using environment variables using the connection URI format.
The environment variable naming convention is AIRFLOW_CONN_<conn_id>, all uppercase.
So if your connection id is my_prod_db then the variable name should be AIRFLOW_CONN_MY_PROD_DB.
In general, Airflow’s URI format is like so:
my-conn-type://my-login:my-password#my-host:5432/my-schema?param1=val1&param2=val2
Note that connections registered in this way do not show up in the Airflow UI.

To use session = settings.Session(), it assumes the airflow database backend has been initiated. For those who haven't set it up for your development environment, a hybrid method using both Connection class and environment variables will be a workaround.
Below is the example for setting up a S3Hook
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from airflow.models.connection import Connection
import os
import json
aws_default = Connection(
conn_id="aws_default",
conn_type="aws",
login='YOUR-AWS-KEY-ID',
password='YOUR-AWS-KEY-SECRET',
extra=json.dumps({'region_name': 'us-east-1'})
)
os.environ["AIRFLOW_CONN_AWS_DEFAULT"] = aws_default.get_uri()
s3_hook = S3Hook(aws_conn_id='aws_default')
s3_hook.list_keys(bucket_name='YOUR-BUCKET', prefix='YOUR-FILENAME')

Python code working in REPL but not when imported/executed as a module

So I have a module that's part of my Flask web app that contains the database connection. It's pretty simple.
from sqlalchemy import create_engine, MetaData
import os
from sqlalchemy.schema import Table, Column
host = os.environ.get("DATABASE_PORT_5432_TCP_ADDR") or "localhost"
port = os.environ.get("DATABASE_PORT_5432_TCP_PORT") or "5432"
url = host + ":" + port
engine = create_engine('postgresql+pg8000://postgres:password#%s/empathy' % url)
_meta = MetaData(bind=engine)
_meta.reflect(views=True, schema="biomart")
If I run python and copy these commands into the REPL, it executes fine, and I can run _meta.tables to get a list of tables in my database.
However if I run python database.py or simply import database in the REPL, this causes an error: 'communication error', error(111, 'Connection refused').
How can a connection be refused when the code is in a module, but it's fine when it's being run in the REPL? Are there any differences between these two situations I can use to debug my code?
The only other piece of information is that this is all in a docker container, and that's why I'm checking for the environment variables at the top. But I don't see this being an issue.