We Keep Coding

How to pass a Jinja2 to HiveQL using Python

I'm using Gcloud Composer as my Airflow. When I try to use Jinja in my HQL code, it does not translate it correctly.
I know that the HiveOperator has a Jinja translator as I'm used to it, but the DataProcHiveOperator doesn't.
I've tried to use the HiveConf directly into my HQL files, but when setting those values to my Partition (i.e. INSERT INTO TABLE abc PARTITION (ds = ${hiveconf:ds}))`, it doesn't work.
I have also added the following to my HQL file:
SET ds=to_date(current_timestamp());
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
But it didn't work as HIVE is transforming the formula above into a STRING.
So my idea was to combine both operators to have the Jinja translator working fine, but when I do that, I get the following error: ERROR - submit() takes from 3 to 4 positional arguments but 5 were given.
I'm not very familiar with Python coding and any help would be great, see below code for the operator I'm trying to build;
Header of the Python File (please note that the file contains other Operators not mentioned in this question):
import ntpath
import os
import re
import time
import uuid
from datetime import timedelta
from airflow.contrib.hooks.gcp_dataproc_hook import DataProcHook
from airflow.contrib.hooks.gcs_hook import GoogleCloudStorageHook
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults
from airflow.version import version
from googleapiclient.errors import HttpError
from airflow.utils import timezone
from airflow.utils.operator_helpers import context_to_airflow_vars
modified DataprocHiveOperator:
class DataProcHiveOperator(BaseOperator):
template_fields = ['query', 'variables', 'job_name', 'cluster_name', 'dataproc_jars']
template_ext = ('.q',)
ui_color = '#0273d4'
#apply_defaults
def __init__(
self,
query=None,
query_uri=None,
hiveconfs=None,
hiveconf_jinja_translate=False,
variables=None,
job_name='{{task.task_id}}_{{ds_nodash}}',
cluster_name='cluster-1',
dataproc_hive_properties=None,
dataproc_hive_jars=None,
gcp_conn_id='google_cloud_default',
delegate_to=None,
region='global',
job_error_states=['ERROR'],
*args,
**kwargs):
super(DataProcHiveOperator, self).__init__(*args, **kwargs)
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
self.query = query
self.query_uri = query_uri
self.hiveconfs = hiveconfs or {}
self.hiveconf_jinja_translate = hiveconf_jinja_translate
self.variables = variables
self.job_name = job_name
self.cluster_name = cluster_name
self.dataproc_properties = dataproc_hive_properties
self.dataproc_jars = dataproc_hive_jars
self.region = region
self.job_error_states = job_error_states
def prepare_template(self):
if self.hiveconf_jinja_translate:
self.query_uri= re.sub(
"(\$\{(hiveconf:)?([ a-zA-Z0-9_]*)\})", "{{ \g<3> }}", self.query_uri)
def execute(self, context):
hook = DataProcHook(gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to)
job = hook.create_job_template(self.task_id, self.cluster_name, "hiveJob",
self.dataproc_properties)
if self.query is None:
job.add_query_uri(self.query_uri)
else:
job.add_query(self.query)
if self.hiveconf_jinja_translate:
self.hiveconfs = context_to_airflow_vars(context)
else:
self.hiveconfs.update(context_to_airflow_vars(context))
job.add_variables(self.variables)
job.add_jar_file_uris(self.dataproc_jars)
job.set_job_name(self.job_name)
job_to_submit = job.build()
self.dataproc_job_id = job_to_submit["job"]["reference"]["jobId"]
hook.submit(hook.project_id, job_to_submit, self.region, self.job_error_states)
I would like to be able to use Jinja templating inside my HQL code to allow partition automation on my data pipeline.
P.S: I'll use the Jinja templating mostly for Partition DateStamp
Does anyone know what is the error message I'm getting + help me solve it?
ERROR - submit() takes from 3 to 4 positional arguments but 5 were given
Thank you!

It is because of the 5th argument job_error_states which is only in master and not in the current stable release (1.10.1).
Source Code for 1.10.1 -> https://github.com/apache/incubator-airflow/blob/76a5fc4d2eb3c214ca25406f03b4a0c5d7250f71/airflow/contrib/hooks/gcp_dataproc_hook.py#L219
So remove that parameter and it should work.

How to decode python string

I have some code which I would like to be decoded but not having much luck in guessing what the codepage is, if any is being used. Any help would be much appreciated.
i am using python command line in windows 7 pc,if any python guru guide me how to decrypt and see the code thaat would be appreciated.
exec("import re;import base64");exec((lambda p,y:(lambda o,b,f:re.sub(o,b,f))(r"([0-9a-f]+)",lambda m:p(m,y),base64.b64decode("NTQgYgo1NCA3CjU0IDMKNTQgMWUKNTQgOQo1NCAxOAozZiAgICAgICA9IGIuMTAoKQoxNiAgID0gIjQzOi8vMTIuM2QvNGMvMWQuMjUuZi00ZC4zZSIKYSA9ICIxZC4yNS5mIgoyYSA4KDYpOgoJMzMgMy5jKCczYy4yZSglNTIpJyAlIDYpID09IDEKCjJhIDE1KDM1KToKCTUgPSAzLjQoMWUuNS4xYygnMTM6Ly8yZC8xZicsJzMwJykpCgkyMyA1CgkyMSA9IDcuMTQoKQoJMjEuMzgoIjEwIDI4IiwiMjAgMTAuLiIsJycsICczNiA0MCcpCgkxMT0xZS41LjFjKDUsICdlLjNlJykKCTM5OgoJCTFlLjFhKDExKQoJMWI6CgkJMmMKCQk5LmUoMzUsIDExLCAyMSkKCQkyID0gMy40KDFlLjUuMWMoJzEzOi8vMmQnLCcxZicpKQoJCTIzIDIKCQkyMS4zNCgwLCIiLCAiM2IgNDciKQoJCTE4LjQ4KDExLDIsMjEpCgkJCgkJMy41MygnMjIoKScpOyAKCQkzLjUzKCcyNigpJyk7CgkJMy41MygiNDUuZCgpIik7IAoJCTE5PTcuMzcoKTsgMTkuNTAoIjMyISIsIjJmIDNhIDQ5IDQxIDI5IiwiICAgWzI0IDQ2XTMxIDUxIDRhIDRlIDE3LjNkWy8yNF0iKQoJCSIiIgoJCTM5OgoJCQkxZS4xYSgxMSkKCQkxYjoKCQkJMmMKCQkJIzI3KCkKCQk0Mjo0NCgpCgkJIiIiCgoyYSAyYigpOgoJNGYgNGIgOChhKToKCQkxNSgxNikKCQoKCjJiKCk=")))(lambda a,b:b[int("0x"+a.group(1),16)],"0|1|addonfolder|xbmc|translatePath|path|script_name|xbmcgui|script_chk|downloader|scriptname|xbmcaddon|getCondVisibility|UpdateLocalAddons|download|supermax|Addon|lib|supermaxwizard|special|DialogProgress|INSTALL|website|SuperMaxWizard|extract|dialog|remove|except|join|plugin|os|addons|Installing|dp|UnloadSkin|print|COLOR|video|ReloadSkin|FORCECLOSE|Installer|Installed|def|Main|pass|home|HasAddon|SuperMax|packages|Brought|Success|return|update|url|Please|Dialog|create|try|Wizard|Nearly|System|com|zip|addon|Wait|been|else|http|quit|XBMC|gold|Done|all|has|You|not|sm|MP|By|if|ok|To|s|executebuiltin|import".split("|")))

The code is uglified. You can unobfuscate it yourself by executing the contents of exec(...) in your Python shell.
import re
import base64
print ((lambda p,y.....split("|")))
EDIT: As snakecharmerb says, it is generally not safe to execute unknown code. I analysed the code to find that running the insides of exec will only decrypt, and leaving off the exec itself will just result in a string. This procedure ("execute stuff inside exec") is by no means a generally safe method to decrypt uglified code, and you need to actually analyse what it does. But, at this point, I was asking you to trust my judgement, which, if it is wrong, theoretically could expose you to an attack. In addition, it seems you have problems getting it to run on your Python; so here's what I'm getting from the above:
import xbmcaddon
import xbmcgui
import xbmc
import os
import downloader
import extract
addon = xbmcaddon.Addon()
website = "http://supermaxwizard.com/sm/plugin.video.supermax-MP.zip"
scriptname = "plugin.video.supermax"
def script_chk(script_name):
return xbmc.getCondVisibility('System.HasAddon(%s)' % script_name) == 1
def INSTALL(url):
path = xbmc.translatePath(os.path.join('special://home/addons','packages'))
print path
dp = xbmcgui.DialogProgress()
dp.create("Addon Installer","Installing Addon..",'', 'Please Wait')
lib=os.path.join(path, 'download.zip')
try:
os.remove(lib)
except:
pass
downloader.download(url, lib, dp)
addonfolder = xbmc.translatePath(os.path.join('special://home','addons'))
print addonfolder
dp.update(0,"", "Nearly Done")
extract.all(lib,addonfolder,dp)
xbmc.executebuiltin('UnloadSkin()');
xbmc.executebuiltin('ReloadSkin()');
xbmc.executebuiltin("XBMC.UpdateLocalAddons()");
dialog=xbmcgui.Dialog(); dialog.ok("Success!","SuperMax Wizard has been Installed"," [COLOR gold]Brought To You By SuperMaxWizard.com[/COLOR]")
"""
try:
os.remove(lib)
except:
pass
#FORCECLOSE()
else:quit()
"""
def Main():
if not script_chk(scriptname):
INSTALL(website)
Main()

What replaces the now-deprecated Carbon.File.FSResolveAliasFile in Python on OSX?

In Python 2, I can use the following code to resolve either a MacOS alias or a symbolic link:
from Carbon import File
File.FSResolveAliasFile(alias_fp, True)[0].as_pathname()
where alias_fp is the path to the file I'm curious about, stored as a string (source).
However, the documentation cheerfully tells me that the whole Carbon family of modules is deprecated. What should I be using instead?
EDIT: I believe the code below is a step in the right direction for the PyObjC approach. It doesn't resolve aliases, but it seems to detect them.
from AppKit import NSWorkspace
def is_alias (path):
uti, err = NSWorkspace.sharedWorkspace().typeOfFile_error_(
os.path.realpath(path), None)
if err:
raise Exception(unicode(err))
else:
return "com.apple.alias-file" == uti
(source)
Unfortunately I'm not able to get #Milliways's solution working (knowing nothing about Cocoa) and stuff I find elsewhere on the internet looks far more complicated (perhaps it's handling all kinds of edge cases?).

The PyObjC bridge lets you access NSURL's bookmark handling, which is the modern (backwards compatible) replacement for aliases:
import os.path
from Foundation import *
def target_of_alias(path):
url = NSURL.fileURLWithPath_(path)
bookmarkData, error = NSURL.bookmarkDataWithContentsOfURL_error_(url, None)
if bookmarkData is None:
return None
opts = NSURLBookmarkResolutionWithoutUI | NSURLBookmarkResolutionWithoutMounting
resolved, stale, error = NSURL.URLByResolvingBookmarkData_options_relativeToURL_bookmarkDataIsStale_error_(bookmarkData, opts, None, None, None)
return resolved.path()
def resolve_links_and_aliases(path):
while True:
alias_target = target_of_alias(path)
if alias_target:
path = alias_target
continue
if os.path.islink(path):
path = os.path.realpath(path)
continue
return path

The following Cocoa code will resolve alias.
NSURL *targetOfAlias(NSURL *url) {
CFErrorRef *errorRef = NULL;
CFDataRef bookmark = CFURLCreateBookmarkDataFromFile (NULL, (__bridge CFURLRef)url, errorRef);
if (bookmark == nil) return nil;
CFURLRef resolvedUrl = CFURLCreateByResolvingBookmarkData (NULL, bookmark, kCFBookmarkResolutionWithoutUIMask, NULL, NULL, false, errorRef);
CFRelease(bookmark);
return CFBridgingRelease(resolvedUrl);
}
I don't know how to invoke Cocoa framework from Python, but I am sure someone has done it
The following link shows code to resolve aslias or symlink https://stackoverflow.com/a/21151368/838253

The APIs those modules use are deprecated by Apple, it appears. You should use POSIX APIs instead.
os.path.realpath(FILE_OBJECT.name)

Django - importing dynamically

I'm trying to do a dynamic import of a python module in django. I have two different apps that I want to import from, and I want to replace these import statements:
from app1.forms import App1ProfileForm
from app2.forms import App2ProfileForm
I am dynamically able to create the strings App1ProfileForm and App2ProfileForm and then instantiate them like so:
globals()[form]()
I tried following some of the instructions in this post: Dynamically import class by name for static access
and so I tried doing this:
theModule = __import__("app1.forms.App1ProfileForm")
but I'm getting an error that says No module named App1ProfileForm
EDIT:::
Ok I tried this code:
theModule = __import__("app1")
print theModule
theClass = getattr(theModule,'forms')
print theClass
theForm = getattr(theClass,'App1ProfileForm')
print theForm
theForm.initialize()
but I get an error that type object 'App1ProfileForm' has no attribute 'initialize'

You don't want to do this. Imports are done when the relevant code is first executed - in the case of module-level imports, it's when the module itself is imported. If you're depending on something in the request, or some other run-time element, to determine what class you want, then this will not work.
Instead, just import them both, and get the code to choose which one you need:
from app1.forms import App1ProfileForm
from app2.forms import App2ProfileForm
forms = {'app1': App1ProfileForm,
'app2': App2ProfileForm}
relevant_form = forms[whatever_the_dependent_value_is]

I don't quite know how you're generting the string to import. I'll assume you generate the whole "path". Try this:
def import_from_strings(paths):
ret = []
for path in paths:
module_name, class_name = path.rsplit('.', 1)
module = __import__(module_name, globals(), locals(), [class_name], -1)
ret.append(getattr(module, class_name))
return ret

Aren't you trying to import a class, and not a module ? I'm not an expert, but I think you must import the module using __import__, then select it's App1ProfileForm class with something like yourmodule.App1ProfileForm

I figured it out. Here's how to do it:
theModule = __import__(module_name+".forms") # for some reason need the .forms part
theClass = getattr(theModule,'forms')
theForm = getattr(theClass,form_name)
then to initialize:
theForm() or theForm(request.POST)

Need help using M2Crypto.Engine to access USB Token

I am using M2Crypto-0.20.2. I want to use engine_pkcs11 from the OpenSC project and the Aladdin PKI client for token based authentication making xmlrpc calls over ssl.
from M2Crypto import Engine
Engine.load_dynamic()
dynamic = Engine.Engine('dynamic')
# Load the engine_pkcs from the OpenSC project
dynamic.ctrl_cmd_string("SO_PATH", "/usr/local/ssl/lib/engines/engine_pkcs11.so")
Engine.cleanup()
Engine.load_dynamic()
# Load the Aladdin PKI Client
aladdin = Engine.Engine('dynamic')
aladdin.ctrl_cmd_string("SO_PATH", "/usr/lib/libeTPkcs11.so")
key = aladdin.load_private_key("PIN","password")
This is the error I receive:
key = pkcs.load_private_key("PIN","eT0ken")
File "/usr/local/lib/python2.4/site-packages/M2Crypto/Engine.py", line 70, in load_private_key
return self._engine_load_key(m2.engine_load_private_key, name, pin)
File "/usr/local/lib/python2.4/site-packages/M2Crypto/Engine.py", line 60, in _engine_load_key
raise EngineError(Err.get_error())
M2Crypto.Engine.EngineError: 23730:error:26096075:engine routines:ENGINE_load_private_key:not initialised:eng_pkey.c:112:
For load_private_key(), what should be passed as the first argument? The M2Crypto documentation does not explain it.
I don't get any errors loading the engines, but I'm not sure if I'm loading them correctly. It seems like the engine ID has to be a specific name but I don't find that list anywhere. 'dynamic' is working for me.
Any help would be appreciated!

Found !!!!
Yes, exactly the way where I came from.
So, actually the ENGINE_init() is not implemented in M2Crypto.Engine. So, only one solution: patching!!! (very small...) so I've created a new Engine method (in Engine.py)
def engine_initz(self):
"""Return engine name"""
return m2.engine_initz(self._ptr)
Why engine_initz ? because engine_init is already define in SWIG/_engine.i,:
void engine_init(PyObject *engine_err) {
Py_INCREF(engine_err);
_engine_err = engine_err;
}
I don't really know what is done, so I've prefered creating a new one... So I've just added the following to SWIG/_engine.i:
%rename(engine_initz) ENGINE_init;
extern int ENGINE_init(ENGINE *);
And recompile the __m2crypto.so, now just add a "pkcs11.engine_initz()" before launching the private key, and it works.....

I don't know what and why the engine_init code present in current M2Crypto is supposed to do. Exposing ENGINE_init() as engine_init2 with the following patch to M2Crypto helps:
Index: SWIG/_engine.i
===================================================================
--- SWIG/_engine.i (revision 719)
+++ SWIG/_engine.i (working copy)
## -44,6 +44,9 ##
%rename(engine_free) ENGINE_free;
extern int ENGINE_free(ENGINE *);
+%rename(engine_init2) ENGINE_init;
+extern int ENGINE_init(ENGINE *);
+
/*
* Engine id/name functions
*/
After this, the following code takes me further (but urllib does not fully work for me currently):
import sys, os, time, cgi, urllib, urlparse
from M2Crypto import m2urllib2 as urllib2
from M2Crypto import m2, SSL, Engine
# load dynamic engine
e = Engine.load_dynamic_engine("pkcs11", "/Users/martin/prefix/lib/engines/engine_pkcs11.so")
pk = Engine.Engine("pkcs11")
pk.ctrl_cmd_string("MODULE_PATH", "/Library/OpenSC/lib/opensc-pkcs11.so")
m2.engine_init2(m2.engine_by_id("pkcs11")) # This makes the trick
cert = e.load_certificate("slot_01-id_01")
key = e.load_private_key("slot_01-id_01", sys.argv[1])
ctx = SSL.Context("sslv23")
ctx.set_cipher_list("HIGH:!aNULL:!eNULL:#STRENGTH")
ctx.set_session_id_ctx("foobar")
m2.ssl_ctx_use_x509(ctx.ctx, cert.x509)
m2.ssl_ctx_use_pkey_privkey(ctx.ctx, key.pkey)
opener = urllib2.build_opener(ctx)
urllib2.install_opener(opener)

Looking at the pastebin link Becky provided, I believe it translates to something like this in the new API:
from M2Crypto import Engine, m2
dynamic = Engine.load_dynamic_engine("pkcs11", "/Users/martin/prefix/lib/engines/engine_pkcs11.so")
pkcs11 = Engine.Engine("pkcs11")
pkcs11.ctrl_cmd_string("MODULE_PATH", "/Library/OpenSC/lib/opensc-pkcs11.so")
r = pkcs11.ctrl_cmd_string("PIN", sys.argv[1])
key = pkcs11.load_private_key("id_01")
So I am betting that if you substitute "/Users/martin/prefix/lib/engines/engine_pkcs11.so" with "/usr/local/ssl/lib/engines/engine_pkcs11.so" and "/Library/OpenSC/lib/opensc-pkcs11.so" with "/usr/lib/libeTPkcs11.so" you might get it to work with Aladdin.

That is exactly the code I've tried. But It ended with the following error:
Traceback (most recent call last):
File "prog9.py", line 13, in <module>
key = pkcs11.load_private_key("id_45")
File "/usr/lib/pymodules/python2.5/M2Crypto/Engine.py", line 70, in load_private_key
return self._engine_load_key(m2.engine_load_private_key, name, pin)
File "/usr/lib/pymodules/python2.5/M2Crypto/Engine.py", line 60, in _engine_load_key
raise EngineError(Err.get_error())
M2Crypto.Engine.EngineError: 11814:error:26096075:engine outines:ENGINE_load_private_key:not initialised:eng_pkey.c:112:
I'm using OpenSC PKCS11 lib, not aladdin lib. But I don't think the problem is closed.

I tried the code that Heikki suggested (minus one line) and got the same error as Erlo. For load_private_key(), how do I know what to put in for the argument?
dynamic = Engine.load_dynamic_engine("pkcs11", "/usr/local/ssl/lib/engines/engine_pkcs11.so")
# m2.engine_free(dynamic) this line gave me an error TypeError: in method 'engine_free', argument 1 of type 'ENGINE *'
pkcs11 = Engine.Engine("pkcs11")
pkcs11.ctrl_cmd_string("MODULE_PATH", "/usr/lib/libeTPkcs11.so")
r = pkcs11.ctrl_cmd_string("PIN", "password")
key = pkcs11.load_private_key("id_01")

I think the problem is not really the "load_private_key()". It's like something is missing between "MODULE_PATH" definition and the load_private_key() call. What happen if you remplace "/usr/lib/libeTPkcs11.so" by a wrong path ? In my case I have no error related to this.
I've run "pcscd" in foreground with high debug level, there is no call to smartcard during the python execution... So definitly, I don't understand what's wrong...
The equivalent in "openssl" is using "-pre" command. The "-pre" (by opposite to the "-post") are command sent to the engine before loading. Perhaps we need to call a methode which "load" the engine after all "ctrl_cmd_string" calls ?? ...
Lost :-/