We Keep Coding

How to get a Kerberos token in Python? I have working C# and PS syntax but would like to convert it

I'm trying to use Python to get a Kerberos security token. I have some C# sharp syntax that does want I need but I have no clue where to begin to converting that to Python. Here is the C# code:
{
AppDomain.CurrentDomain.SetPrincipalPolicy(System.Security.Principal.PrincipalPolicy.WindowsPrincipal);
var domain = Domain.GetCurrentDomain().ToString();
using (var domainContext = new PrincipalContext(ContextType.Domain, domain))
{
KerberosSecurityTokenProvider tokenProvider = new KerberosSecurityTokenProvider(serviceName, System.Security.Principal.TokenImpersonationLevel.Identification, CredentialCache.DefaultNetworkCredentials);
KerberosRequestorSecurityToken securityToken = tokenProvider.GetToken(TimeSpan.FromMinutes(5)) as KerberosRequestorSecurityToken;
string serviceToken = Convert.ToBase64String(securityToken.GetRequest());
}
I've tried to use the 'microsoft.identitymodel.dll' and using the ctype and clr library, but, I am pretty new to Python so I'm not sure if that's even the correct avenue. I also have some Powershell code that does what I need as such:
[System.IdentityModel.Selectors.KerberosSecurityTokenProvider]$tokenProvider = [System.IdentityModel.Selectors.KerberosSecurityTokenProvider]::new($ServicePrincipalName, [System.Security.Principal.TokenImpersonationLevel]::Identification, $PSBoundParameters.Credential)
I know what the $ServicePrincipalName should be and I know I could use the subprocess library and just call the command using Powershell.exe, but I wanted to keep everything in Python.
Any help would be appreciated!

The primary non-CLR interface to Windows authentication is SSPI (likewise GSSAPI on Linux). The CLR libraries are just fancy wrappers around SSPI, so don't waste time trying to use them from Python – it's going to be more work than using SSPI directly.
If you're working exclusively on Windows, try the sspi module which is part of pywin32. Google tells me it works somewhat like this:
import sspi
context = sspi.ClientAuth(pkg_name="Negotiate",
targetspn="HTTP/foo.example.com")
# [receive input_token from server]
while True:
err, bufs = context.authorize(input_token)
output_token = bufs[0].Buffer
# [send output_token to server, receive input_token back]
if err == 0:
break
The basic model of SSPI/GSSAPI is "call step() in a loop until the mechanism reports it's complete", but for HTTP Negotiate authentication there is just one step (and the input token is usually empty), so you could skip the loop:
import sspi
ctx = sspi.ClientAuth("Negotiate",
targetspn="HTTP/foo.example.com")
in_token = None
err, bufs = ctx.authorize(in_token)
assert(err == 0)
out_token = bufs[0].Buffer
Since most Windows protocols use SPNEGO rather than raw Kerberos, python-pyspnego would be the best choice for cross-platform operation – it provides a unified interface to SPNEGO over both Windows SSPI and over Unix GSSAPI. (It can also do NTLM on Unix.)
import spnego
ctx = spnego.client(hostname="foo.example.com",
service="HTTP",
protocol="negotiate")
in_token = None
out_token = ctx.step(in_token)
The Linux/Unix equivalent of the sspi module would be python-gssapi, which uses very similar concepts. However, by default it deals with raw Kerberos tokens, not SPNEGO tokens.
import gssapi
target = gssapi.Name("HTTP#foo.example.com",
gssapi.NameType.hostbased_service)
ctx = gssapi.SecurityContext(name=target,
usage="initiate")
in_token = None
out_token = ctx.step(in_token)
To force it to use SPNEGO, use:
SPNEGO = gssapi.Mechanism.from_sasl_name("SPNEGO")
ctx = gssapi.SecurityContext(name=target,
mech=SPNEGO,
usage="initiate")
(Note: do not install the gssapi module on Windows – if you do, then the spnego module will stop using SSPI and will start complaining about missing MIT GSSAPI libraries.)
Two other modules exist: pykerberos for GSSAPI (Unix) and winkerberos for SSPI (Windows). They work similarly, but the API they provide is a bit ugly and non-Pythonic (pykerberos was originally a wrapper for Objective-C interfaces in MacOS).
And of course if you're trying to use this for HTTP, there already are several modules for that – e.g. requests-kerberos adds SSPI (and GSSAPI) support through pyspnego:
import requests
import requests_kerberos
resp = requests.get("https://foo.example.com",
auth=requests_kerberos.HTTPKerberosAuth())
Similarly, requests-gssapi and httpx-gssapi do the same via gssapi on Unix.

How to Connect kafka consumer topic using keytab in python

I am new to Kafka and trying to read messages from kafka consumer topics using python. I am using below piece of code to read the messages.
from kafka import KafkaConsumer
topic = 'topic'
bootstrap_servers = 'server'
consumer = KafkaConsumer(bootstrap_servers = [bootstrap_servers],
auto_offset_reset = 'earliest',
enable_auto_commit = True,
security_protocol = 'SASL_PLAINTEXT',
sasl_mechanism = 'GSSAPI',
consumer_timeout_ms = 1000)
When I run this, got the error message 'Could not find KfW installation' and failed to connect Kafka. Installed Kerberos for Windows MSI and reran, its able to establish the connectivity.
However, I am trying to avoid KfW installation in the local system, instead find a way to pass the keytab file and principal to use in the authentication process and read the data from kafka topic. (if its possible?)
But not sure, which argument of KafkaConsumer holds the keytab file.
Please suggest any better way available?

kafka authentication and authorization

I read Kafka's documents
But I did not understand. Can I use username and password for Python Producers?
Can specify that any Producer can only produce a Topic, like MySQL .(producer has written with Python)

yes, you can have user/pass per topic. see official documentation Authorization and ACLs.
You can enable security with either SSL or SASL, Kafka's SASL support:
SASL/GSSAPI (Kerberos) - starting at version 0.9.0.0
SASL/PLAIN - starting at version 0.10.0.0
SASL/SCRAM-SHA-256 and SASL/SCRAM-SHA-512 - starting at version 0.10.2.0
From the docs, example of adding Acls:
Suppose you want to add an acl "Principals User:Bob and User:Alice are allowed to perform Operation Read and Write on Topic Test-Topic from IP 198.51.100.0 and IP 198.51.100.1". You can do that by executing the CLI with following options:
1
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Bob --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic
Also in this Blog post you can find some information
I'm not sure what library you are using but it should just be a matter of passing the proper properties to the producer/client; kafka-python has support:
Support SASL/Kerberos
Support for ACL based kafka

If you want to use username+password for authentication, you need to enable SASL authentication using the Plain mechanism on your cluster. See the Authentication using SASL section on the Kafka website for the full instructions.
Note that you also probably want to enable SSL (SASL_SSL), as otherwise, SASL Plain would transmit credentials in plaintext.
Several Python clients support SASL Plain, for example:
kafka-python: https://github.com/dpkp/kafka-python
confluent-kafka-python: https://github.com/confluentinc/confluent-kafka-python
Regarding authorizations, using the default authorizer, kafka.security.auth.SimpleAclAuthorizer, you can restrict a producer to only be able to produce to a topic. Again this is all fully documented on Kafka's website in the Authorization and ACLs section.
For example with SASL Plain, by default, the Principal name is the username that was used to connect. Using the following command you can restrict user Alice to only be able to produce to the topic named testtopic:
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:Alice --producer --topic testtopic

do you mean something like this :
topic = "test"
sasl_mechanism = "PLAIN"
username = "admin"
password = "pwd$"
security_protocol = "SASL_PLAINTEXT"
#context = ssl.create_default_context()
#context.options &= ssl.OP_NO_TLSv1
#context.options &= ssl.OP_NO_TLSv1_1
consumer = KafkaConsumer(topic, bootstrap_servers='kafka1:9092',
#api_version=(0, 10),
security_protocol=security_protocol,
#ssl_context=context,
#ssl_check_hostname=True,
#ssl_cafile='../keys/CARoot.pem',
sasl_mechanism = sasl_mechanism,
sasl_plain_username = username,
sasl_plain_password = password)
#ssl_certfile='../keys/certificate.pem',
#ssl_keyfile='../keys/key.pem')#,api_version = (0, 10))

What is "principal" argument of kerbros-sspi?

I was trying to connect to remote machine via WinRM in Python (pywinrm) using domain account, following the instruction in
How to connect to remote machine via WinRM in Python (pywinrm) using domain account?
using
session = winrm.Session(server, auth=('user#DOMAIN', 'doesNotMatterBecauseYouAreUsingAKerbTicket'), transport='kerberos')
but I got this:
NotImplementedError("Can't use 'principal' argument with kerberos-sspi.")
I googled "principal argument" and I got its meaning in mathematics,which is in complex_analysis (https://en.m.wikipedia.org/wiki/Argument_(complex_analysis)) and definitely not the right meaning. I'm not a native English speaker and I got stuck here.
The original code is here:
https://github.com/requests/requests-kerberos/blob/master/requests_kerberos/kerberos_.py
def generate_request_header(self, response, host, is_preemptive=False):
"""
Generates the GSSAPI authentication token with kerberos.
If any GSSAPI step fails, raise KerberosExchangeError
with failure detail.
"""
        # Flags used by kerberos module.
        gssflags = kerberos.GSS_C_MUTUAL_FLAG | kerberos.GSS_C_SEQUENCE_FLAG
        if self.delegate:
            gssflags |= kerberos.GSS_C_DELEG_FLAG
        try:
            kerb_stage = "authGSSClientInit()"
            # contexts still need to be stored by host, but hostname_override
            # allows use of an arbitrary hostname for the kerberos exchange
            # (eg, in cases of aliased hosts, internal vs external, CNAMEs
            # w/ name-based HTTP hosting)
            kerb_host = self.hostname_override if self.hostname_override is not None else host
            kerb_spn = "{0}#{1}".format(self.service, kerb_host)
            
            kwargs = {}
            # kerberos-sspi: Never pass principal. Raise if user tries to specify one.
            if not self._using_kerberos_sspi:
                kwargs['principal'] = self.principal
            elif self.principal:
                raise NotImplementedError("Can't use 'principal' argument with kerberos-sspi.")
Any help will be greatly appreciated.

Can I call the Bluemix message hub service from Python?

The kafka-python client supports Kafka 0.9 but doesn't obviously include the new authentication and encryption features so my guess is that it only works with open servers (as in previous releases). In any case, even the Java client needs a special message hub login module to connect (or so it would seem from the example) which suggests that nothing will work unless there is a similar module available for Python.
My specific scenario is that I want to use the message hub service from a Jupyter notebook also hosted in Bluemix (the Apache Spark service).

I was able to connect using the kafka-python library:
$ pip install --user kafka-python
Then ...
from kafka import KafkaProducer
from kafka.errors import KafkaError
import ssl
############################################
# Service credentials from Bluemix UI:
############################################
bootstrap_servers = # kafka_brokers_sasl
sasl_plain_username = # user
sasl_plain_password = # password
############################################
sasl_mechanism = 'PLAIN'
security_protocol = 'SASL_SSL'
# Create a new context using system defaults, disable all but TLS1.2
context = ssl.create_default_context()
context.options &= ssl.OP_NO_TLSv1
context.options &= ssl.OP_NO_TLSv1_1
producer = KafkaProducer(bootstrap_servers = bootstrap_servers,
sasl_plain_username = sasl_plain_username,
sasl_plain_password = sasl_plain_password,
security_protocol = security_protocol,
ssl_context = context,
sasl_mechanism = sasl_mechanism,
api_version=(0,10))
# Asynchronous by default
future = producer.send('my-topic', b'raw_bytes')
# Block for 'synchronous' sends
try:
record_metadata = future.get(timeout=10)
except KafkaError:
# Decide what to do if produce request failed...
log.exception()
pass
# Successful result returns assigned partition and offset
print (record_metadata.topic)
print (record_metadata.partition)
print (record_metadata.offset)
This worked for me from Bluemix spark as a service from a jupyter notebook, however, note that this approach is not using spark. The code is just running on the driver host.

The SASL support in the Kafka Python client has been requested : https://github.com/dpkp/kafka-python/issues/533 but until the username/password login method used by Message Hub is supported, it won't work

Until this is natively supported by the Bluemix Apache Spark Service, you can follow the same approach as the Realtime Sentiment Analysis project. Helper code for this can be found on the cds labs spark samples github repo.

We've added some text to our documentation on non-Java language support - see the "CONNECTING AND AUTHENTICATING IN A NON-JAVA APPLICATION" section:
https://www.ng.bluemix.net/docs/services/MessageHub/index.html
Our current authentication method is non-standard and not supported by the Apache project, but was a temporary solution. The Message Hub team is working with the Apache Kafka community to develop KIP-43. Once this is finalised, we'll change the Message Hub authentication implementation to match and it will be possible to implement clients to that spec in any language.