I´ve made a simple pipeline in Python to read from kafka, the thing is that the kafka cluster is on confluent cloud and I am having some trouble conecting to it.
Im getting the following log on the dataflow job:
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:820)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:631)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:612)
at org.apache.beam.sdk.io.kafka.KafkaIO$Read$GenerateKafkaSourceDescriptor.processElement(KafkaIO.java:1495)
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
So I think Im missing something while passing the config since it mentions something related to it, Im really new to all of this and I know nothing about java so I dont know how to proceed even reading the JAAS documentation.
The code of the pipeline is the following:
import apache_beam as beam
from apache_beam.io.kafka import ReadFromKafka
from apache_beam.options.pipeline_options import PipelineOptions
import os
import json
import logging
os.environ['GOOGLE_APPLICATION_CREDENTIALS']='credentialsOld.json'
with open('cluster.configuration.json') as cluster:
data=json.load(cluster)
cluster.close()
def logger(element):
logging.INFO('Something was found')
def main():
config={
"bootstrap.servers":data["bootstrap.servers"],
"security.protocol":data["security.protocol"],
"sasl.mechanisms":data["sasl.mechanisms"],
"sasl.username":data["sasl.username"],
"sasl.password":data["sasl.password"],
"session.timeout.ms":data["session.timeout.ms"],
"auto.offset.reset":"earliest"
}
print('======================================================')
beam_options = PipelineOptions(runner='DataflowRunner',project='project',experiments=['use_runner_v2'],streaming=True,save_main_session=True,job_name='kafka-stream-test')
with beam.Pipeline(options=beam_options) as p:
msgs = p | 'ReadKafka' >> ReadFromKafka(consumer_config=config,topics=['users'],expansion_service="localhost:8088")
msgs | beam.FlatMap(logger)
if __name__ == '__main__':
main()
I read something about passing a property java.security.auth.login.config in the config dictionary but since that example is with java and I´am using python Im really lost at what I have to pass or even if that´s the property I have to pass etc.
btw Im getting the api key and secret from here and this is what I am passing to sasl.username and sasl.password
I faced the same error the first time I tried the beam's expansion service. The key sasl.mechanisms that you are supplying is incorrect, try with sasl.mechanism also you do not need to supply the username and password since you are connection is authenticated by jasl basically the consumer_config like below worked for me:
config={
"bootstrap.servers":data["bootstrap.servers"],
"security.protocol":data["security.protocol"],
"sasl.mechanism":data["sasl.mechanisms"],
"session.timeout.ms":data["session.timeout.ms"],
"group.id":"tto",
"sasl.jaas.config":f'org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="Kafka" username=\"{data["sasl.username"]}\" password=\"{data["sasl.password"]}\";',
"auto.offset.reset":"earliest"
}
I got a partial answer to this question since I fixed this problem but got into another one:
config={
"bootstrap.servers":data["bootstrap.servers"],
"security.protocol":data["security.protocol"],
"sasl.mechanisms":data["sasl.mechanisms"],
"sasl.username":data["sasl.username"],
"sasl.password":data["sasl.password"],
"session.timeout.ms":data["session.timeout.ms"],
"group.id":"tto",
"sasl.jaas.config":f'org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="Kafka" username=\"{data["sasl.username"]}\" password=\"{data["sasl.password"]}\";',
"auto.offset.reset":"earliest"
}
I needed to provide the sasl.jaas.config porpertie with the api key and secret of my cluster and also the service name, however, now Im facing a different error whe running the pipeline on dataflow:
Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
This error shows after 4-5 mins of trying to run the job on dataflow, actually I have no idea how to fix this but I think is related to my broker on confluent rejecting the connection, I think this could be related to the zone execution since the cluster is in a different zone than job region.
UPDATE:
I tested the code on linux/ubuntu and I dont know why but the expansión service gets downloaded automatically so you wont get unsoported signal error, still having some issues trying to autenticate to confluent kafka tho.
We are trying to add a locust grpc client and got the code from examples directory here.
Our use case is such that locust will used as a library and not through UI.
The issues we are seeing stats outputted either in csv file or the code. I only difference between what we code we see in locust doc here and our code is service is started independently.
The issue is reproduced in this GitHub repo : https://github.com/saurabhsharma721/grpc_locust_stats_repro_prj
This issue is only observed in library mode. If we run it through ui the stats produced are correct. Please help.
I'm not sure, but have you tried adding this?
# start a greenlet that save current stats to history
gevent.spawn(stats_history, env.runner)
(from the documentation https://docs.locust.io/en/stable/use-as-lib.html)
I was able to solve this using following question with following code :
Added two static methods to be called on init and tear down
#staticmethod
def on_init(environment, **kwargs) -> None: # type: ignore[no-untyped-def]
print("Perform global setup to create a global state in LocustRunner")
#staticmethod
def on_quit(environment, **kwargs) -> None: # type: ignore[no-untyped-def]
print('Perform global teardown to clear the global state in LocustRunner')
Then added this code in driver
events.init.add_listener(LocustRunner.on_init)
env = Environment(user_classes=[TestClass], events=events)
# Using local runner in start
runner = env.create_local_runner()
# init event
env.events.init.fire(environment=env, runner=runner)
Driver code
# quit event
env.events.quitting.fire(environment=env, reverse=True)
There is already a stack-overflow answer for this but unable to search for that currently
I have this python script that I use to collect info on the EC2 instances for my Beanstalk applications.
It worked perfectly fine for a long time, and then it simply stopped producing results, but it throws no errors and there are no authentication problems.
What am I missing? Was there a change to the API?
The script is below:
import boto3
regions = ['us-east-1','us-west-2']
for region in regions:
ebs_client = boto3.client('elasticbeanstalk', region_name=region)
ec2_client = boto3.client('ec2', region_name=region)
apps = ebs_client.describe_applications()
print(apps)
for app in apps['Applications']:
appname = app['ApplicationName']
print(appname)
envs = ebs_client.describe_environments(ApplicationName=appname)
for env in envs['Environments']:
envname = env['EnvironmentName']
envid = env['EnvironmentId']
[... some more code ...]
Right on the first call to describe_applications it returns a 200 OK status code but with zero results. And I have a lot of Beanstalk apps running in those regions.
Problem solved. It had to do with the credentials used.
I removed them and used the Instance Role instead and it worked.
Still weird, though. Because there should be an error somewhere instead of producing zero results.
Im using python 2.7.9, windows 7...
The overall goal: Have another application access our custom com server (already running at this point) and send it a message to be displayed. Obviously, there needs to be a single server, multiple clients.
Im trying to use some custom code as a com server. The class was created as:
class StatusServerClass:
_public_methods_ = ...
_reg_progid_ = "CseStatusServerLib.CseStatusServer"
_reg_verprogid_ = "CseStatusServerLib.CseStatusServer"
_reg_progid_ = "CseStatusServerLib.CseStatusServer"
_reg_clsid_ = "{<GUID here>}"
_reg_desc_ = 'CSE Status Server'
_typelib_guid_ = "{<typelib GUID here>}"
_typelib_version_ = 1, 0 # Version 1.0
_reg_clsctx_ = pythoncom.CLSCTX_LOCAL_SERVER
_reg_threading_ = "Apartment" # Not used?
def __init__.....
and registered using:
win32com.server.register.UseCommandLine(StatusServerClass)
I can see it in regedit and as far as i know, it looks ok.
The GUID is right, name is right.
Now when i go to use it, this works just fine:
self.StatusClient = Dispatch('CseStatusServerLib.CseStatusServer')
but when i want to attach to a running instance from another exe (or even another python window for debug) using:
win32com.client.GetActiveObject("CseStatusServerLib.CseStatusServer")
it just gives me:
dispatch = pythoncom.GetActiveObject(resultCLSID)
com_error: (-2147221021, 'Operation unavailable', None, None)
Tells me that its not registered?
Ive tried using the GUID, Ive tried using pythoncom.GetObject with both the ID and the GUID... no luck.
Ive tried comtypes package and get the same thing.
Any ideas on what im doing wrong? Why does Dispatch find it by name, but GetActiveObject gets mad?
Seems that Dispatch working by name would suggest that the registering worked?
What else can i verify in the regedit?
thanks!!!
UPDATED 6/6/2016
In case you haven't realized yet, I know very little about this. But I have read that for win32com.client.GetActiveObject() to work, the server needs to be in the "running object table"... and its not.
So, I found some more example code that i used to register the class this way:
import win32com.server.util
wrapped = win32com.server.util.wrap(StatusServerClass)
flags = pythoncom.REGCLS_MULTIPLEUSE|pythoncom.REGCLS_SUSPENDED
handle = pythoncom.RegisterActiveObject(wrapped,
StatusServerClass._reg_clsid_,flags)
and that does allow the server to show in the running object table, and i can get this:
testSW = win32com.client.GetActiveObject("CseStatusServerLib.CseStatusServer")
to return without error.
So now, I can use Dispatch or GetActiveObject just fine in python, pythonWin, and even interact with the server in Excel/VB <-> python and it appears to share namespsace.
BUT...
I still cant get this IE-based third party app to use the existing server... even with GetActiveObject. Always wants to create new instance and use its own namespace... not good
Is there something with IE or Chrome that would prevent the existing server from being used? Again, it works fine in Excel/VB. The application is supposed to execute "python myscript.py" (which works fine in idle, pythonwin and cmdline), but doesnt execute the com server stuff when called from IE/Chrome App (although other python functions like file writing work just fine).
Also, seeing as how i know very little about this, by all means, suggest a better way of doing this: starting a server in python as a singleton and then accessing it from another application.
thanks!!
i started using couchdb with python-couchdb recently. The problem is when i use futon run my views written in python i get the following error message:
Error: os_process_error
{exit_status,1}
even for the default view it crashes.
def fun(doc):
yield None, doc
i haven't yet found much information regarding this issue so at this point im really lost. This is the log i get from couchdb:
{<0.3907.0>,crash_report,
[[{initial_call,{couch_file,init,['Argument__1']}},
{pid,<0.3907.0>},
{registered_name,[]},
{error_info,
{exit,
{os_process_error,{exit_status,1}},
[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[<0.3906.0>,couch_view,couch_secondary_services,couch_server_sup,
<0.33.0>]},
{messages,[]},
{links,[#Port<0.1483>,<0.3910.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,1423}],
[{neighbour,
[{pid,<0.3910.0>},
{registered_name,[]},
{initial_call,{couch_ref_counter,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},
{ancestors,
[<0.3906.0>,couch_view,couch_secondary_services,
couch_server_sup,<0.33.0>]},
{messages,[]},
{links,[#Port<0.1483>,<0.3910.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,1423}],
[{neighbour,
[{pid,<0.3910.0>},
{registered_name,[]},
{initial_call,{couch_ref_counter,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},
{ancestors,
[<0.3906.0>,couch_view,couch_secondary_services,
couch_server_sup,<0.33.0>]},
{messages,
[{'DOWN',#Ref<0.0.0.16475>,process,<0.3906.0>,
{os_process_error,{exit_status,1}}}]},
{links,[<0.3907.0>]},
{dictionary,[]},
{trap_exit,false},
{status,runnable},
{heap_size,233},
{stack_size,9},
{reductions,47}]}]]}}
Im running this on Ubuntu 10.04, with Django, couchdb and python-couchdb. The views on javascript works fine.
For couchdb-python query server exit status 1 means some error that.
What version of couchdb/couchdb-python you're uses?
Whats output will be if you run couchpy (or /usr/local/bin/couchpy or whatever you have setted in query_servers section for python key) directly from command line? Example as it should be:
$
~ $ couchpy
["reset"]
true
["add_fun", "def fun(doc): yield None, None"]
true
["map_doc", {}]
[[[null, null]]]
If p.2 works fine, try to enable CouchDB debug log level to trace query server commands and notice when it falls. If you're sure that this is a python query server bug, please write some story about it. Thanks(:
P.S. I hope that you have setup python query server correctly, but double things check never was useless(;