We Keep Coding

How to send messages from Flask server to RabbitMQ server queue?

I'm trying to send a message from Flask server, which acts as a producer, to a RabbitMQ server queue. The port I'm using to produce the messages is '5672'.
I've created an exchange and a queue on RabbitMQ's Management server and the main goal is to receive a message in the server's queue.
This is the code that I have at the moment, it does not produce any errors and returns that the message has been sent:
#app.route("/create/<message>")
def create_bucket(message):
credentials = pika.PlainCredentials("guest", "guest")
connection = pika.BlockingConnection(pika.ConnectionParameters(host="localhost", credentials=credentials))
channel = connection.channel()
channel.queue_declare(queue="TestQueue", durable=True)
channel.basic_publish(exchange="TestExchange", routing_key="TestQueue", body=message, properties=pika.BasicProperties(delivery_mode=2))
connection.close()
return "[x] Message sent %s" % message
if __name__ == "__main__":
app.run(debug=True, port=5672)
Though the message does not appear in RabbitMQ's server's queue.
Are there any resources or ways to send a message from Flask's server to RabbitMQ's server queue?

Managed to solve the problem by deleting routing_key="TestQueue", as the routing_key was not declared in RabbitMQ server.

Proper way to resend messages to topic

I load messages from kafka topic to database. Loading to database can fail. Also I do not want to lose unsent messages.
App code:
import faust
app = faust.App('App', broker='kafka://localhost:9092')
source_topic = app.topic('source_topic')
failed_channel = app.channel() # channel for unsent messages
#app.agent(source_topic)
async def process(stream):
async for batch in stream.take(100_000, within=60):
# here we have not info about partitions and keys
# to reuse them when resending if sending failed
try:
pass # send to database. can fail
except ConnectionError:
for record in batch:
# sending to channel is faster than sending to topic
await failed_channel.send(value=record)
#app.agent(failed_channel)
async def resend_failed(stream):
async for unsent_msg in stream:
await source_topic.send(value=unsent_msg)
Maybe there is more standart way to handle such situations? Adding app.topic('source_topic', acks=False) works only after restarting app.

I load messages from kafka topic to database
Maybe there is more standart way to handle such situations
Yes - it's called Kafka Connect :-)
The standard pattern is to do any processing on your data and write it [back to] Kafka topics. Then you use the Kafka topic as a source for a Kafka Connect sink connector, in this case the Kafka Connect JDBC Sink connector.
Kafka Connect is part of Apache Kafka, and handles restarts, scaleout, failures, etc etc.
See also Kafka Connect in Action: JDBC Sink

Sending message from Kafka to socket.io in python

I have an end-to-end pipeline of an web application like below in Python3.6
Socket(connection from client to server) -> Flask Server -> Kafka Producer ->Kafka Consumer ->NLPService
Now when I get some result back from the NLPService, I need to send it back to the client. I am thinking below steps
NLP service writes the result to a different topic on Kafka producer (done)
Kafka consumer retrieves the result from Kafka broker (done)
Kafka consumer needs to write the result to the flask server
Then flask server will send the result back to the socket
Socket writes to client
I have already done steps 1-2. But stuck at step 3, 4. How do I write from Kafka to the flask server? If I just call a function at my server.py, then logically it seems like I have to create a socket within at function at server.py which will do the job of sending to client through socket. But syntax wise it looks weird. What am I missing?
at consumer.py
#receiving reply
topicReply = 'Reply'
consumerReply = KafkaConsumer(topicReply, value_deserializer=lambda m: json.loads(m.decode('ascii')))
for message in consumerReply:
#send reply back to Server
fromConsumer(message.value)
at server.py
socketio = SocketIO(app)
def fromConsumer(msg):
#socketio.on('reply')
def replyMessage(msg):
send(msg)
The above construct in server.py doesn't make sense to me. Please suggest.

KafkaProducer for Python sending message if done on command line but not if done through a Python script

I've got my zookeeper and broker running.
When I send a message from the command line like so:
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('topic', b'some_message_bytes')
I'm able to receive it on the other end in my consumer running this:
from kafka import KafkaConsumer
consumer = KafkaConsumer('topic')
for msg in consumer:
print msg
However, when I include the exact same code for the producer inside of a python script and run it, the consumer doesn't print the message and furthermore, it does not even get sent. I confirmed this by running this on my command line:
./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
localhost:9092 --topic topic --time -1 --offsets 1 | awk -F ":" '{sum +=
$3} END {print sum}'
This shows the number of messages in the current topic. It goes up as I send more messages from the command line-version of the python producer, but the number of messages in the topic does not get incremented after running the python script.
Does anyone know what's going on here?

Running a program in shell vs running it in a file is different. In shell when you send the message it's sent with some fixed flush time in Kafka and Kafka producer instance is still alive in the shell. When running the code in a text file, even before the message can be flushed the program ends and the producer object is garbage collected.
To fix this in producer code use producer.flush()
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('topic', b'some_message_bytes')
# This is important
producer.flush()

Python pubsub / message queue over HTTP?

I have a python script that will run on a local machine that needs to access a message queue (RabbitMQ) or receive subscribed events over HTTP. I've researched several solutions, but none seem natively designed to allow desktop clients to access them over HTTP. I'm thinking that using Twisted as a proxy is an option as well. Any guidance or recommendations would be greatly appreciated. Thanks in advance.

I've read this tutorial on RabbitMQ site, and they provide name of some libraries that could solve receiving messages.
Sender: send.py
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',
routing_key='hello',
body='Hello World!')
print " [x] Sent 'Hello World!'"
connection.close()
Receiver: receive.py
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
print ' [*] Waiting for messages. To exit press CTRL+C'
def callback(ch, method, properties, body):
print " [x] Received %r" % (body,)
channel.basic_consume(callback,
queue='hello',
no_ack=True)
channel.start_consuming()
Now we can try out our programs in a terminal. First, let's send a message using our send.py program:
$ python send.py
[x] Sent 'Hello World!'
The producer program send.py will stop after every run. Let's receive it:
$ python receive.py
[*] Waiting for messages. To exit press CTRL+C
[x] Received 'Hello World!'
Hurray! We were able to send our first message through RabbitMQ. As you might have noticed, the receive.py program doesn't exit. It will stay ready to receive further messages, and may be interrupted with Ctrl-C.
Try to run send.py again in a new terminal.
We've learned how to send and receive a message from a named queue. It's time to move on to part 2 and build a simple work queue.

I've decided to use wamp http://wamp.ws/. Still experimenting with it, but it's working quite well at the moment.

Choice #1
You may be interested in this RabbitHub
Choice #2
If you want it to be on port#80, cant you do port forwarding using a proxy? It could be challenging, but
Choice #3
If your script is not tightly coupled with RMQ message format , you can try celery ( which uses RMQ underneath), then u can try celery Http gateway or celery web hooks if u want any other application to be triggered directly
It might be time consuming to get it up. However, Celery opens up loads of flexibility
Choice #4
For one of my projects, I developed an intermediate web service (Flask Service) to use RMQ
Not ideal, but it served the purpose at that time.