I'm trying to translate the following example HTTP server based on 0MQ written in C seen here: Hintjens' blog into Python.
def test_http_socket():
ctx = zmq.Context()
sock = ctx.socket(zmq.ROUTER)
sock.setsockopt(zmq.ROUTER_RAW, 1)
sock.bind("tcp://*:8080")
while True:
id_bytes = sock.recv()
print("id=",id_bytes)
request = sock.recv()
print("request=",request)
if b'/close' in request:
return
sock.send(id_bytes, zmq.SNDMORE)
sock.send(b"""HTTP/1.0 200 OK
Content-Type: text/plain
Hello, world !""")
sock.send(zmq.NULL)
The problem is on the next line, where I try to translate the C expression
zmq_send (router, NULL, 0, 0);
In Python I have the following stack trace:
Traceback (most recent call last):
File "<pyshell#96>", line 1, in <module>
test_http_socket()
File "<pyshell#95>", line 18, in test_http_socket
sock.send(zmq.NULL)
File "socket.pyx", line 565, in zmq.backend.cython.socket.Socket.send (zmq\backend\cython\socket.c:5104)
File "socket.pyx", line 612, in zmq.backend.cython.socket.Socket.send (zmq\backend\cython\socket.c:4868)
File "socket.pyx", line 168, in zmq.backend.cython.socket._send_copy (zmq\backend\cython\socket.c:1914)
File "buffers.pxd", line 200, in buffers.asbuffer_r (zmq\backend\cython\socket.c:6833)
File "buffers.pxd", line 151, in buffers.asbuffer (zmq\backend\cython\socket.c:6270)
TypeError: 0 does not provide a buffer interface.
In fact, sock.send can only be used to send buffers or messages.
Is there a way to use 0MQ's zmq_send in another manner from Python, to send a NULL frame ?
By the way, a comment in Github entry shows that it may not close the connection to the client even in C.
How can I ask the remote client to close its connection (in Python) ?
I'm using libzmq 4.0.1 and PyZMQ 14.0.1 over Python 3.3.2 on Windows 32bits.
First point: zmq.NULL is the ZMQ_NULL constant, for use in zeromq's security mechanism. For example:
socket.mechanism = zmq.NULL # or zmq.PLAIN or zmq.CURVE
It is not the NULL special C constant.
To send an empty message, simply send an empty bytestring:
socket.send(b'')
The second point is that you need to send the empty frame as a separate message, which you are not doing.
Here is a working example:
def http_hello_world():
ctx = zmq.Context()
sock = ctx.socket(zmq.ROUTER)
sock.router_raw = True
sock.bind("tcp://*:8080")
while True:
id_bytes, request = sock.recv_multipart()
print("id: %r" % id_bytes)
print("request:", request.decode('utf8'))
if b'/close' in request:
return
# send the body of the response
sock.send_multipart([
id_bytes,
b"""HTTP/1.0 200 OK
Content-Type: text/plain
Hello, world!
"""
])
# send an empty message to finish the response
sock.send_multipart([
id_bytes,
b''
])
By the way, a comment in Github entry shows that it may not close the connection to the client even in C. How can I ask the remote client to close its connection (in Python) ?
I think, as long as you send the empty frame to terminate the message, connections should be closed.
Related
HTTP/2 is a major revision of the HTTP network protocol used by the World Wide Web. The primary goals for HTTP/2 are to reduce latency by enabling full request and response multiplexing, minimize protocol overhead via efficient compression of HTTP header fields, and add support for request prioritization and server push
I am trying to implement a simple server a long with a client using HTTP/2 I followed the examples given here https://python-hyper.org/projects/h2/en/stable/twisted-post-example.html# However each time i try to send a file from the client to the server i get this error
AttributeError: 'NoneType' object has no attribute 'write'
The error comes from this part of the code
def sendFileData(self):
"""
Send some file data on the connection.
"""
# Firstly, check what the flow control window is for stream 1.
window_size = self.conn.local_flow_control_window(stream_id=1)
# Next, check what the maximum frame size is.
max_frame_size = self.conn.max_outbound_frame_size
# We will send no more than the window size or the remaining file size
# of data in this call, whichever is smaller.
bytes_to_send = min(window_size, self.file_size)
# We now need to send a number of data frames.
while bytes_to_send > 0:
chunk_size = min(bytes_to_send, max_frame_size)
data_chunk = self.fileobj.read(chunk_size)
self.conn.send_data(stream_id=1, data=data_chunk)
bytes_to_send -= chunk_size
self.file_size -= chunk_size
# We've prepared a whole chunk of data to send. If the file is fully
# sent, we also want to end the stream: we're done here.
if self.file_size == 0:
self.conn.end_stream(stream_id=1)
else:
# We've still got data left to send but the window is closed. Save
# a Deferred that will call us when the window gets opened.
self.flow_control_deferred = defer.Deferred()
self.flow_control_deferred.addCallback(self.sendFileData)
This line--> self.transport.write(self.conn.data_to_send())
HTTP/2 uses self.channel to write response. So change self.transport.write() to self.channel.write().
I am have a python client listening to SSE events from a server with node.js API
The flow is I sent an event to the node.js API through call_notification.py and run seevents.py in loop using run.sh(see below)
However I don't see that python client is receiving this SSE event? any guidance on why is that?
call_notification.py
import requests
input_json = {'BATS':'678910','root_version':'12A12'}
url = 'http://company.com/api/root_event_notification?params=%s'%input_json
response = requests.get(url)
print response.text
node.js API
app.get("/api/root_event_notification", (req, res, next) => {
console.log(req.query.params)
var events = require('events');
var eventEmitter = new events.EventEmitter();
//Create an event handler:
var myEventHandler = function () {
console.log('new_root_announced!');
res.status(200).json({
message: "New root build released!",
posts: req.query.params
});
}
seevents.py (python client listening to SSE events)
import json
import pprint
import sseclient
def with_urllib3(url):
"""Get a streaming response for the given event feed using urllib3."""
import urllib3
http = urllib3.PoolManager()
return http.request('GET', url, preload_content=False)
def with_requests(url):
"""Get a streaming response for the given event feed using requests."""
import requests
return requests.get(url, stream=True)
url = 'http://company.com/api/root_event_notification'
response = with_urllib3(url) # or with_requests(url)
client = sseclient.SSEClient(response)
#print client.events()
for event in client.events():
print "inside"
pprint.pprint(json.loads(event.data))
run.sh
#!/bin/sh
while [ /usr/bin/true ]
do
echo "Running sseevents.py"
python sseevents.py 2>&1 | tee -a sseevents.log.txt
echo "sleeping for 30 sec"
sleep 30
done
OUTPUT:-
Run call_notification.py on Terminal
node.js API OUTPUT
new_root_announced!
{'root_version': 'ABCD', 'BATS': '143'}
./run.sh --> DON'T SEE ABOVE EVENT below
Running sseevents.py
sleeping for 30 sec
Running sseevents.py
sleeping for 30 sec
Running sseevents.py
sleeping for 30 sec
Very short answer to you question:
The server code is not sending a SSE message back to the client.
Why? Because you need to follow the SSE format.
According to JASON BUTZ in Server-Sent Events With Node
You should send a Connection: keep-alive header to ensure the client keeps the connection open as well. A Cache-Control header should be sent with the value no-cache to discourage the data being cached. Finally, the Content-Type needs to be set to text/event-stream.
With all of that done a newline (\n) should be sent to the client and then the events can be sent. Events must be sent as strings, but what is in that string doesn’t matter. JSON strings are perfectly fine.
Event data must be sent in the format "data: <DATA TO SEND HERE>\n".
It’s important to note that at the end of each line should be a newline character. To signify the end of an event an extra newline character needs to be added as well.
Multiple data lines are perfectly fine.
Long answer to your question:
According to Eric Bidelman in html5rocks.com:
When communicating using SSEs, a server can push data to your app whenever it wants, without the need to make an initial request. In other words, updates can be streamed from server to client as they happen.
But, in order for this to happen, the client has to "start" by asking for it AND prepare to receive a stream of messages (when they happen).
The "start" is done by calling a SSE API endpoint (in your case, calling the Node.js API code).
The preparation is done by preparing to handle a stream of asynchronous messages.
SSEs open a single unidirectional channel between server and client.*
* The emphasis is mine
This means that the server has a "direct" channel to the client. It is not intended to be "started" (opened) by some other process/code that is not "the client" code.
Assuming from OP comments...
Expected behavior (verbose)
A client Alice calls the API endpoint with params {name: "Alice"}, nothing (visible) happens.
...then a client Bob calls the API endpoint with params {name: "Bob"}, client Alice receives a SSE with payload {name: "Bob", says: "Hi"}.
...then a client Carol calls the API endpoint with params {name: "Carol"}, clients Alice AND Bob each one receives a SSE with payload {name: "Carol", says: "Hi"}.
...and so on. Every time a new client calls the API endpoint with params, every other client who has a channel "open" will receive a SSE with the new "Hi" payload.
...and then client Bob "disconnects" from the server, client Alice, client Carol and all the clients that have a channel "open" will receive a SSE with payload {name: "Bob", says: "Bye"}.
...and so on. Every time an old client "disconnects" from the server, every other client who has a channel "open" will receive a SSE with the new "Bye" payload.
Abstracted behavior
Each new client that asks to "open" a channel sending some params or an old client "disconnects" from the server, they cause and event in the server.
Every time such an event happens in the server, the server sends a SSE message with the params and a message as payload to all the "open" channels.
Note on blocking Each client with an "open" channel will be "stuck" in an infinite waiting loop for events to happen. It is client design responsibility to use "threading" code techniques to avoid blocking.
Code
Your Python client should "ask" to start the single unidirectional channel AND keep waiting UNTIL the channel is closed. Should not end and start all over again with a different channel. It should keep the same channel open.
From the network perspective, it will be like a "long" response that does not end (until the SSE messaging is over). The response just "keeps coming and coming".
Your Python client code does that. I noted it is the exact sample code used from sseclient-py library.
Client code for Python 3.4
To include the parameters you want to send to the server, use some code from the Requests library docs/#passing-parameters-in-urls.
So, mixing those samples we end up with the following code as your Python 3.4 client:
import json
import pprint
import requests
import sseclient # sseclient-py
# change the name for each client
input_json = {'name':'Alice'}
#input_json = {'name':'Bob'}
#input_json = {'name':'Carol'}
url = 'http://company.com/api/root_event_notification'
stream_response = requests.get(url, params=input_json, stream=True)
client = sseclient.SSEClient(stream_response)
# Loop forever (while connection "open")
for event in client.events():
print ("got a new event from server")
pprint.pprint(event.data)
Client code for Python 2.7
To include the parameters you want to send to the server, encode them in the URL as query parameters using urllib.urlencode() library.
Make the http request with urllib3.PoolManager().request() so you will end up with a stream response.
Note that the sseclient library returns event data as unicode string. To convert back the JSON object to python object (with python strings) use byteify, a recursive custom function ( thanks to Mark Amery ).
Use the following code as your Python 2.7 client:
import json
import pprint
import urllib
import urllib3
import sseclient # sseclient-py
# Function that returns byte strings instead of unicode strings
# Thanks to:
# [Mark Amery](https://stackoverflow.com/users/1709587/mark-amery)
def byteify(input):
if isinstance(input, dict):
return {byteify(key): byteify(value)
for key, value in input.iteritems()}
elif isinstance(input, list):
return [byteify(element) for element in input]
elif isinstance(input, unicode):
return input.encode('utf-8')
else:
return input
# change the name for each client
input_json = {'name':'Alice'}
#input_json = {'name':'Bob'}
#input_json = {'name':'Carol'}
base_url = 'http://localhost:3000/api/root_event_notification'
url = base_url + '?' + urllib.urlencode(input_json)
http = urllib3.PoolManager()
stream_response = http.request('GET', url, preload_content=False)
client = sseclient.SSEClient(stream_response)
# Loop forever (while connection "open")
for event in client.events():
print ("got a new event from server")
pprint.pprint(byteify(json.loads(event.data)))
Now, the server code should:
emit an inside-server 'hello' event so other clients listen to the event
"open" the channel
Register to listen for all possible inside-server events to happen (this means, keeping the channel "open" and not sending anything between messages, just keeping the channel "open").
This includes to emit an inside-server 'goodbye' event so other clients listen to the event WHEN channel is closed by the client/network (and finally "wrap up").
Use the following Node.js API code:
var EventEmitter = require('events').EventEmitter;
var myEmitter = new EventEmitter;
function registerEventHandlers(req, res) {
// Save received parameters
const myParams = req.query;
// Define function that adds "Hi" and send a SSE formated message
const sayHi = function(params) {
params['says'] = "Hi";
let payloadString = JSON.stringify(params);
res.write(`data: ${payloadString}\n\n`);
}
// Define function that adds "Bye" and send a SSE formated message
const sayBye = function(params) {
params['says'] = "Bye";
let payloadString = JSON.stringify(params);
res.write(`data: ${payloadString}\n\n`);
}
// Register what to do when inside-server 'hello' event happens
myEmitter.on('hello', sayHi);
// Register what to do when inside-server 'goodbye' event happens
myEmitter.on('goodbye', sayBye);
// Register what to do when this channel closes
req.on('close', () => {
// Emit a server 'goodbye' event with "saved" params
myEmitter.emit('goodbye', myParams);
// Unregister this particular client listener functions
myEmitter.off('hello', sayHi);
myEmitter.off('goodbye', sayBye);
console.log("<- close ", req.query);
});
}
app.get("/api/root_event_notification", (req, res, next) => {
console.log("open -> ", req.query);
// Emit a inside-server 'hello' event with the received params
myEmitter.emit('hello', req.query);
// SSE Setup
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
});
res.write('\n');
// Register what to do when possible inside-server events happen
registerEventHandlers(req, res);
// Code execution ends here but channel stays open
// Event handlers will use the open channel when inside-server events happen
})
...continue quoting Eric Bidelman in html5rocks.com:
Sending an event stream from the source is a matter of constructing a plaintext response, served with a text/event-stream Content-Type, that follows the SSE format. In its basic form, the response should contain a "data:" line, followed by your message, followed by two "\n" characters to end the stream
In the client code, the sseclient-py library takes care of interpreting the SSE format so every time the two "\n" characters arrive, the library "iterates" a new "iterable" object (a new event) that has the data property with the message sent from the server.
This is how I tested the code
Started server with Node.js API code
Run a client with only the "Alice" line uncommented (Nothing is seen on this client console yet).
Run a second client with only "Bob" line uncommented. The console of the first client "Alice" shows: Bob saying "Hi" (Nothing is seen on Bob's client console yet).
Run a third client with only "Carol" line uncommented. Alice's and Bob's consoles show: Carol saying "Hi" (Nothing is seen on Carol's client console yet).
Stop/kill Bob's client. Alice's and Carol's consoles show: Bob saying "Bye".
So, code works OK :)
I'm trying to send an email from within Google App Engine.
Whenever I try to send, I get a "Missing subject error".
When I run the code on the development server, it seems to work fine - the console output looks right and I'm taken to the page that I was expecting. But when I upload it and run it, I get:
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 636, in __call__
handler.post(*groups)
File "/base/data/home/apps/spam-tool/1.349401522260793315/spam-tool.py", line 34, in post
body=cgi.escape(self.request.get('content')))
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 297, in send_mail
message.send(make_sync_call)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/mail.py", line 799, in send
raise ERROR_MAP[e.application_error](e.error_detail)
BadRequestError: Missing subject
But there's definitely a subject.
The code I'm using is:
> message = mail.EmailMessage(sender="admin_address#gmail.com>",
subject="test")
message.subject=self.request.get('content')
message.to = addr_to_send_to
message.body = self.request.get('content')
message.send()
(Yes, the subject is set twice... I've tried only setting it in one place or the other, and neither worked.)
Thanks in advance.
Try this:
def email_member(request):
to_addr = request.POST.get('email_to')
if not mail.is_email_valid(to_addr):
# Return an error message...
pass
else:
# note we can also send html email, e.g. html='<html><head></head><body>Hello world</body><html>'
mail.send_mail(
sender='admin#yourdomain.com',
to=to_addr,
subject=request.POST.get('email_title'),
body=request.POST.get('email_body'))
I suggest you also send your email within the taskqueue, so you could setup something like this, you could pass over a unique parameter so that you are less likely to be hacked.
def mail_member_information(email_to, email_title, email_body):
taskqueue.add(
url = 'email_message',
params =
{
'email_to': email_to,
'email_title': email_title,
'email_body': email_body,
'valid_guid': 'ae9e34ca-a2c5-476e-b8be-72d807e3dc6b'
}
)
If you just want to send to the administrators then use this:
mail.send_mail_to_admins(sender='admin#yourdomain.com',
subject=request.POST.get('email_title'),
body=request.POST.get('email_body'),
html=request.POST.get('email_body'))
I don't know if it is what is throwing you off... but sender should be like this:
sender = '<admin_address#gmail.com>',
you are missing the '<' for the sender. Sometimes it throws an error downstream of where the actual error is.
Also, make sure the admin_address#gmail.com is a Google Account that has been added as a developer for that app.
I think you can send like this, make sure that sender is a admin of the app
from google.appengine.api import mail
mail.send_mail(sender="something#gmail.com",
to=email_valid.email,
subject="Please validate your email",
body=body_text,
html=htmlbody_text)
Ok, so bascially what I want to do is intercept some packets that I know contains some JSON data. But HTTP packets aren't human-readable, so that's my problem, I need to make the entire packet (not just the header, which is already plain text), human-readable. I have no experience with networking at all.
import pcap
from impacket import ImpactDecoder, ImpactPacket
def print_packet(pktlen, data, timestamp):
if not data:
return
decoder = ImpactDecoder.EthDecoder()
ether = decoder.decode(data)
iphdr = ether.child()
tcphdr = iphdr.child()
if iphdr.get_ip_src() == '*******':
print tcphdr
p = pcap.pcapObject()
dev = 'wlan0'
p.open_live(dev, 1600, 0, 100)
try:
p.setfilter('tcp', 0, 0)
while 1:
p.loop(1, print_packet)
except KeyboardInterrupt:
print 'shutting down'
I've found tools like libpcap-python, scapy, Impacket pcapy and so on. They all seem good, but I can't figure out how to decode the packets properly with them.
Wireshark has this thing called "Line-based text data: text/html" which basically displays the information I'm after, so I thought it would be trivial to get the same info with python, it turns out it was not.
Both HTTP and JSON are human readable. On Wireshark, select a packet that relates to your HTTP transaction and right-click, select Follow TCP Stream, which should display the transaction in a Human readable form.
I have a working python program that is fetching a large volume of data via SOAP using suds. The web service is implemented with a paging function such that I can grab nnn rows with each fetch call and grab the next nnn with subsequent calls. If I authenticate to the HTTP server with code like the following
client = suds.client.Client(url=url, location=location, username=username, password=password, timeout=timeout)
everything works great. If, however, I use the following
t = suds.transport.https.HttpAuthenticated(username=username, password=password)
t.handler = urllib2.HTTPBasicAuthHandler(t.pm)
t.urlopener = urllib2.build_opener(t.handler)
client = suds.client.Client(url=url, location=location, timeout=timeout, transport=t)
it works for exactly 6 iterations. That is if I specify a fetch limit of 10 rows per fetch, I get back 60 rows. On the seventh fetch, I receive
File "build/bdist.linux-i686/egg/suds/client.py", line 542, in __call__
File "build/bdist.linux-i686/egg/suds/client.py", line 602, in invoke
File "build/bdist.linux-i686/egg/suds/client.py", line 649, in send
File "build/bdist.linux-i686/egg/suds/client.py", line 698, in failed
AttributeError: 'NoneType' object has no attribute 'read'
Does anyone have any suggestions as to what might be causing this. It is definitely this change that is causing the problem. I can swap authentication styles back and forth and it is completely reproducible.
I am running python 2.6.6 with suds 0.4.
Thanks
The problem seems to be that an urllib2.HTTPError is being raised from a lower level, and its fp attribute is None:
Line 81 in suds.transport.http:
except u2.HTTPError, e:
if e.code in (202,204):
result = None
else:
raise TransportError(e.msg, e.code, e.fp)
That exception eventually gets passed to line 698 in suds.client where that error.fp.read() line blows up:
def failed(self, binding, error):
status, reason = (error.httpcode, tostr(error))
reply = error.fp.read()
I'd suggest monkey-patching the suds.SoapClient class to get the HTTP error code and message. Add these lines before you construct your suds.Client, then run it to see what HTTP error the 7th fetch raises:
class TestClient(suds.client.SoapClient):
def failed(self, binding, error):
print error.httpcode, error.args
suds.client.SoapClient = TestClient