Websocket timeout in Django Channels / Daphne - python

Short question version: what am I doing wrong in my Daphne config, or my Consumer code, or my client code?
channels==1.1.8
daphne==1.3.0
Django==1.11.7
Details below:
I am trying to keep a persistent Websocket connection open using Django Channels and the Daphne interface server. I am launching Daphne with mostly default arguments: daphne -b 0.0.0.0 -p 8000 my_app.asgi:channel_layer.
I am seeing the connections closing after some idle time in the browser, shortly over 20 seconds. The CloseEvent sent with the disconnect has a code value of 1006 (Abnormal Closure), no reason set, and wasClean set to false. This should be the server closing the connection without sending an explicit close frame.
The Daphne CLI has --ping-interval and --ping-timeout flags with default values of 20 and 30 seconds, respectively. This is documented as "The number of seconds a WebSocket must be idle before a keepalive ping is sent," for the former, and "The number of seconds before a WebSocket is closed if no response to a keepalive ping," for the latter. I read this as Daphne will wait until a WebSocket has been idle for 20 seconds to send a ping, and will close the Websocket if no response is received 30 seconds later. What I am seeing instead is connections getting closed after being 20 seconds idle. (Across three attempts with defaults, closed after 20081ms, 20026ms, and 20032ms)
If I change the server to launch with daphne -b 0.0.0.0 -p 8000 --ping-interval 10 --ping-timeout 60 my_app.asgi:channel_layer, the connections still close, around 20 seconds idle time. (After three attempts with updated pings, closed after 19892ms, 20011ms, 19956ms)
Code below:
consumer.py:
import logging
from channels import Group
from channels.generic.websockets import JsonWebsocketConsumer
from my_app import utilities
logger = logging.getLogger(__name__)
class DemoConsumer(JsonWebsocketConsumer):
"""
Consumer echos the incoming message to all connected Websockets,
and attaches the username to the outgoing message.
"""
channel_session = True
http_user_and_session = True
#classmethod
def decode_json(cls, text):
return utilities.JSONDecoder.loads(text)
#classmethod
def encode_json(cls, content):
return utilities.JSONEncoder.dumps(content)
def connection_groups(self, **kwargs):
return ['demo']
def connect(self, message, **kwargs):
super(DemoConsumer, self).connect(message, **kwargs)
logger.info('Connected to DemoConsumer')
def disconnect(self, message, **kwargs):
super(DemoConsumer, self).disconnect(message, **kwargs)
logger.info('Disconnected from DemoConsumer')
def receive(self, content, **kwargs):
super(DemoConsumer, self).receive(content, **kwargs)
content['user'] = self.message.user.username
# echo back content to all groups
for group in self.connection_groups():
self.group_send(group, content)
routing.py:
from channels.routing import route
from . import consumers
channel_routing = [
consumers.DemoConsumer.as_route(path=r'^/demo/'),
]
demo.js:
// Tracks the cursor and sends position via a Websocket
// Listens for updated cursor positions and moves an icon to that location
$(function () {
var socket = new WebSocket('ws://' + window.location.host + '/demo/');
var icon;
var moveTimer = null;
var position = {x: null, y: null};
var openTime = null;
var lastTime = null;
function sendPosition() {
if (socket.readyState === socket.OPEN) {
console.log('Sending ' + position.x + ', ' + position.y);
socket.send(JSON.stringify(position));
lastTime = Date.now();
} else {
console.log('Socket is closed');
}
// sending at-most 20Hz
setTimeout(function () { moveTimer = null; }, 50);
};
socket.onopen = function (e) {
var box = $('#websocket_box');
icon = $('<div class="pointer_icon"></div>').insertAfter(box);
box.on('mousemove', function (me) {
// some browsers will generate these events much closer together
// rather than overwhelm the server, batch them up and send at a reasonable rate
if (moveTimer === null) {
moveTimer = setTimeout(sendPosition, 0);
}
position.x = me.offsetX;
position.y = me.offsetY;
});
openTime = lastTime = Date.now();
};
socket.onclose = function (e) {
console.log("!!! CLOSING !!! " + e.code + " " + e.reason + " --" + e.wasClean);
console.log('Time since open: ' + (Date.now() - openTime) + 'ms');
console.log('Time since last: ' + (Date.now() - lastTime) + 'ms');
icon.remove();
};
socket.onmessage = function (e) {
var msg, box_offset;
console.log(e);
msg = JSON.parse(e.data);
box_offset = $('#websocket_box').offset();
if (msg && Number.isFinite(msg.x) && Number.isFinite(msg.y)) {
console.log((msg.x + box_offset.left) + ', ' + (msg.y + box_offset.top));
icon.offset({
left: msg.x + box_offset.left,
top: msg.y + box_offset.top
}).text(msg.user || '');
}
};
});
asgi.py:
import os
from channels.asgi import get_channel_layer
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings")
channel_layer = get_channel_layer()
settings.py:
CHANNEL_LAYERS = {
'default': {
'BACKEND': 'asgi_redis.RedisChannelLayer',
'ROUTING': 'main.routing.channel_routing',
'CONFIG': {
'hosts': [
'redis://redis:6379/2',
],
'symmetric_encryption_keys': [
SECRET_KEY,
],
}
}
}

The underlying problem turned out to be the nginx proxy in front of the interface server. The proxy was set to proxy_read_timeout 20s;. If there were keepalive pings generated from the server, these were not getting counted toward the upstream read timeout. Increasing this timeout to a larger value allows the Websocket to stay open longer. I kept proxy_connect_timeout and proxy_send_timeout at 20s.

Related

Apple Watch stops transmitting gyroscope/accelometer data after a few minutes for unknown reason

I have created a watchOS app that fetches data from the accelerometer and gyroscope and sends it to a socket server. The server just prints the data on the console. The socket server is made in python. My app works fine for a few minutes but then stops working.
I tested by creating an iPhone app and it's working fine. The problem is with the apple watch app.
Can someone help me with this? I don't understand what's going on.
server.py
import socket
localIP = ""
localPort = 20001
bufferSize = 10240
msgFromServer = "Hello UDP Client"
bytesToSend = str.encode(msgFromServer)
# Create a datagram socket
UDPServerSocket = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
# Bind to address and ip
UDPServerSocket.bind((localIP, localPort))
print("UDP server up and listening")
# Listen for incoming datagrams
while(True):
bytesAddressPair = UDPServerSocket.recvfrom(bufferSize)
message = bytesAddressPair[0]
address = bytesAddressPair[1]
clientMsg = "Message from Client:{}".format(message)
clientIP = "Client IP Address:{}".format(address)
print(clientMsg)
print(clientIP)
# Sending a reply to client
UDPServerSocket.sendto(bytesToSend, address)
InterfaceController.swift
#IBOutlet weak var stopButton: WKInterfaceButton!
#IBOutlet weak var startButton: WKInterfaceButton!
var session = WKExtendedRuntimeSession()
private var host: NWEndpoint.Host = "172.16.105.162"
private var port: NWEndpoint.Port = 20001
private var updateTimeInterval: Double = 1/60
override func awake(withContext context: Any?) {
// Configure interface objects here.
// startButton.setHidden(false)
startButton.setEnabled(true) // // stopButton.setHidden(true)
stopButton.setEnabled(false)
setUPSession()
}
override func willActivate() {
// This method is called when watch view controller is about to be visible to user
print("ACTIVATE")
}
override func didDeactivate() {
// This method is called when watch view controller is no longer visible
print("DEACTIVATE")
}
#IBAction func actionStop() { // startButton.setHidden(false)
startButton.setEnabled(true) // // stopButton.setHidden(true)
stopButton.setEnabled(false)
self.stopSession()
// MotionDatafetcher.shared.startSession() //stopWorkoutSEssion() //stopFetch()
}
#IBAction func actionStart() { // startButton.setHidden(true)
startButton.setEnabled(false) // // stopButton.setHidden(false)
stopButton.setEnabled(true)
self.startSession()
// MotionDatafetcher.shared.startFetch()
//MotionDatafetcher.shared.stopSession() //startWorkoutSession() //startDeviceMotionFetch()
}
}
extension InterfaceController : WKExtendedRuntimeSessionDelegate{
func setUPSession() {
// Create the session object.
session = WKExtendedRuntimeSession()
// Assign the delegate.
session.delegate = self
MySocketManager.shared.setUpConn()
}
func startSession() {
session.start()
MySocketManager.shared.connectToUDP(host, port)
MotionDatafetcher.shared.startDeviceMotionFetch(updateTimeInterval)
}
func stopSession() {
session.invalidate()
MotionDatafetcher.shared.stopFetch()
MySocketManager.shared.cancelConnToUDP()
}
func extendedRuntimeSession(_ extendedRuntimeSession: WKExtendedRuntimeSession, didInvalidateWith reason: WKExtendedRuntimeSessionInvalidationReason, error: Error?) {
}
func extendedRuntimeSessionDidStart(_ extendedRuntimeSession: WKExtendedRuntimeSession) {
}
func extendedRuntimeSessionWillExpire(_ extendedRuntimeSession: WKExtendedRuntimeSession) {
self.stopSession()
}
}

.py file executed by C# process not waiting to finish

I want to run .py file from my C# project, and get the result. The python script is making an API request, and returns an auth_key token, which I want to use in my C# code. The only problem is that, for some reason the C# code doesn't wait for the process to finish, and thus that not every account has auth_key. Here is my C# code.
private static void GenerateTokens()
{
var url = ConfigurationManager.AppSetting[GeSettingsNode() + ":ip"];
for (int i = 0; i < accounts.Count; i++)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = ConfigurationManager.AppSetting["PythonPath"];
start.Arguments = string.Format($"python_operation_processor.py {accounts[i].client_key_id} {accounts[i].key_sercret_part} {url}");
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
Process process = Process.Start(start);
using (StreamReader reader = process.StandardOutput)
{
accounts[i].auth_key = reader.ReadToEnd().Trim();
}
}
}
And here is my Python script ( python_operation_processor.py )that's making the API requests.
if __name__ == '__main__':
client_key_id = sys.argv[1]
client_secret = sys.argv[2]
API_URL = sys.argv[3]
nonce = str(uuid.uuid4())
d = datetime.datetime.now() - datetime.timedelta(hours=3)
timestamp = d.strftime('%Y-%m-%dT%H:%M:%S.%f')[:-3] + 'Z'
signature = b64encode(hmac.new(b64decode(client_secret), msg=bytes(client_key_id + nonce + timestamp, 'utf-8'),
digestmod=hashlib.sha256).digest()).decode('utf-8')
r = requests.post(API_URL + '/v1/authenticate',
json={'client_key_id': client_key_id, 'timestamp': timestamp, 'nonce': nonce,
'signature': signature})
if r.status_code != 200:
raise Exception('Failed to authenticate: ' + r.text)
auth_token = r.json()['token']
print(auth_token)
Do you have any idea, how I can wait for the execution of every process, and get the token for every account ?
I recently created something similar and ended up with this because, whilst waiting for the process is easy, it is tricky to get the output stream filled correctly.
The method presented also allow you to display the output into a textblock or similar in your application.
If you use it like this, the token will be written to the StringBuilder, and used as return value.
private async Task<string> RunCommand(string fileName, string args)
{
var timeoutSignal = new CancellationTokenSource(TimeSpan.FromMinutes(3));
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = fileName;
start.Arguments = string.Format("{0}", args);
start.RedirectStandardOutput = true;
start.RedirectStandardError = true;
start.UseShellExecute = false;
start.CreateNoWindow = true;
var sb = new StringBuilder();
using (Process process = new Process())
{
process.StartInfo = start;
process.OutputDataReceived += (sender, eventArgs) =>
{
sb.AppendLine(eventArgs.Data); //allow other stuff as well
};
process.ErrorDataReceived += (sender, eventArgs) => {};
if (process.Start())
{
process.EnableRaisingEvents = true;
process.BeginOutputReadLine();
process.BeginErrorReadLine();
await process.WaitForExitAsync(timeoutSignal.Token);
//allow std out to be flushed
await Task.Delay(100);
}
}
return sb.ToString();
}
To render this to a textblock in a UI application, you'll need to:
implement an event which signals a new line has been read, which means forwarding the process.OutputDataReceived event.
if your thinking about a live feed, make sure you flush the stdio buffer in python setting flush to true: print(""hello world"", flush=True)
If you're using an older .net version; you can implement the WaitForExitAsync as described here: https://stackoverflow.com/a/17936541/2416958 as an extention method:
public static class ProcessHelpers
{
public static Task<bool> WaitForExitAsync(this Process process, TimeSpan timeout)
{
ManualResetEvent processWaitObject = new ManualResetEvent(false);
processWaitObject.SafeWaitHandle = new SafeWaitHandle(process.Handle, false);
TaskCompletionSource<bool> tcs = new TaskCompletionSource<bool>();
RegisteredWaitHandle registeredProcessWaitHandle = null;
registeredProcessWaitHandle = ThreadPool.RegisterWaitForSingleObject(
processWaitObject,
delegate(object state, bool timedOut)
{
if (!timedOut)
{
registeredProcessWaitHandle.Unregister(null);
}
processWaitObject.Dispose();
tcs.SetResult(!timedOut);
},
null /* state */,
timeout,
true /* executeOnlyOnce */);
return tcs.Task;
}
}

Is there any possiblity to exit socket.receive() if it is empty

I am running my server using python like this:
Running multiple sockets using asyncio in python
c# client code:
Task class for calling server multiple times which run continously:
SocketConnector socketConnector = new SocketConnector();
while (true) {
socketConnector.establishConnection(60001);
Task task1 = Task.Factory.StartNew(() => doLogic(1, socketConnector));
socketConnector.establishConnection(60002);
Task task2 = Task.Factory.StartNew(() => doLogic(2, socketConnector));
socketConnector.establishConnection(60003);
Task task3 = Task.Factory.StartNew(() => doLogic(3, socketConnector));
socketConnector.establishConnection(60004);
Task task4 = Task.Factory.StartNew(() => doLogic(4, socketConnector));
socketConnector.establishConnection(60005);
Task task5 = Task.Factory.StartNew(() => doLogic(5, socketConnector));
Task.WaitAll(task1, task2, task3,task4, task5);
}
void doLogic(int batchIdentifier,SocketConnector socketConnector)
{
GC.Collect();
z = socketConnector.ServerRequest(batchIdentifier);
///process data
}
SocketConnector class
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using System.Net.Sockets;
using System;
using System.IO;
public class SocketConnector
{
public string ip = "127.0.0.1";
public int port = 60000;
private Socket client;
[SerializeField]
private int dataOut;
private byte[] dataIn; //debugging
public SocketConnector()
{
}
public void establishConnection(int port)
{
client = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
client.Connect(ip, port);
if (!client.Connected)
{
Debug.LogError("Connection Failed");
}
}
public byte[] ServerRequest(int dataOut)
{
this.dataOut = dataOut; //debugging
this.dataIn = SendAndReceive(dataOut); //debugging
return this.dataIn;
}
private byte[] SendAndReceive(int dataOut)
{
byte[] intBytes = BitConverter.GetBytes(dataOut);
client.Send(intBytes);
byte[] bytes = new byte[1024 * 1024 * 1500];
Array.Clear(bytes, 0, bytes.Length);
byte[] z = new byte[1024 * 1024 * 1500];
int lenght = 1, byteIndex = 0;
while( (lenght = client.Receive(bytes)) != 0) // blocking point
{
Buffer.BlockCopy(bytes, 0, z, byteIndex, lenght);
byteIndex += lenght;
}
Array.Resize(ref z, byteIndex);
return z;
}
}
}
The problem that i think cause infinite loop/ block is the fact that every socket is running forever on server side so when i go for a second time to do client.Receive() and it finds nothing will just remain in an infinite waiting state. I need to use while for client.Receive() cos i saw that sometimes it is needed to come back in client.Receive to not lose some parts of the bytes send from server. Do you have any idea how i can tell to the receiver to move on if it is noting in receive without killing connection from server side.
Thank you in advance.

How to automatically start and stop a python script in nodejs?

I am developing a temperature monitoring application in a hen house with a web interface. I use two arduinos and a Raspberry.
Arduino 1: I connected a temperature / humidity sensor and an RF433Mhz transmitter.
Arduino 2: An RF433Mhz receiver is connected to it. It receives data from Arduino 1.
Raspberry: Arduino 2 is connected to my raspberry which reads the data received in the serial monitor and send them to the web page via the websockets (package ws of nodejs).
At first I wanted to read this data directly with Nodejs, but I had some problems with the installation of the serial port package.
So I changed my approach: I read the data in the serial monitor with python, write it in files, and Nodejs reads these files and sends the data to the web page.
here are the two codes I use:
Phyton script
import serial
import time
ser = serial.Serial('/dev/ttyACM0', 9600)
while True:
data = ser.readline()
if data:
t = data[0:2]
h = data[6:8]
#decode utf-8
tc = t.decode("utf-8")
hc = h.decode("utf-8")
#write the temperature in the temp file
fileTc=open('temp', 'w')
fileTc.write(str(tc))
fileTc.close
#write the humidity in the hum file
fileHc=open('hum', 'w')
fileHc.write(str(hc))
fileHc.close
#sleep
time.sleep(.1)
Nodejs Script
var express = require("express");
const WebSocket = require('ws');
const wss = new WebSocket.Server({port: 4400});
var path = require("path");
var fs = require("fs");
var sys = require("util");
var exec = require("child_process").exec;
var tempcpu = 0;
var temp = 0;
var hum = 0;
var app = express();
app.set("port", process.env.PORT || 5500);
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "ejs");
app.use('/', express.static('public'));
wss.on('connection', function connection(ws) {
ws.on('message', function incoming(message) {
console.log('received: %s', message);
});
setInterval(function(){
child1 = exec("cat /sys/class/thermal/thermal_zone0/temp",
function(error, stdout,stderr){
if (error !== null){
console.log('exec error: ' +error);
} else{
tempcpu = parseFloat(stdout)/1000;
}
});
child2 = exec("cat temp", function(error, stdout,stderr){
if (error !== null){
console.log('exec error: ' +error);
} else{
temp = parseFloat(stdout);
}
});
child3 = exec("cat hum", function(error, stdout,stderr){
if (error !== null){
console.log('exec error: ' +error);
} else{
hum = parseFloat(stdout);
}
});
var tempCPU = JSON.stringify(["cpu",tempcpu]);
var temperature = JSON.stringify(["temp",temp]);
var humidity = JSON.stringify(["hum",hum]);
ws.send(tempCPU);
ws.send(temperature);
ws.send(humidity);
}, 5000);
});
app.get("/", function(request, response) {
response.render("dashboard");
});
app.listen(app.get("port"), function() {
console.log("Server started at port " + app.get("port"));
});
for now I have to launch both scripts separately. I would like to run my python script directly from nodejs when I start the node server, and stop it when I stop my nodejs code (CTRL + C).
Do you have an idea of ​​how to do it?
What you want to achieve is spawn a new process in which you execute something from either a Node app or a Python app:
NodeJS approach: Child process
const { spawn } = require('child_process');
const pythonApp = spawn('python', ['my_python_app.py']);
Python approach: Subprocess
import subprocess
node_app = subprocess.Popen(["node","my_node_app.js"], stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
EDIT
Regarding catching the INTERRUPT (CTRL+C) signal, this can also be done in both languages; and leveraged to kill the process you spawned:
With NodeJS:
process.on('SIGINT', () => {
console.log("Caught interrupt signal");
if(pythonApp) pythonApp.exit();
});
With Python:
import sys
try:
# Your app here...
except KeyboardInterrupt:
print("Caught interrupt signal")
if node_app: node_app.kill()
sys.exit()

socketIO-client - RuntimeError: maximum recursion depth exceeded

I am trying to communicate between a node.js application and python using socket.io with the socketIO-client 0.5.1 library on the python side. Below are the two code chunks:
var http = require('http').createServer();
http.listen(6000);
var io = require('socket.io').listen(http, { log: false });
var events = require('events');
var eventEmitter = new events.EventEmitter();
var counter = 0;
io.sockets.on('connection', function (socket) {
eventEmitter.on('observe', function(message){
socket.emit('sendVoltage', {});
});
socket.on('voltage', function (data) {
console.log(counter + " - " + data.voltage);
if (counter++ < 500) {
eventEmitter.emit('observe', '');
}
});
eventEmitter.emit('observe', '');
});
#!/usr/bin/python
import time, signal, sys
from Adafruit_ADS1x15 import ADS1x15
from socketIO_client import SocketIO
ADS1115 = 0x01 # 16-bit ADC
adc = ADS1x15(ic=ADS1115)
socketIO = SocketIO('chair01', 6000)
def on_voltage_response(*args):
print 'on_voltage_response', args
def on_sendVoltage(*args):
voltage = adc.readADCSingleEnded(0, 4096, 250) / 1000
print "Emitting voltage"
socketIO.emit('voltage', {'voltage': voltage}, on_voltage_response)
socketIO.wait_for_callbacks(seconds=1)
print "Voltage emitted"
socketIO.on('sendVoltage', on_sendVoltage)
socketIO.wait(seconds=1)
When the sendVoltage message is detected the on_sendVoltage function is executed, the voltage message is emitted, but control is never returned from the emit. The next message comes in, and the next etc.with never a return. At some point, it dies with too many recursive calls.
What am I doing wrong ?
You're calling socketIO.wait_for_callbacks() in the on_sendVoltage function. Your main should instead be something like this:
state = 'waiting'
while state == 'waiting':
self.socket.wait(seconds=1)
That will keep Python running waiting forever but your callbacks will still run. You can change the state in your callbacks if you want to do something else.

Categories