I had been learning how to use raw sockets in python.
Raw sockets in python can be used to send any arbitrary binary data, to any particular interface, without any other details such as MAC address or IP address.
For example, consider the following piece of code:
import socket
sock = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(3))
sock.bind(('lo',0))
sock.send('This is a sample Raw Socket Code'.encode())
sock.close()
When this is script is executed as a root user, we can capture a packet in Wireshark (or any sniffer) in the loopback interface (since I've bound the socket to 'lo'), that contains the binary string of the above text, and nothing else (of course, this is a malformed packet).
Thing is, I can't find out an implementation of this use case in C. Every implementation of raw sockets in C that I can find, deals with some kind of address. The sendto() requires the argument (struct sockaddr*), which is a generic version of (struct sockaddr_ll*) or (struct sockaddr_in*), which eventually requires an address.
How can we make an implementation of the above use case in C? I know we can straight away use python implementation, as it is available. But I'm curious to know how to implement in C.
You can use following code for that
int send(int fd, int ifindex, const uint8_t* buf, size_t len) {
struct sockaddr_ll to;
struct sockaddr* to_ptr = (sockaddr*)&to;
to.sll_family = AF_PACKET;
to.sll_ifindex = ifindex;
to.sll_halen = 6;
memcpy(&to.sll_addr, buf, 6);
int ret = sendto(fd, buf, len, 0, to_ptr, sizeof(to));
return ret;
}
where
fd - socket id
ifindex - ifindex of the interface you want to use for sending
buf - is a buffer with a formed ethernet frame
len - len of the buffer
Related
I wrote a python script that sends and receive data to and from an Arduino. I can receive data without problems, but when I send data it seems like the Arduino is receiving only garbage.
The Arduino is connected to a linux pc via USB and the pc is running Python 3.8.
Since the original code contains a lot of unrelated stuff and may be distracting, I prepared the simplest possible sketch to demontrate the problem:
Python code:
#! /usr/bin/env python
import serial
import time
if __name__ == '__main__':
ser = serial.Serial("/dev/ttyACM0", 9600, timeout=0)
time.sleep(5) # Wait for Arduino to reset and init serial
ser.write(67)
Arduino sketch
const byte LED_PIN = 13;
void setup() {
pinMode(LED_PIN, OUTPUT);
digitalWrite(LED_PIN, LOW);
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}
}
void loop() {
if (Serial.available() > 0) {
byte b = Serial.read();
if (b == 67) {
digitalWrite(LED_PIN, HIGH);
delay(500);
digitalWrite(LED_PIN, LOW);
}
}
}
This code flashes the onboard LED when receives a byte with value 67 (uppercase C). This works with the Arduino Serial Monitor, but not when running the Python script.
I feel like my problem may be related to this question, but in that case the focus was on the user not considering the case of an empty serial input buffer, while I believe the problem is actually in the sent data. This is why I decided to leave out the receiving part and simplify the example using only the onboard led.
Update
I made it work changing the last line in the python script to:
ser.write(bytes([67]))
(note the [] added around the integer value).
Anyone can explain why this syntax produces the correct result? It seems like I'm passing a single entry array to the function bytes().
Pardon my poor skills in Pyton, I know the question is probably basic.
It's really simple; the methods you tried that worked all conform to the specification for the pyserial library.
Per the Pyserial documentation:
write(data)
Parameters: data – Data to send.
Returns: Number of bytes written.
Return type: int
Raises: SerialTimeoutException – In case a write timeout is configured for the port and the time is exceeded.
Write the bytes data to the port. This should be of type bytes (or compatible such as bytearray or memoryview). Unicode strings must be encoded (e.g. 'hello'.encode('utf-8').
This is why the b'C style worked as well as the bytes call.
Attribution: Pyserial Page
The topic of an Arduino serving data via Ethernet seems to have been discussed in a few places:
1) Arduino ethernet communication
and
2) Dumping data over ethernet with an arduino client / python server
The way I like the most is the Arduino WebClient option which was mentioned in the first post:
https://www.arduino.cc/en/Tutorial/WebClient
The second post involves some Python (2.7), but it didn't seem like the problem was solved. I was also wondering if it was easier to do with wget.
If you have an Arduino acting as a simple server giving information:
/*
Simply put out data as a server
*/
#include <SPI.h>
#include <Ethernet.h>
unsigned long current_time;
unsigned long old_time;
// Ethernet stuff
// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = {
0xDE, 0xAD, 0xBE, 0xEF, 0x12, 0x34
};
// Initialize the Ethernet server library
// with the IP address and port you want to use
// (port 80 is default for HTTP):
EthernetServer server(80);
boolean notYetConnected;
// IP Address is set here
IPAddress ip(192, 168, 3, 50);
void setup()
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
{
int i;
Serial.begin(9600);
// Ethernet option
// start the Ethernet connection and the server:
Ethernet.begin(mac, ip);
server.begin();
Serial.print("# For Ethernet connection, server is at ");
Serial.println(Ethernet.localIP());
Serial.print("# \n");
}
void loop()
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
{
int i;
current_time = millis();
// dump data every 100 ms
if ((current_time - old_time) > 100)
{
// data from sensor spoofed here
int datavalue = random(0, 100);
Serial.print(current_time);
Serial.print(",");
Serial.print(datavalue);
Serial.print("\n");
server.print(current_time);
server.print(",");
server.print(datavalue);
server.print("\n");
// get delta time
old_time = current_time;
}
}
... you can use 'wget 192.168.3.50' to get the data, which dumps to a file (default index.html).
This isn't the typical client/server thing, where a program asks for information, and then it is returned; the server just dumps data out, and you can point a web browser to the IP address, or, as seen above, use wget.
When you 'set and forget' the wget command, data is recorded pretty well. I just did a test for 1.75+ hours and got 60K+ lines (once every 100 ms), and the system works.
I've noticed that if I stop the 'wget' command, and restart it, after a few times, the wget process hangs up, and I have to reset my Arduino.
A full client-server program seems a better way as per:
https://giovanniorgantini.wordpress.com/2015/03/14/getting-data-over-the-internet-with-arduino/
... and I'll be working on this now (the original client is in C, if someone can point me to a simple python-Arduino program, otherwise, I'll be looking at a simple python client), but was wondering:
1) Why would stopping the 'wget' (control-C) cause problems in restarting the wget process, where the system hangs up with:
user#machine:$ wget 192.168.3.50
--2018-02-12 19:58:54-- http://192.168.3.50/ Connecting to 192.168.3.50:80...
One reason to stop the data stream is when you are stopping a test, or, programmatically, to start another data file.
2) Is it possible to parse wget output, so that data can be saved in files every N data points or N seconds?
The client-server method seems like the way to go, but the example above seems to work with only using a web browser, or a single command line function. This might seem a bit easier to use, for some applications.
This is a simple application, just being used to dump data from a set of sensors.
In my research, I've also seen UDP client-server:
http://www.toptechboy.com/tutorial/python-with-arduino-lesson-16-simple-client-server-configuration-over-ethernet/
Didn't know if there was a preferred way of doing this.
If you're throwing stuff over the network with no specific client in mind, I think UDP might be a better option.
As for limiting file size, I suggest using logrotate like this answer suggests - https://unix.stackexchange.com/questions/17209/how-to-limit-log-file-size-using
I wrote this snippet of Python code (using pybluez) to send raw BNEP Bluetooth packet over L2CAP. The purpose is to do some fuzzing-like testing.
BNEP_PSM = 0x000F
btSock = bluetooth.BluetoothSocket(bluetooth.L2CAP)
btSock.connect(('<some BDADDR>', BNEP_PSM))
for i in range(10):
btSock.send('<some payload>')
This is working quite fine and as expected creating multiple BNEP packet even if the payload is malformed.
Now, I'm trying to write the same function in C++ using Qt, but it is not working the same way. An excerpt of the code is the following:
QBluetoothSocket btSock(QBluetoothServiceInfo::L2capProtocol);
btSock.connectToService(QBluetoothAddress("<some BDADDR>"), QBluetoothUuid::Bnep);
QObject::connect(&btSock, &QBluetoothSocket::connected, [&btSock](){
int i = 10;
while (i--)
btSock.write("<some payload>");
});
Running it with i = 1 works just fine sending a single packet with the specified payload.
Running it with i = 10 will results in a single packet with the payload equals to ten times the specified payload.
For instance setting a payload of "AAAA" in a loop of 3 will result in the first case using Python in
+------------+----+ +------------+----+ +------------+----+
|L2CAP Header|AAAA| --> |L2CAP Header|AAAA| --> |L2CAP Header|AAAA|
+------------+----+ +------------+----+ +------------+----+
In the second case using Qt in
+------------+------------+
|L2CAP Header|AAAAAAAAAAAA|
+------------+------------+
How could I force Qt socket's write to behave like Python socket's send?
UPDATE:
Looking at the documentation it says that
The bytes are written when control goes back to the event loop
How could I force buffer to flush before going back to the event loop?
How could I force buffer to flush before going back to the event loop?
You can't, because the sending can only be done asynchronously, not synchronously.
But we can queue a flush the same way the packets are queued. Namely: send each packet after the previous one has been sent. Thus we shall send it every time the event loop has processed all other work. The idiom for that is zero-duration timers - note that this has nothing at all to do with timers, it's a weird overloading of the timer concept that really makes no sense otherwise.
int i = 10;
while (i--)
QTimer::singleShot(0, this, [this]{ m_btSocket.write("<some payload>"); });
m_btSocket must be a member of the class, and must be a value member - otherwise the code will be unsafe.
If you wish to ensure that stale packets are dumped in case of a disconnection and won't affect any subsequent connections, keep track of their generation and send only if it's current:
class Foo : public QObject {
unsigned int m_generation = {}; // unsigned: modulo math w/o overflows
QBluetoothSocket m_btSocket{QBluetoothServiceInfo::L2CAP};
...
bool isBtConnected() const { return m_btSocket::state() == QBluetoothSocket::ConnectedState; }
void sendSinglePacket(const QByteArray & data) {
if (!isBtConnected()) return;
auto gen = m_generation;
QTimer::singleShot(0, this, [this, gen, data] {
if (m_generation == gen)
m_btSocket.write(data);
});
}
Foo(QObject * parent = {}) : QObject(parent) {
connect(&m_btSocket, &QBluetoothSocket::Disconnected, this, [this]{
m_generation++; // drops all in-flight packets
});
...
}
};
I did not found a proper solution using QBluetoothSocket's methods, but I made it work with a little hack.
Just used C header sys/socket.h (I need to support POSIX compliant only OSs) and changed
btSock.write("<some payload>");
to
send(btSock.socketDescriptor(), "<some payload>", <payload length>);
I'm currently trying to implement a sendall() function in a server written in C, and a recvall() function on the corresponding client written in python.
I can get the server and the client to work together when they're both written in the same language (i.e. both in c or both in python), but I can't get it to work with the server in c and the client in python. currently, i want to send a simple string from the c server to the python client.
Server sendall() implementation in C as follows (referenced from Beej's guide):
int sendall(int socket, char * buf, int *len){
int total = 0; // how many bytes we've sent
int bytesleft = *len; // how many we have left to send
int n;
while(total < *len) {
n = send(socket, buf + total, bytesleft, 0);
if (n == -1) { break; }
total += n;
bytesleft -= n;
}
*len = total; // return number actually sent here
return n==-1?-1:0; // return -1 on failure, 0 on success
}
Calling the function:
char buf[10] = "test";
int len;
len = strlen(buf);
if (sendall(command_socket, buf, &len) == -1) {
perror("sendall");
printf("We only sent %d bytes because of the error!\n", len);
}
printf("Bytes sent: %d\n", len);
Client recvall() implementation in Python (referenced from http://stupidpythonideas.blogspot.com/2013/05/sockets-are-byte-streams-not-message.html):
def recv_one_message(sock):
lengthbuf = recvall(sock, 4)
length, = struct.unpack('!I', lengthbuf)
return recvall(sock, length)
def recvall(sock, count):
buf = ''
while count:
newbuf = sock.recv(count)
print newbuf
if not newbuf:
return None
buf += newbuf
count -= len(newbuf)
return buf
Called as:
buf = recv_one_message(command_socket)
print buf
Whenever I send a message from the C-server to the Python-client, I get a return of "None." I've traced the the response coming in on the client side -- it is getting the message I sent, but the final response is always "none" and the message won't be printed. I've also tried just returning the message instead of having the None return, which also results in nothing getting printed. Can't see where I'm going wrong, any thoughts? Thanks.
Say you have a perfectly fluent English speaker and a perfectly fluent French speaker. They will not be able to communicate with each other very well. Who is at fault? -- Whoever put the two of them together expecting them to be able to communicate without first agreeing on how they would communicate.
Your send function and your receive function implement totally different protocols. The send function requires the receiver to already know the length of the data to receive. Your receive function requires the sender to send the length prior to the data. So you cannot mix and match them because they do not use the same wire format
Here's some valuable advice gained over decades of experience: Never attempt to use TCP without first documenting the protocol you're going to use on top of TCP at the byte level. As it is, there's no way to know which function is right and which is wrong because nobody has any idea what bytes are supposed to be sent over the wire.
TCP is a byte-stream protocol, and for two programs to work correctly with a TCP connection between them, each must send precisely the stream of bytes the other expects to receive. The best way to make sure this happens is to first document that stream of bytes.
If you have no idea where to start with such documentation, have a look at the simplest existing protocol that's analogous to what you're trying to do. Possible protocols to look at it include HTTP, SMTP, IRC, and DNS.
Situation:
I have tcp client made with Python and tcp server made with Qt. I try to send bytes with my client but I can't get Qt server to read these bytes.
Using Python made client and server, everything works fine. Also I can get my Python client work with C# server with no problems.
Code for Python client:
import socket
import sys
HOST = 'localhost'
PORT = 50505
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error as msg:
sys.stderr.write("[ERROR] %s\n" % msg)
sys.exit(1)
try:
sock.connect((HOST, PORT))
except socket.error as msg:
sys.stderr.write("[ERROR] %s\n" % msg)
sys.exit(2)
sock.send(b'Hello World!\r\n')
I have tried e.q fortuneserver/fortuneclient code examples but they didn't work. Bytesavailable() is always 0.
So the question is how can I read in my Qt application that "Hello World!" line ? I just need that function which starts when a server emits newConnection() signal.
connect(tcpServer, SIGNAL(newConnection()), this, SLOT(startRead()));
UPDATE:
Part of the code for Qt server:
void Server::startRead()
{
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_4_0);
QString ipAddress;
if (blockSize == 0) {
if (tcpSocket->bytesAvailable() < (int)sizeof(quint16))
return;
in >> blockSize;
}
if (tcpSocket->bytesAvailable() < blockSize)
return;
QString nextFortune;
in >> nextFortune;
statusLabel->setText(nextFortune);
ABOVE IS FROM FORTUNE CLIENT EXAMPLE.
BELOW IS FROM ANOTHER EXAMPLE.
/*
char buffer[128] = {0};
QTcpSocket *client = tcpServer->nextPendingConnection();
qDebug() << client->bytesAvailable();
for (int i=0; i<100; i++)
{
client->read(buffer, client->bytesAvailable());
qDebug() << buffer;
std::string sString(buffer);
QString qString(sString.c_str());
statusLabel->setText(qString);
}
*/
}
That last part is rather bad. I tried to make something but I have no clue what I'm doing with Qt :)
You need to arrange for your code to read when there is data available. From your description, there is not data available yet when startRead() runs.
I assume you called QTcpServer::nextPendingConnection to get your tcpSocket in startRead()? If not, you need to do so.
Just connect the readyRead signal from your tcpSocket or client to a doRead() slot, and check bytesAvailable() and read in that slot.
This slot will get called whenever new data arrives.
You might also be interested in the disconnected() signal from your TCP socket to know when it is done sending data.
The server code you attached works this way: Every 'packet' (logical one) is consisted of a size field, and following that size worth of data.
Basically it goes like this:
Read a quint16, if there aren't
enough bytes to be read, wait for
more.
quint16 read, wait until we have
atleast the amount of the quint16
to be read.
Once we have enough, we can read in
the data.
After we understood that, even if you write the packet length like the server expects, what you're trying to do still won't work. This is because the way the server reads the QString from the socket. The client is writing a raw python string, I'm guessing it's just writing a plain old ASCII to the socket. The server is using a QDataStream to read from the socket (this is the standard way to serialize in Qt). If you look at the implementation of QString's operator<< here, it's doing much more than reading an ASCII string.
If you can't change your server, change your client. You can see the implementation of the fortune client in the examples that come with PyQt.