We Keep Coding

Why does os.path.realpath convert c:\windows\system32 into c:/windows/syswow64?

If we take a look to https://docs.python.org/3/library/pathlib.html#pathlib.Path.resolve source code we can see the routine is calling behind the curtains https://docs.python.org/3/library/os.path.html#os.path.realpath
I'm trying to really understand how os.path.realpath works in certain directories such as c:\windows\system32, ie:
>>> from pathlib import Path
>>> Path("c:/windows/system32")
WindowsPath('c:/windows/system32')
>>> Path("c:/windows/system32").resolve()
WindowsPath('C:/Windows/SysWOW64')
>>> Path("c:/windows/system32").resolve(strict=True)
WindowsPath('C:/Windows/SysWOW64')
>>> Path("c:/windows/system32").resolve()==Path("c:/windows/system32")
False
Or directories such as c:/windows/system32/openssh where you can get "unexpected results" such as below:
>>> list(Path("c:/windows/system32/openssh").resolve().glob("*"))
[]
>>> list(Path("c:/windows/system32/openssh").glob("*"))
[]
or
>>> os.listdir(r"C:\Windows\System32\OpenSSH")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Windows\\System32\\OpenSSH'
If you do dir /a you'll get
24/09/2022 10:03 <DIR> System32
So you can see it's neither a SYMLINKD nor JUNCTION.
Could you explain how os.path.realpath works in these cases? Why can't i list/glob the content of that c:\windows\system32\openssh folder?
References: ntpath.py

On windows, os.path.realpath is using GetFinalPathNameByHandleW behind the curtains, for more info you can check here.
If we make a simple experiment to see how that routine works:
#include <windows.h>
#include <iostream>
#define MAXPATHLEN 1024
void os__getfinalpathname_impl(const std::wstring &path) {
std::wcout << L"analizing: " << path << std::endl;
HANDLE hFile;
wchar_t buf[MAXPATHLEN];
int result_length;
hFile = CreateFileW(path.c_str(), 0, 0, NULL, OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS, NULL);
if (hFile == INVALID_HANDLE_VALUE) {
std::wcout << L"CreateFileW:" << path << std::endl;
}
result_length =
GetFinalPathNameByHandleW(hFile, buf, MAXPATHLEN, VOLUME_NAME_DOS);
if (!result_length) {
std::wcout << L"GetFinalPathNameByHandleW" << std::endl;
}
std::wcout << L"result: " << buf << std::endl;
}
void main(int argc, char *argv[]) {
std::wcout << L"test1" << std::endl;
os__getfinalpathname_impl(L"c:/windows/system32");
std::wcout << L"test2" << std::endl;
os__getfinalpathname_impl(L"c:\\windows\\system32");
}
If we compile it for a 32-bit target we'll get:
test1
analizing: c:/windows/system32
result: \\?\C:\Windows\SysWOW64
test2
analizing: c:\windows\system32
result: \\?\C:\Windows\SysWOW64
If we compile it for a 64-bit target we'll get:
test1
analizing: c:/windows/system32
result: \\?\C:\Windows\System32
test2
analizing: c:\windows\system32
result: \\?\C:\Windows\System32
Now, if we want to confirm the above and checking the differences in a python context, we can do that easily by just spawning a couple oneliners:
python\3.10.6-amd64\python.exe -c "from pathlib import Path; print(Path('c:/windows/system32').resolve())"
C:\Windows\System32
python\3.10.1\python.exe -c "from pathlib import Path; print(Path('c:/windows/system32').resolve())"
C:\Windows\SysWOW64
Which makes total sense accordingly what's described in the below link https://learn.microsoft.com/en-us/windows/win32/winprog64/file-system-redirector

Same BSD socket API program works in Python but not in C++ or Rust

I've been trying to listen to multicast UDP messages from my Yeelight smart bulb. At a regular interval, the bulb broadcasts its presence on IP 239.255.255.250 port 1982. Also, when sending a specific request (found on page 5 - section 3.1) on the same IP and port, the bulb will respond.
Using sockets, I am trying to establish the communication. On two machines (macOS and Linux), the Python program shown below run at the REPL works, but the Rust code and C++ code also shown below (essentially doing the same thing) doesn't. The Rust program seems to successfully receive 0 bytes twice and then hangs waiting. In the C++ program, the recv function just hangs right away, never returning.
What could be the problem? Why can only Python successfully communicate?
import socket as s
import struct
sock = s.socket(s.AF_INET, s.SOCK_DGRAM, s.IPPROTO_UDP)
sock.setsockopt(s.IPPROTO_IP, s.IP_ADD_MEMBERSHIP, struct.pack("4sL", s.inet_aton("239.255.255.250"), s.INADDR_ANY))
sock.sendto("M-SEARCH * HTTP/1.1\r\nHOST: 239.255.255.250:1982\r\nMAN: \"ssdp:discover\"\r\nST: wifi_bulb\r\n".encode('UTF-8'), ("239.255.255.250", 1982))
sock.recv(4096) # Message received successfully right away from first call
#[macro_use] extern crate log;
use socket2::{Socket, Domain, Type, Protocol, SockAddr};
use std::net::{SocketAddrV4, Ipv4Addr};
fn main() {
env_logger::init();
let yeelight_ip = Ipv4Addr::new(239, 255,255, 250);
let yeelight_port: u16 = 1982;
let socket = Socket::new(
Domain::ipv4(),
Type::dgram(),
Some(Protocol::udp())
).expect("Failed to create socket!");
socket.join_multicast_v4(
&yeelight_ip,
&Ipv4Addr::UNSPECIFIED
).expect("Unable to join multicast broadcast!");
let msg = "M-SEARCH * HTTP/1.1\r\nHOST: 239.255.255.250:1982\r\nMAN: \"ssdp:discover\"\r\nST: wifi_bulb\r\n";
match socket.send_to(
msg.as_bytes(),
&SockAddr::from(
SocketAddrV4::new(
yeelight_ip,
yeelight_port
)
)
) {
Ok(bytes_sent) => {
// Some lines of debug printing
},
Err(_) => eprintln!("Error broadcasting request for identification!")
loop {
let mut buffer = Vec::with_capacity(1024 * 1024);
let received_bytes = socket
.recv(&mut buffer)
.expect("Unable to receive message!");
debug!("Received {} bytes", received_bytes);
}
}
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <iostream>
#include <string>
#include <array>
const std::string msg = "M-SEARCH * HTTP/1.1\r\nHOST: 239.255.255.250:1982\r\nMAN: \"ssdp:discover\"\r\nST: wifi_bulb\r\n";
int main() {
auto yeelight_ip = in_addr();
inet_aton("239.255.255.250", &yeelight_ip);
unsigned int yeelight_port = 1982;
auto sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
std::cerr << "Created socket!" << std::endl;
auto mreq = ip_mreq();
mreq.imr_interface.s_addr = INADDR_ANY;
mreq.imr_multiaddr.s_addr = yeelight_ip.s_addr;
if (setsockopt(sock, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq)) != 0) {
std::cerr << "Error joining multicast!" << std::endl;
exit(errno);
}
else {
std::cerr << "Joined multicast!" << std::endl;
}
auto addr = sockaddr_in();
addr.sin_port = yeelight_port;
for (int i = 0; i < 8; i++) addr.sin_zero[i] = 0;
addr.sin_family = AF_INET;
addr.sin_addr = yeelight_ip;
if (sendto(sock, msg.c_str(), msg.length(), 0, (sockaddr*)(&addr), sizeof(addr)) == -1) {
std::cerr << "Error broadcasting request for identification!" << std::endl;
std::cerr << errno << std::endl;
}
else {
std::cerr << "Sent broadcast request for idetification!" << std::endl;
}
while (true) {
std::array<char, 1024 * 1024> buffer;
auto bytes_read = recv(sock, buffer.data(), buffer.size(), 0);
if (bytes_read == -1) {
std::cerr << "Unable to receive message!" << std::endl;
std::cerr << errno << std::endl;
}
else {
std::cerr << "Read " << bytes_read << " bytes..." << std::endl;
std::cout << std::string(buffer.begin(), buffer.begin() + bytes_read) << std::endl;
}
}
return 0;
}

The Python program works as expected due to Python nicely implementing conversions behind the scenes.
The C(++) program didn't work because the endianness of the port was wrong. I forgot that I have to use addr.sin_port = htons(yeelight_port);.
As for the Rust program, I forgot that with_capacity only allocated the internal buffer, but the Vec is still empty. The solution, described here, is to call buffer.resize(buffer.capacity(), 0);.

Stdin and stdout don't appear to be redirected to pipe

I have to run a bunch of programs in parallel, via fork(), dup2, and pipes;
(programs: rgen, A1, A2 and the driver program itself A3).
- A3 starts and executes rgen (via fork) with cml arguments;
- rgen passes stdout to A1 (via pipe 'toA1');
- A1 needs to take stdin from rgen (via pipe 'toA1'), process that and then send stdout to A2 (via pipe 'toA2');
- A2 needs to take stdin from A1 (from pipe 'toA2');
- A2 then takes that input and sends what it got from A1 to stdout (screen), it then waits for user input from A3 (via pipe 'toA2') or for input from A1 via pipe 'toA2';
- Then the driver program needs to get stdin from the keyboard and send stdout to A2 (via pipe 'toA2');
The issue is that pipe 'toA2' doesn't appear to be working. I know that A1 sends the correct output to stdout. And I know that A2 receives input from stdin. Currently A2 receives stdin from the keyboard as noted from "std::cin >> use_input" and the fact that when I run the program it doesn't do anything else unless I input something into the command line.
It should look like this in the terminal:
$./A3 -s 5 -n 4 -l 5
V 8
E {<0,2>,<0,3>,<0,4>,<1,3>,<4,7>,<5,2>,<5,6>}
s 2 4
2-0-4
where "./A3 -s 5 -n 4 -l 5" and "s 2 4" are user inputs into the command line and "V 8" and "E {...}" and "2-0-4" are all output to the screen by A2.
*fyi all of the individual programs work just fine (maybe not A3 since it's the only with the potential issue), I'm more worried about getting the information from other processes.
I've removed all of the code in my programs that are not to do with stdin or stdout and replaced it with "...". If it is necessary to show, let me know, I just thought it'd be easier to read this way since I know it works once I have then input.
Program A3 (c++):
int main(int argc, char **argv) {
std::string user_input;
std::vector<pid_t> kids;
pid_t child_pid;
char* empty_arg1[2];
empty_arg1[0] = "A1.py";
empty_arg1[1] = nullptr;
char* empty_arg2[2];
empty_arg2[0] = "A2";
empty_arg2[1] = nullptr;
int toA1[2];
int toA2[1];
pipe(toA1);
pipe(toA2);
child_pid = fork();
//runs rgen; Sends output to 'toA1'
if (child_pid == 0) {
dup2(toA1[1], STDOUT_FILENO); //Sends stdout to pipe 'toA1'
close(toA1[0]);
close(toA1[1]);
return execv("./rgen", argv);
}
kids.push_back(child_pid);
child_pid = fork();
//runs A1; Sends output to 'toA2'; Gets input from pipe 'toA1'
if (child_pid == 0) {
dup2(toA1[0], STDIN_FILENO); //A1 gets stdin from pipe 'toA1'
close(toA1[0]);
close(toA1[1]);
dup2(toA2[1], STDOUT_FILENO); //sends stdout to pipe 'toA2'
close(toA2[0]);
close(toA2[1]);
return execvp("./A1.py", empty_arg1);
}
kids.push_back(child_pid);
child_pid = fork();
//runs A2; Gets input from 'toA2'
if (child_pid == 0) {
std::string use_input = "blank";
dup2(toA2[0], STDIN_FILENO); //A2 gets stdin from pipe 'toA2'
close(toA2[0]);
close(toA2[1]);
std::cout << "use_input[1] = " << use_input << std::endl;
return execv("./A2", empty_arg2);
}
kids.push_back(child_pid);
dup2(toA2[1], STDOUT_FILENO); //Sends output to 'toA2'
close(toA2[0]);
close(toA2[1]);
while (!std::cin.eof()) {
getline(std::cin, user_input); //Assumes there will be a space after the command
std::cout << user_input << std::endl;
}
// sends kill signal to all children processes
for (int i = 0; i < kids.size(); i++) {
int status;
kill(kids[i], SIGTERM);
waitpid(kids[i], &status, 0);
}
return 0;
}
Program rgen (c++):
...
int main(int argc, char** argv) {
........
// removes pre-existing streets
std::cout << output << std::endl;
street_names.pop_back();
}
}
........
//Outputs the 'a' command to add the streets for assignment 1
std::cout << output << std::endl;
std::cout << "g" << std::endl;
sleep(wait_sec);
}
return 0;
}
Program A1 (python):
...
def main():
sys.stdout.write("From A1 forked process\n")
.....
sys.stdout.write("V " + str(len(vertex_list)) + "\n")
sys.stdout.write(str_edges + "\n")
if __name__ == '__main__':
main()
Program A2 (c++):
int main() {
std::string use_input = "NOPE";
std::cerr << "A2 is running" << std::endl;
std::cin >> use_input;
std::cout << "use_input[2] = " << use_input << std::endl;
while (!std::cin.eof()) {
getline(std::cin, user_input); //Assumes there will be a space after the command
std::istringstream iss(user_input);
iss >> command;
if (command == 'V') {
....
}
else if (command == 'E') {
...
iss >> edges;
std::cout << "V " << num_vert << std::endl;
std::cout << "E " << edges << std::endl;
...
}
else if (command == 's') {
...
iss >> str_vert >> end_vert;
....
std::cout << path;
std::cout << std::endl;
}
else {
std::cerr << "Error: Invalid command[a2]" << std::endl;
}
}
return 0;
}

Python socket not receiving all data from C++ Boost asio

In a nutshell
I am trying to send encoded and compressed image data (~300kB) from a boost::asio server (embedded system) and receive it via python socket client (laptop). The client seems to be receiving only 65532 bytes every time while the client is sending around 350000 bytes.
My thoughts are that maybe something is limiting the TCP packet size (either boost::asio or python socket or else) to around 65535 bytes.
Protocol
The client send READY, a string with length 5, to the server.
The server receives READY from the client.
The server transmit the length of the message, a 4-byte uint, to the client (the length is only of the message; i.e. excluding the length itself).
The server transmit the message to the client.
The client receives the length and message from the server.
Description
The server will create a new img_session object when a new connection is made. When a read event occurs, The handle_read() function will be called. img_stream is the zlib compressed and base64 encoded std::string which is 'synchronized' via img_lock.
The client calls run_image() once in a thread. The method will start by sending a READY signal and wait for the server to respond. The server's first response will be the length of the rest of the message. Then it will continuously receive the rest of the message and keep it in the variable message. Meanwhile, a concurrent thread will decode and decompress the message and keep the image in the variable frame.
Code
Server Code (img_session.h)
#ifndef _IMG_SESSION_H_
#define _IMG_SESSION_H_
#include <session.h>
#include <boost/bind.hpp>
#include <boost/asio.hpp>
#include <iostream>
#include <utility.h>
#include <boost/thread.hpp>
class img_session : public session
{
public:
img_session(boost::asio::io_service& io_service, std::string * img_stream, int* img_lock);
~img_session();
/* Starts the session. This is called in the constructor.*/
void start();
private:
void handle_read(const boost::system::error_code& error, size_t bytes_transferred);
void handle_write(const boost::system::error_code& error);
std::string * img_stream;
int * img_lock;
};
#endif /* _IMG_SESSION_H */
Server Code (img_session.cpp)
#include "img_session.h"
#include <session.h>
/* Public Methods */
img_session:: img_session(boost::asio::io_service& io_service, std::string * img_stream, int * img_lock) : session(io_service), img_stream(img_stream), img_lock(img_lock){
}
img_session::~img_session() { }
void img_session::start()
{
std::cout << "connection from " << _socket.remote_endpoint() << "\n";
std::cout << "img_session waiting for READY signal..." << std::endl;
_socket.async_read_some(boost::asio::buffer(_data, max_length),
boost::bind(&img_session::handle_read, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
/* Private Methods */
void img_session::handle_read(const boost::system::error_code& error, size_t bytes_transferred)
{
if (!error)
{
char otherString[6];
std::cout << "[INFO ] IMG READ" << std::endl;
if (strncmp(_data, "READY", 5) == 0) {
std::cout << "[INFO ] READY signal received..." << std::endl;
while (*img_lock == 1) boost::this_thread::yield();
*img_lock = 1;
std::string message = std::string(*img_stream);
unsigned int len = message.length();
std::cout << "len = " << len << std::endl;
unsigned char bytes[4];
bytes[0] = (len >> 24) & 0xFF;
bytes[1] = (len >> 16) & 0xFF;
bytes[2] = (len >> 8) & 0xFF;
bytes[3] = len & 0xFF;
message = std::string((char* ) bytes, 4) + message;
std::cout << "img_session.cpp: bytes[] = " << (int) bytes[0] << " " << (int)bytes[1] << " " << (int)bytes[2] << " " << (int)bytes[3] << std::endl;
std::cout << "img_session.cpp: message.length() = " << message.length() << std::endl;
std::cout << "img_session.cpp: HEAD: " << message.substr(0, 1024) << std::endl;
std::cout << "img_session.cpp: TAIL: " << message.substr(message.length() - 1024, message.length() - 1);
boost::asio::async_write(_socket,
boost::asio::buffer(message.c_str(), message.length()),
boost::bind(&img_session::handle_write, this,
boost::asio::placeholders::error));
*img_lock = 0;
}
else {
std::cout << "[INFO ] READY signal not received..." << std::endl;
boost::asio::async_write(_socket,
boost::asio::buffer("", 1),
boost::bind(&img_session::handle_write, this,
boost::asio::placeholders::error));
}
clean();
}
else
{
delete this;
}
}
void img_session::handle_write(const boost::system::error_code& error)
{
if (!error)
{
std::cout << "img_session waiting for READY signal..." << std::endl;
_socket.async_read_some(boost::asio::buffer(_data, max_length),
boost::bind(&img_session::handle_read, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
else
{
delete this;
}
}
Client Code (controller.py)
...
def run_image(hostname, port, interrupt):
global frame
sck = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sck.connect((hostname, port))
sck.settimeout(100)
message = None
while(interrupt.is_set()):
buffer = message
t = threading.Thread(target=decodeImg, args=(buffer,))
t.start()
answer = b" "
message = b""
sck.send(b"READY")
print("Sent READY signal")
answer = sck.recv(4)
num = int(answer[0] << 24) + int(answer[1] << 16) + int(answer[2] << 8) + int(answer[3])
print("num:", num)
try:
for i in range(math.ceil(num/1024)):
print("Receiving... (", i,"of",math.ceil(num/1024),")")
answer = sck.recv(1024) # row size
if (i == 0):
print("#### HEAD ####")
print(answer)
message += answer
except socket.timeout:
pass
print("#### TAIL ####")
print(answer)
print("###########################################")
print("##### Received! message (len=", len(message),") #####")
print("###########################################")
t.join()
#print("Updated")
sck.close()
print("run_image exited")
decompress = zlib.decompress
decode = base64.b64decode
toframe = pygame.image.fromstring
def decodeImg(raw):
if (raw is None or raw == ""):
return
global frame
print("Message length :", len(raw))
raw = decode(raw)
print("Decoded length :", len(raw))
raw = decompress(raw)
print("Decompressed length:", len(raw))
frame = toframe(raw, image_size, 'RGB')
if (frame is None):
print("ERR None frame")
...
(Full client code is at https://pastebin.com/ut6L70XM)
Response
Server's log
...
connection from 127.0.0.1:61616
img_session waiting for READY signal...
[DEBUG] server.h: instance of img_session found
[INFO ] IMG READ
[INFO ] READY signal received...
[INFO ] img_stream.length() = 349544
len = 349544
img_session.cpp: bytes[] = 0 5 85 104
img_session.cpp: message.length() = 349548
img_session.cpp: HEAD: UheJy0vYGZpLgSrIsJMkEmyARMwASZIBNkAiZgQpnQJowJY8Ix4ZUyI
jITqnrPnnvv62V7KBBCUP3xKyJTYtvKdv8p6Td+2u1jeR5gP/W5oaSVtVRuKh8FvhyTV7095VkDz1u+V
ZVrbR8nq79cxfbtFI/Pv5/stmcvpbDJVc0sreC8a1fZarPf7722mddXuBeNLPZTsRfFCiusqrzVdw3vO
1wqjl2rONf6VO0O1Lq1dSCqKmulbLbwp64jqhVchzaUuS3r8Lax2rLpRJWfV2MKm6fL8SuIC6z2x5CLq
VX5o58xrhTNbjjpvu4edjWW8Yt6X8Yqs5ZNK9iCu7eXWm1LteW9/X0D/ZCKG871OByLGmC3ay3vjU3F3
rsaitmdbFar3dL1VduS7nBlDWt7LXu1b8m3V9SwVvZW2/7+3Nr+/r+uXY01vy9lLVZmLdu7YFsre92Hr
b+XrRylHK0eRz1aO+p7qX2vx96wvNd7q/1o63fVeqmjvbfYxrbKj2NtGdV+71qOtKy9+9jXmUdpo+/js
OW9sar8e1dL6+O92Mpuh7dbnR1lViVt/Y6Tro39vYy9t/1drOvjKvn+2HdbWh+rGdj7bsYqbCddB2L7f
vTdVt4fsfKuoR8dy7vCVc8x5rGaus57jHasSxv9XWx9fP9+r8+DW+b7nH28D+32+73MPrE+bOO0LcOW9
8f35mOtzPf2MdZGlZ/Vtmt5l8PhOHDiqLXBtpza+F7KmPvgRhRGtUOHHKp2tyYNtootPHBdOOMx38t5j
HqM4te789rfy/sGHrxXuF3vjb2tjbh16+69b/tm6/sqj9u4bjWL2eGo7X2WceC8645V+43t3bbbxzn89
rK221XUdRPsRg0rOXiXum3B3XivzHHyhhxz1dPfV3mui8WucU5bxvsujmvYlvf6NVeBc66P/b1MFFjHv
rfs65avwnNOHD77+yzn+irnuXcWu/p52oroVjLmHjR5bKm3YjcA/bKdp/hCKVX4laoJrL9C7yuUP8vc1
1lz/Ydj
img_session.cpp: TAIL: sC5fp569IQ/fnp9NsY4EfMdjx8cR77ftC8426c2vPM27KFV/q+Crrf1tH
/fnYt+KoafkL+S9PaSs96I4qAmH7fHK/p2qcY6IFrOdR14uAs87ssSvGTw+rx/ib0ci+kXMpRGd2D2Sx
cHWl4rCb42a3MHfFyUzlPUTwhm5rA+DWF1Z6JPUtpwUqX90KiWpYMzOer1mRRv5K1yOUPHB36F/H6PjY
Hw4NkqwQb0WHrJn2ws85PB3owhbaZ4iECNkp2D0aylzFBMLVOK4USwzpKFWQZpFXd2lOm8eicRyLAuKL
y3Gou2FvyGELfezceNR8I31QOXroJiuOOH9OMhSpxVgmPGlQ4XMXw5by49xfWOSrSIlVVOoBlUxAnp9u
xwSTPras50W+LMZAE6iSe26qbDbAPcyfeyQpIQ+91BTfkKdYbcaJBWH35RQ8qVLKKuoMaAvQlnIBms4l
XNeiFJesperkO000isv5NQ48FkQZ1Ua/w7AqRI6M1EEGuc4sSw7GdFmibbpNMGUnyjCxtkGQzjnaouUr
2o+ajPq3JSM2pt03zroAyzDs1axEu05eHWd5WDo5ZC0rvEd8RR496IsuxuzWCyeNszWWM9lQOGMAmtI4
Dm1I/egQrD6knzxq3sY2oXHNbJk1i5At0YpbLrOntb1AgHKcuyxZ/pXBvIAMpDac/LVXtNhXvA6Hel2Z
PvwOBj2GaXgpt7Evaj/Vm6WeePzRrQ+nQfzYDPUHkr5/B0rGvgQKrgi/Hz+ehT4zLPgu9GCHvmNIYStZ
fC7L7luR1NVhtBwKmSuQNip9DjfAe9/0a0k2fkx0Vko4HKijDkh62YmXF5YTBgG3jSgmRv/qvbeIZwP/
NsG+HIdYWGoqmxX0C5PamF936VxCd0bYz8rtjnf1v35pj8soTpHasDdfTFsvGuuRdMtFLycccHIVgK3I
gX9UnpZIjQnaeD2HpK2vubAyVc8CrIXAzdemYjOXHTo3EeI3/7RCR3yDA7qDsZIkNWj7vKl84XsQeH/A
X1TLNQ=
...
Client's log
Sent READY signal
num: 349544
Receiving... ( 0 of 342 )
#### HEAD ####
b'eJy0vYGZpLgSrIsJMkEmyARMwASZIBNkAiZgQpnQJowJY8Ix4ZUyIjITqnrPnnvv62V7KBBCUP3xKyJTYtvKdv8p6Td+2u1jeR5gP/W5oaSVtVRuKh8FvhyTV7095VkDz1u+VZVrbR8nq79cxfbtFI/Pv5/stmcvpbDJVc0sreC8a1fZarPf7722mddXuBeNLPZTsRfFCiusqrzVdw3vO1wqjl2rONf6VO0O1Lq1dSCqKmulbLbwp64jqhVchzaUuS3r8Lax2rLpRJWfV2MKm6fL8SuIC6z2x5CLqVX5o58xrhTNbjjpvu4edjWW8Yt6X8Yqs5ZNK9iCu7eXWm1LteW9/X0D/ZCKG871OByLGmC3ay3vjU3F3rsaitmdbFar3dL1VduS7nBlDWt7LXu1b8m3V9SwVvZW2/7+3Nr+/r+uXY01vy9lLVZmLdu7YFsre92Hrb+XrRylHK0eRz1aO+p7qX2vx96wvNd7q/1o63fVeqmjvbfYxrbKj2NtGdV+71qOtKy9+9jXmUdpo+/jsOW9sar8e1dL6+O92Mpuh7dbnR1lViVt/Y6Tro39vYy9t/1drOvjKvn+2HdbWh+rGdj7bsYqbCddB2L7fvTdVt4fsfKuoR8dy7vCVc8x5rGaus57jHasSxv9XWx9fP9+r8+DW+b7nH28D+32+73MPrE+bOO0LcOW98f35mOtzPf2MdZGlZ/Vtmt5l8PhOHDiqLXBtpza+F7KmPvgRhRGtUOHHKp2tyYNtootPHBdOOMx38t5jHqM4te789rfy/sGHrxXuF3vjb2tjbh16+69b/tm6/sqj9u4bjWL2eGo7X2WceC8645V+43t3bbbxzn89rK221XUdRPsRg0rOXiXum3B3XivzHHyhhxz1dPfV3mui8WucU5bxvsujmvYlvf6NVeBc66P/b1MFFjHvrfs65avwnNOHD77+yzn+irnuXcWu/p52oroVjLmHjR5bKm3YjcA/bKdp/hCKVX4laoJrL9C7yuUP8vc11lz/Ydjfr/M'
...
Receiving... ( 63 of 342 )
b'JGPgXARwxzUdiKKMzTRc4OaHzidOcNJrBQHbD+M6nZ6PKBrUlsRiq9KhIQY/JTOVqSd5ufKNbatfImARAtARU6pbAa2mh0YHVQrR04zzmEVA5Pnrc4Gt4kUiaWzVqu/Nv7tWw63102hqqZHgKyrXxsBq8emZeBQCriWU18xHfJZ4Fwq6+8cs0rMdIeCKitrmHTO6gFcZ1W3Q1zTDLQ6rbpMNOhSTBjIQrOwv+7xLiykV3pxd4SpDFC+JxcWaX5mUXWVBI7PLjG5EkKM0GAHgkfm8ynnGLITQs97JCiW/KwLEc+ZVW0IOrwJxiw3ZDnr4z7kRtOnjoX9ZiDSCvgwHP6mCn09UB+9AsCVnhSlt/TpQo0T+ShQnmXDwNpXLPSAQFCxT+6CDTYROLxKRWI6eCQlz57ipCo7OrIuZYFLQdzaATzI62KStlL8tv/CWbwmLUdsnul0T5hyyB/h7nFeiodJy7E5rStiJfCJXxwLl7zv2y/cDI0Apa37CfmjK4SO9ns7qcPgeoYt9zZcGLbeMULcaNMAYL3653y4Wy3ZO2dHKiPv6v8vg/Z+j7yDSy8IWHuC4bYB35eHfEei08vZDUpexFVfB8HEsyDJSIN4/7/uINS/0ptkjI+vi8DJiyijSO/xHfvInih/DS9aEvBv99pLhk7+RV/5GFD0pgcQPBYm0nwmLgE0PG9b/h8LQ+FvlPwf7Sp346Adq5nN6/nNXrhSN2RmpNfDXeUvX3mqRz/3DA6TnvZ16VGOCEgrag9DZHG7hZAZwE3brJ+Wl+BLRRMzvGpNq2mHqGU0cmeSdfJyAvciWWd1XQ2J2/JlGMu4AN7VfjqRiE9ctjQ2kp1sl2npXPKIqbhsDJ30cM4E7yOjIjjNs8yOOIZfb859t/y5sk2GueLHypprjHk2nlfzcKbQrQ95qdVVVOIyuGqz89XqiCqAui8K+CBP3pgjyhGwo9MXSqyCZrf8zrWtMaTQhGPJ5EWcfaf5BK/hlEZJiwTEX0oNuNFUw/Ge50BvJ'
Receiving... ( 64 of 342 )
b''
Receiving... ( 65 of 342 )
b''
...
Receiving... ( 340 of 342 )
b''
Receiving... ( 341 of 342 )
b''
#### TAIL ####
b''
###########################################
##### Received! message (len= 65532 ) #####
###########################################
Message length : 65532
Decoded length : 49149
Exception in thread Thread-4:
Traceback (most recent call last):
File "C:\Python35\lib\threading.py", line 914, in _bootstrap_inner
self.run()
File "C:\Python35\lib\threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "C:\Python35\controller.py", line 204, in decodeImg
raw = decompress(raw)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

You're sending from a local variable, but it goes out of scope before writing completes.
Make message a member variable.
NOTE: The img_lock is not thread safe
Here's a working demo:
Live On Coliru
#ifndef _IMG_SESSION_H_
#define _IMG_SESSION_H_
#include <boost/bind.hpp>
#include <boost/asio.hpp>
#include <iostream>
#include <boost/thread.hpp>
struct session {
session(boost::asio::io_service& svc) : svc(svc) {
}
void clean() {}
boost::asio::io_service& svc;
boost::asio::ip::tcp::socket _socket{svc};
int max_length = 1024;
//std::array<char, 1024> _data;
char _data[1024];
};
class img_session : public session
{
public:
img_session(boost::asio::io_service& io_service, std::string * img_stream, int* img_lock);
~img_session();
/* Starts the session. This is called in the constructor.*/
void start();
private:
void handle_read(const boost::system::error_code& error, size_t bytes_transferred);
void handle_write(const boost::system::error_code& error, size_t bytes_transferred);
std::string * img_stream;
int * img_lock;
std::string message;
};
#endif /* _IMG_SESSION_H */
/* Public Methods */
img_session:: img_session(boost::asio::io_service& io_service, std::string * img_stream, int * img_lock) : session(io_service), img_stream(img_stream), img_lock(img_lock){
}
img_session::~img_session() { }
void img_session::start()
{
std::cout << "connection from " << _socket.remote_endpoint() << "\n";
std::cout << "img_session waiting for READY signal..." << std::endl;
_socket.async_read_some(boost::asio::buffer(_data, max_length),
boost::bind(&img_session::handle_read, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
/* Private Methods */
void img_session::handle_read(const boost::system::error_code& error, size_t /*bytes_transferred*/)
{
if (!error)
{
//char otherString[6];
std::cout << "[INFO ] IMG READ" << std::endl;
if (strncmp(_data, "READY", 5) == 0) {
std::cout << "[INFO ] READY signal received..." << std::endl;
while (*img_lock == 1) boost::this_thread::yield();
*img_lock = 1;
message = "0000" + *img_stream;
size_t len = message.length();
std::cout << "len = " << len << std::endl;
message[0] = (len >> 24) & 0xFF;
message[1] = (len >> 16) & 0xFF;
message[2] = (len >> 8) & 0xFF;
message[3] = len & 0xFF;
std::cout << "img_session.cpp: bytes[] = " << (int) message[0] << " " << (int)message[1] << " " << (int)message[2] << " " << (int)message[3] << std::endl;
std::cout << "img_session.cpp: message.length() = " << message.length() << std::endl;
std::cout << "img_session.cpp: HEAD: " << message.substr(4, 1024) << std::endl;
std::cout << "img_session.cpp: TAIL: " << message.substr(message.length() - 1024, message.length() - 1);
boost::asio::async_write(_socket,
boost::asio::buffer(message),
boost::bind(&img_session::handle_write, this,
boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
*img_lock = 0;
}
else {
std::cout << "[INFO ] READY signal not received..." << std::endl;
/*
*boost::asio::async_write(_socket,
* boost::asio::buffer("", 1),
* boost::bind(&img_session::handle_write, this,
* boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
*/
}
clean();
}
else
{
delete this;
}
}
void img_session::handle_write(const boost::system::error_code& error, size_t bytes_transferred)
{
if (!error)
{
std::cout << "img_session waiting for READY signal..." << std::endl;
_socket.async_read_some(boost::asio::buffer(_data, max_length),
boost::bind(&img_session::handle_read, this,
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
else
{
std::cout << __FUNCTION__ << ":" << error.message() << "\n";
delete this;
}
}
int main() {
boost::asio::io_service svc;
std::string img_stream(300<<10, '*');
int img_lock = 0;
auto s = new img_session(svc, &img_stream, &img_lock);
using boost::asio::ip::tcp;
tcp::acceptor a(svc);
a.open(tcp::v4());
a.set_option(tcp::acceptor::reuse_address(true));
a.bind({{}, 6767});
a.listen(1);
a.accept(s->_socket);
s->start();
svc.run();
}
This, when using a client
netcat localhost 6767 | wc
READY
Prints
0 1 307204

How can I store and interact with multiple individual python sessions in c++?

Within my c++ program, I have a class called "pEngine" that is supposed to handle interaction between my program and a python session. There can be any number of these pEngines and each one needs to be working individually since during their initialization they each execute a different python script (unknown till run time).
Currently I have each script defining a main function that takes a single variable, the payload, and depending on the script produces a different output which my program pipes back. The pEngine is then deleted. For a single pEngine my code works great allowing me to preload data in the script for the function, then call it later when needed.
My problem is that since it closes its session, all other pEngines are effected. I cannot load multiple scripts into the same session since they all have have the same prototype for their main function. What I was looking for was a way to have a python session instance object that is separate and individual so I could run commands against it.
My pEngine.h
#include <cstring>
#include "Engine.h"
#include "/usr/include/python2.7/Python.h"
#ifndef PENGINE_H
#define PENGINE_H
class pEngine : public Engine
{
public:
pEngine();
~pEngine();
std::string execute(const std::string& cmd);
int setwd(const std::string& dir);
int source(const std::string& script);
std::string main(const std::string& payload);
private:
PyObject *pModule;
PyObject *stdOut;
};
#endif /* PENGINE_H */
My pEngine.cpp
#include "pEngine.h"
pEngine::pEngine()
{
/* Start a python session and create a class to capture the standard out
text. Define it as the standard out location and run this code in the
session. Then store a pointer to this python object to extract returned
values. */
Py_Initialize();
pModule = PyImport_AddModule("__main__"); //create main module
std::stringstream pythonOutputs;
pythonOutputs << "import sys" << "\n"
<< "class CatchStdOut:" << "\n"
<< "\t" << "def __init__(self):" << "\n"
<< "\t" << "\t" << "self.value = ''" << "\n"
<< "\t" << "def write(self, txt):" << "\n"
<< "\t" << "\t" << "self.value += txt" << "\n"
<< "\t" << "def clear(self):" << "\n"
<< "\t" << "\t" << "self.value = ''" << "\n"
<< "StdOut = CatchStdOut()" << "\n"
<< "sys.stdout = StdOut";
PyRun_SimpleString((pythonOutputs.str()).c_str());
stdOut = PyObject_GetAttrString(pModule,"StdOut");
}
pEngine::~pEngine(){
Py_Finalize();
}
std::string pEngine::execute(const std::string& cmd)
{
/* Place cmd in try except statement to protect against unknown errors. */
std::stringstream pString;
pString << "program_status = 200" << "\n"
<< "StdOut.clear()" << "\n"
<< "try:" << "\n"
<< "\t" << cmd << "\n"
<< "except Exception as e:" << "\n"
<< "\t" << "program_status = 400" << "\n"
<< "\t" << "print(e)";
/* Run Command against the interpreter and store returned answer. */
PyRun_SimpleString( (pString.str()).c_str() );
PyErr_Print(); //make python print
//returned payload
std::string RPL =
PyString_AsString(PyObject_GetAttrString(stdOut,"value"));
/* Clear the payload from the capture object and run a command to fill in
the status. */
std::stringstream statusCollector;
statusCollector << "StdOut.clear()\n"
<< "print program_status";
/* Run command and extract status. */
PyRun_SimpleString( (statusCollector.str()).c_str() );
PyErr_Print();
//status
std::string PST =
PyString_AsString(PyObject_GetAttrString(stdOut,"value"));
if (!RPL.empty() && RPL[RPL.length()-1] == '\n') RPL.erase(RPL.length()-1);
if (!PST.empty() && PST[PST.length()-1] == '\n') PST.erase(PST.length()-1);
/* Set Program status and return the payload from the python session. */
program_status = std::stoi( PST );
return RPL;
}
std::string pEngine::main(const std::string& payload)
{
std::string returned = execute( "print main("+toEscapedString(payload)+")");
Py_Finalize();
if( program_status != 200 ) status = Engine::ENG_EXECUTE_ERROR;
return returned;
}
int pEngine::setwd(const std::string& dir)
{
std::string pString = "import os\n\tos.chdir(" + toEscapedString(dir) + ")";
execute( pString );
if( program_status != 200 ) status = Engine::ENG_SESSION_ERROR;
return status;
}
int pEngine::source(const std::string& script)
{
std::string pString = "execfile(" + toEscapedString(script) + ")";
execute( pString );
if( program_status != 200 ) status = Engine::ENG_SESSION_ERROR;
return status;
}