We Keep Coding

Why does os.path.realpath convert c:\windows\system32 into c:/windows/syswow64?

If we take a look to https://docs.python.org/3/library/pathlib.html#pathlib.Path.resolve source code we can see the routine is calling behind the curtains https://docs.python.org/3/library/os.path.html#os.path.realpath
I'm trying to really understand how os.path.realpath works in certain directories such as c:\windows\system32, ie:
>>> from pathlib import Path
>>> Path("c:/windows/system32")
WindowsPath('c:/windows/system32')
>>> Path("c:/windows/system32").resolve()
WindowsPath('C:/Windows/SysWOW64')
>>> Path("c:/windows/system32").resolve(strict=True)
WindowsPath('C:/Windows/SysWOW64')
>>> Path("c:/windows/system32").resolve()==Path("c:/windows/system32")
False
Or directories such as c:/windows/system32/openssh where you can get "unexpected results" such as below:
>>> list(Path("c:/windows/system32/openssh").resolve().glob("*"))
[]
>>> list(Path("c:/windows/system32/openssh").glob("*"))
[]
or
>>> os.listdir(r"C:\Windows\System32\OpenSSH")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Windows\\System32\\OpenSSH'
If you do dir /a you'll get
24/09/2022 10:03 <DIR> System32
So you can see it's neither a SYMLINKD nor JUNCTION.
Could you explain how os.path.realpath works in these cases? Why can't i list/glob the content of that c:\windows\system32\openssh folder?
References: ntpath.py

On windows, os.path.realpath is using GetFinalPathNameByHandleW behind the curtains, for more info you can check here.
If we make a simple experiment to see how that routine works:
#include <windows.h>
#include <iostream>
#define MAXPATHLEN 1024
void os__getfinalpathname_impl(const std::wstring &path) {
std::wcout << L"analizing: " << path << std::endl;
HANDLE hFile;
wchar_t buf[MAXPATHLEN];
int result_length;
hFile = CreateFileW(path.c_str(), 0, 0, NULL, OPEN_EXISTING,
FILE_FLAG_BACKUP_SEMANTICS, NULL);
if (hFile == INVALID_HANDLE_VALUE) {
std::wcout << L"CreateFileW:" << path << std::endl;
}
result_length =
GetFinalPathNameByHandleW(hFile, buf, MAXPATHLEN, VOLUME_NAME_DOS);
if (!result_length) {
std::wcout << L"GetFinalPathNameByHandleW" << std::endl;
}
std::wcout << L"result: " << buf << std::endl;
}
void main(int argc, char *argv[]) {
std::wcout << L"test1" << std::endl;
os__getfinalpathname_impl(L"c:/windows/system32");
std::wcout << L"test2" << std::endl;
os__getfinalpathname_impl(L"c:\\windows\\system32");
}
If we compile it for a 32-bit target we'll get:
test1
analizing: c:/windows/system32
result: \\?\C:\Windows\SysWOW64
test2
analizing: c:\windows\system32
result: \\?\C:\Windows\SysWOW64
If we compile it for a 64-bit target we'll get:
test1
analizing: c:/windows/system32
result: \\?\C:\Windows\System32
test2
analizing: c:\windows\system32
result: \\?\C:\Windows\System32
Now, if we want to confirm the above and checking the differences in a python context, we can do that easily by just spawning a couple oneliners:
python\3.10.6-amd64\python.exe -c "from pathlib import Path; print(Path('c:/windows/system32').resolve())"
C:\Windows\System32
python\3.10.1\python.exe -c "from pathlib import Path; print(Path('c:/windows/system32').resolve())"
C:\Windows\SysWOW64
Which makes total sense accordingly what's described in the below link https://learn.microsoft.com/en-us/windows/win32/winprog64/file-system-redirector

PyBytes_FromString returning a None object

I was wondering if I could get some help. Say I've got the following function to serialize an object:
PyObject * CKKSwrapper::SerializeContext() {
std::string s;
std::ostringstream os(s);
Serial::Serialize(m_cc, os, SerType::BINARY);
const std::string tmp = os.str();
const char *msg = tmp.c_str();
std::cout << "Length of context: " << tmp.length() << "\n";
return PyBytes_FromString(msg);
}
The Boost module file has
BOOST_PYTHON_MODULE (pycrypto) {
class_<pycrypto::CKKSwrapper>("CKKSwrapper")
.def("SerializeContext", &pycrypto::CKKSwrapper::SerializeContext,
return_value_policy<manage_new_object>());
where I am managing the object.
However, when I call the method in Python, I get a None object, and the output is
import pycrypto
a = pycrypto.SerializeContext()
a is None and I get Length of context: X to my console

Can C++ program string search as fast as and/or faster than python?

I'm not sure why I'm having easier time string searching in program I wrote in python faster than a program I wrote in C++. Is there a trick I'm missing?
Generating Use Case
This is for a single line use case, however in the real use case I care about multiple lines.
#include "tchar.h"
#include "stdio.h"
#include "stdlib.h"
#include <string>
#include <sstream>
#include <iostream>
#include <fstream>
#include <ctime>
using namespace std;
void main(void){
ofstream testfile;
unsigned int line_idx = 0;
testfile.open("testfile.txt");
for(line_idx = 0; line_idx < 50000u; line_idx++)
{
if(line_idx != 43268u )
{
testfile << line_idx << " dontcare" << std::endl;
}
else
{
testfile << line_idx << " care" << std::endl;
}
}
testfile.close();
}
The regular expression
Using regular expression ^(\d*)\s(care)$
The C++ Program takes 13.954 seconds
#include "tchar.h"
#include "stdio.h"
#include "stdlib.h"
#include <string>
#include <sstream>
#include <iostream>
#include <fstream>
#include <ctime>
using namespace std;
void main(void){
double duration;
std::clock_t start;
ifstream testfile("testfile.txt", ios_base::in);
unsigned int line_idx = 0;
bool found = false;
string line;
regex ptrn("^(\\d*)\\s(care)$");
start = std::clock(); /* Debug time */
while (getline(testfile, line))
{
std::smatch matches;
if(regex_search(line, matches, ptrn))
{
found = true;
}
}
testfile.close();
duration = ( std::clock() - start ) / (double) CLOCKS_PER_SEC;
std::cout << "Found? " << (found ? "yes" : "no") << std::endl;
std::cout << " Total time: " << duration << std::endl;
}
Python Program takes 0.02200 seconds
import sys, os # to navigate and open files
import re # to search file
import time # to benchmark
ptrn = re.compile(r'^(\d*)\s(care)$', re.MULTILINE)
start = time.time()
with open('testfile.txt','r') as testfile:
filetext = testfile.read()
matches = re.findall(ptrn, filetext)
print("Found? " + "Yes" if len(matches) == 1 else "No")
end = time.time()
print("Total time", end - start)

Implemented Ratah's recommendation to 8.923
about 5 seconds improvement, by reading file to single string
double duration;
std::clock_t start;
ifstream testfile("testfile.txt", ios_base::in);
unsigned int line_idx = 0;
bool found = false;
string line;
regex ptrn("^(\\d*)\\s(care)$");
std::smatch matches;
start = std::clock(); /* Debug time */
std::string test_str((std::istreambuf_iterator<char>(testfile)),
std::istreambuf_iterator<char>());
if(regex_search(test_str, matches, ptrn))
{
found = true;
}
testfile.close();
duration = ( std::clock() - start ) / (double) CLOCKS_PER_SEC;
std::cout << "Found? " << (found ? "yes" : "no") << std::endl;
std::cout << " Total time: " << duration << std::endl;
After UKMonkey's note, reconfigured project to release which also includes \O2 and brought it down to 0.086 seconds
Thanks to Jean-Francois Fabre, Ratah, UKMonkey

Poco, child communication hangs (python interpreter is child)

I want communicate with python & lua scripts, but python not working as expected. Python hangs:
Log:
lua << 'hello!', lua >> 'hello!'
lua << 'hello!', lua >> 'hello!'
lua << 'hello!', lua >> 'hello!'
python << 'hello!',
Main C++ application:
#include <iostream>
#include "Poco/Process.h"
#include "Poco/PipeStream.h"
void test( char const* interpreter, char const* filename )
{
std::vector<std::string> args { filename };
Poco::Pipe outPipe;
Poco::Pipe inPipe;
Poco::ProcessHandle process_handle = Poco::Process::launch( interpreter, args, &inPipe, &outPipe , nullptr/*errPipe*/ );
Poco::PipeInputStream output_reader(outPipe);
Poco::PipeOutputStream input_writer(inPipe);
for(int repeat_counter=0; repeat_counter<3; ++repeat_counter)
{
auto send_str("hello!");
input_writer << send_str << std::endl;
std::cout << interpreter << " << '" << send_str << "', " );
std::cout.flush();
std::string receiv_str;
output_reader >> receiv_str;
std::cout << interpreter << " >> '" << receiv_str << "'" << std::endl;
}
}
int main()
{
test("lua","test.lua");
test("python","test.py");
return 0;
}
Lua script:
for i=1,10 do
print(io.read())
end
Python script:
for i in range(0,10):
print(raw_input())
In Terminal both scripts works identically.
Solution: For Python must use only sys.stdout.write & flush, Thanks #greatwolf

My guess is that python's print function isn't directly outputting to stdout or there's some funny business happening behind the scenes. Try replacing it with sys.stdout.write instead. Don't forget to import sys first.

How to return output from pyrun_simplefile in c code

The code is
{
char name[MAX_JSON_FIELD];
FILE *fp;
copy_cJSON(name,objs[0]);
if ( (fp= fopen(name, "r")) != 0 )
{
Py_Initialize();
PyRun_SimpleFile(fp, name);
Py_Finalize();
fclose(fp);
}
return(clonestr("return string"));
}
How can I get it to return the output of the python file instead of printing it?

I achieved this using a huge workaround. I made both C and Python read and write into a file. I didn't find a better option yet.
I found an actual solution. It consists of 2 files: "main.c" that opens the script-file "script.py" which compares two strings (here: "Hello" and "Mars") and returns the longer one. I still find it strange that it takes ~20 commands to achieve this, maybe there's a better solution out there.
[main.c]
//compile me with "gcc main.c -I/usr/include/python2.7 -lpython2.7"
//original source: "http://python.haas.homelinux.net/python_kapitel_26_003.htm"
//owner is Peter Kaiser and Johannes Ernesti who published the Book "Python" under Galileo Computing
//Translation from german with many additional (and very unprofessional) comments and slight adaption by Cupacoffee, 17.02.2015.
//bugs, grammar mistakes and wrong explainations are my contribution
#include <Python.h>
int main (int argc, char *argv[])
{
char *result;//This char will receive the return value.
PyObject *module, *func, *prm, *ret;//These are some helping variables i don't understand.
Py_Initialize();
PySys_SetPath(".");//Sets the working path to the current path
module = PyImport_ImportModule("script");//Import of the script-file, note that the actual script name is "script.py"!
if (module != 0)//Asks if the script was loaded at all.
{
func = PyObject_GetAttrString(module, "compare_function");//Opens a function within the python script. Notice that you must use a function within the python script, because otherwise you can't return anything.
prm = Py_BuildValue("(ss)", "Hello", "Mars");//The "(ss)" means two strings are passed (replace with "i" for integer for instance), the "Hello" and "Mars" are the strings i pass to the script.
ret = PyObject_CallObject(func, prm);//Returns some python object i have literally no idea about ...
result = PyString_AsString(ret);// ... but luckily there's a function to cast it back to a c-compatible char*!
printf("The Script decdided that '%s' is longer!",result);
Py_DECREF(module);//cleanup?
Py_DECREF(func);//cleanup?
Py_DECREF(prm);//cleanup?
Py_DECREF(ret);//cleanup?
}
else//No script found
{
printf("Error: No script file named \"script.py\" was found!\n");
}
Py_Finalize();
return 0;
}
[script.py]
def compare_function(a, b):#this function takes 2 parameters, they are strings
return (a if min(a) < min(b) else b)#they get compared and returned to the c-program
Good luck.
*Grumble, took me over 2 hours to format this text so I could post it.*

A friend of mine gave me some code snippets which answer the problem. I didn't want to edit the old post because these two programs are entirely new approaches; One in C and one in C++. Both use the same Python Script.
He also pointed me to the manual page of "system" [Unix: "man system"] and "popen" [Unix: "man popen"]. The second one allows interactive communication, which might be useful later.
[C-File:]
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
int callScript()
{
char *cmd = "./script.py hello world";
return WEXITSTATUS(system(cmd));
}
int main(int argc, char *argv[])
{
printf("main - argc: %d, arguments:\n", argc);
for (int i = 0; i < argc; i++)
printf("\targv[%d]: %s\n", i, argv[i]);
int ret = callScript();
printf("exit code of script %d\n", ret);
return 0;
}
[C++-File:]
#include <string>
#include <sstream>
#include <iostream>
#include <stdlib.h>
#include <sys/wait.h>
int callScript(std::string args)
{
std::string cmd = "./script.py" + args;
int ret = system(cmd.c_str());
return WEXITSTATUS(ret);
}
int main(int argc, char *argv[])
{
std::cout << "main - argc: " << argc << ", arguments:" << std::endl;
std::stringstream args;
for (int i = 0; i < argc; i++)
{
std::cout << "\targv[" << i << "]: " << argv[i] << std::endl;
if (i)
args << " " << argv[i];
}
int ret = callScript(args.str());
std::cout << "exit code of script " << ret << std::endl;
return 0;
}
[Python-Script:]
#!/usr/bin/env python
import sys
def Hello(person):
print "Hello " + person
def PrintArgs(argv):
for arg in argv:
print arg
if __name__ == "__main__":
Hello("World!")
PrintArgs(sys.argv[1:])
sys.exit(2)