We Keep Coding

Working Around the Windows-numpy astype(int) Bug in Pandas

I have a codebase I've been developing on a Mac (and running on Linux machines) based largely on pandas (and therefore numpy). Very commonly I type-cast with astype(int).
Recently a Windows-based developer joined our team. In an effort to make the code base more platform-independent, we're trying to gracefully tackle this tricky issue whereby numpy uses a 32-bit type instead of the 64-bit type, which breaks longer integers.
On a Mac, we see:
ipdb> ids.astype(int)
id
1818726176 1818726176
1881879486 1881879486
2590366906 2590366906
284399109 284399109
299981685 299981685
370708200 370708200
387277023371 387277023371
387343898032 387343898032
406885699892 406885699892
5262665206 5262665206
544687374 544687374
6978317806 6978317806
Whereas on a Windows machine (in PowerShell), we see:
ipdb> ids.astype(int)
id
1818726176 1818726176
1881879486 1881879486
2590366906 -1704600390
284399109 284399109
299981685 299981685
370708200 370708200
387277023371 729966731
387343898032 796841392
406885699892 -1136193228
5262665206 967697910
544687374 544687374
6978317806 -1611616786
Other than using a sed call to change every astype(int) to astype(np.int64) (which would also require an import numpy as np at the top of every module where currently that doesn't exist), is there a way to do this?
In particular, I was hoping to map int to numpy.int64 somehow in a pandas option or something.
Thank you!

I'm not saying that this is a really good idea, but you can simply redefine int to whatever you want:
import numpy as np
x = 2384351503.0
print(np.array(x).astype(int))
#-2147483648
old_int = int
int = np.int64
print(np.array(x).astype(int))
#2384351503
int = old_int
print(np.array(x).astype(int))
#-2147483648
In the case you described I'd, however, strongly prefer to fix the source code instead of redefining standard data types. It's a one-time effort and any IDE can do it easyly.
Numpy is already implicitely imported by pandas, so it doesn't cost any additional time or resources. If you really want to avoid it (for whatever reason), you can use pd.Int64Dtype.type instead of np.int64 (see source).

Python test runner - stestr return codes

Is there any documentation for stestr return codes. I tried to look into https://stestr.readthedocs.io/en/latest/MANUAL.html
but it only talks about non zero return codes but what are the typical return codes, it returns. I am trying to use stestr list

There currently isn't any documentation of the specific returns codes in every error case and honestly the actually value of the exit code is pretty ad-hoc throughout the code base. That being said all the stestr commands will return zero if the command was successful and the value will be greater than zero to indicate an error occurred (which includes a failed test). But adding documentation would be a very welcome addition. You could start by filing an issue about it: https://github.com/mtreinish/stestr/issues/new/choose
The documentation that currently exists that I think you're referring to is about the use of the --subunit flag on several of the commands. That is there because the behavior is different with subunit output enabled. Normally stestr will return an exit code > 0 when there are test failures that occurred during the run. But if you're using the --subunit flag it will always return with 0 even if there are test failures. The only case it will return > 0 is when there was an error generating subunit or another internal error. I wanted to make sure that this difference was clear for users since it might be unexpected.

CTypes error in return value

I'm testing a very simple NASM dll (64-bit) called from ctypes. I pass a single int_64 and the function is expected to return a different int_64.
I get the same error every time:
OSError: exception: access violation writing 0x000000000000092A
where hex value translates to the value I am returning (in this case, 2346). If I change that value, the hex value changes to that value, so the problem is in the value I am returning in rax. I get the same error if I assign mov rax,2346.
I have tested this repeatedly, trying different things, and I've done a lot of research, but this seemingly simple problem is still not solved.
Here is the Python code:
def lcm_ctypes():
input_value = ctypes.c_int64(235)
hDLL = ctypes.WinDLL(r"C:/NASM_Test_Projects/While_Loop_01/While_loops-01.dll")
CallTest = hDLL.lcm
CallTest.argtypes = [ctypes.c_int64]
CallTest.restype = ctypes.c_int64
retvar = CallTest (input_value)
Here is the NASM code:
[BITS 64]
export lcm
section .data
return_val: dq 2346
section .text
finit
lcm:
push rdi
push rbp
mov rax,qword[return_val]
pop rbp
pop rdi
Thanks for any information to help solve this problem.

Your function correctly loads 2346 (0x92a) into RAX. Then execution continues into some following bytes because you didn't jmp or ret.
In this case, we can deduce that the following bytes are probably 00 00, which decodes as add byte [rax], al, hence the access violation writing 0x000000000000092A error message. (i.e. it's not a coincidence that the address it's complaining about is your constant).
As Michael Petch said, using a debugger would have found the problem.
You also don't need to save/restore rdi because you're not touching it. The Windows x86-64 calling convention has many call-clobbered registers, so for a least-common-multiple function you shouldn't need to save/restore anything, just use rax, rcx, rdx, r8, r9, and whatever else Windows lets you clobber (I forget, check the calling convention docs in the x86 tag wiki, especially Agner Fog's guide).
You should definitely use default rel at the top of your file, so the [return_val] load will use a RIP-relative addressing mode instead of absolute.
Also, finit never executes because it's before your function label. But you don't need it either. You had the same finit in your previous asm question: Passing arrays to NASM DLL, pointer value gets reset to zero, where it was also not needed and not executed. The calling convention requires that on function entry (and return), the x87 FPU is already in the state that finit puts it in, more or less. So you don't need it before executing x87 instructions like fmulp and fidivr. But you weren't doing that anyway, you were using SSE FP instructions (which is recommended, especially in 64-bit mode), which don't touch the x87 state at all.
Go read a good tutorial and some docs (some links in the x86 tag wiki) so you understand what's going on well enough to debug a problem like this on your own, or you will have a bad time writing anything more complicated. Guessing what might work doesn't work very well for asm.
From a deleted non-answer: https://www.cs.uaf.edu/2017/fall/cs301/reference/nasm_vs/ shows how to set up Visual Studio to build an executable out of C++ and NASM sources, so you can debug it.

How to emulate os.path.samefile behaviour on Windows and Python 2.7?

Given two paths I have to compare if they're pointing to the same file or not. In Unix this can be done with os.path.samefile, but as documentation states it's not available in Windows.
What's the best way to emulate this function?
It doesn't need to emulate common case. In my case there are the following simplifications:
Paths don't contain symbolic links.
Files are in the same local disk.
Now I use the following:
def samefile(path1, path2)
return os.path.normcase(os.path.normpath(path1)) == \
os.path.normcase(os.path.normpath(path2))
Is this OK?

According to issue#5985 the os.path.samefile and os.path.sameopenfile are now in py3k. I verified this on Python 3.3.0
For older versions of Python here's a way which uses the GetFileInformationByHandle function:
see_if_two_files_are_the_same_file

The os.stat system call returns a tuple with a lot of information about each file - including creation and last modification time stamps, size, file attributes. The chances of different files having the same paramters are very slim. I think it is very resonable to do:
def samefile(file1, file2):
return os.stat(file1) == os.stat(file2)

The real use-case of os.path.samefile is not symbolic links, but hard links. os.path.samefile(a, b) returns True if a and b are both hard links to the same file. They might not have the same path.

I know this is a late answer in this thread. But I use python on Windows, and ran into this issue today, found this thread, and found that os.path.samefile doesn't work for me.
So, to answer the OP, now to emulate os.path.samefile, this is how I emulate it:
# because some versions of python do not have os.path.samefile
# particularly, Windows. :(
#
def os_path_samefile(pathA, pathB):
statA = os.stat(pathA) if os.path.isfile(pathA) else None
if not statA:
return False
statB = os.stat(pathB) if os.path.isfile(pathB) else None
if not statB:
return False
return (statA.st_dev == statB.st_dev) and (statA.st_ino == statB.st_ino)
It is not as tight as possible, because I was more interested in being clear in what I was doing.
I tested this on Windows-10, using python 2.7.15.

fuse utimensat problem

I am developing fuse fs at python (with fuse-python bindings). What method I need to implement that touch correctly work? At present I have next output:
$ touch m/My\ files/d3elete1.me
touch: setting times of `m/My files/d3elete1.me': Invalid argument
File exists "d3elete1.me":
$ ls -l m/My\ files/d3elete1.me
-rw-rw-rw- 1 root root 0 Jul 28 15:28 m/My files/d3elete1.me
Also I was trying to trace system calls:
$ strace touch m/My\ files/d3elete1.me
...
open("m/My files/d3elete1.me", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK|O_LARGEFILE, 0666) = 3
dup2(3, 0) = 0
close(3) = 0
utimensat(0, NULL, NULL, 0) = -1 EINVAL (Invalid argument)
close(0) = 0
...
As you see utimensat failed. I was trying to implement empty utimens and utime but its are not even called.

Try launching fuse with the -f option. Fuse will stay in foreground and you can see errors in the console.

You must implement utimens and getattr. Not all the system calls necessarily map directly to the C calls you might be expecting. Many of them are used internally by FUSE to check and navigate your filesystem, depending on which FUSE options are set.
I believe in your case FUSE is preceding it's interpretation of utimesat to utimens, with a getattr check to verify that the requested file is present, and has the expected attributes.
Update0
This is a great coincidence. There is a comment below suggestion that the issue likes with the fact that FUSE does not support utimensat. This is not the case. I had the exact same traceback you've provided while using fuse-python on Ubuntu 10.04. I poked around a little, it would appear that the fuse-python 0.2 bindings are for FUSE 2.6, it may be that a slight change has introduced this error (FUSE is now at version 2.8). My solution was to stop using fuse-python (the code is an ugly mess), and I found an alternate binding fusepy. I've not looked back, and had no trouble since.
I highly recommend you take a look, your initialization code will be cleaner, and minimal changes are required to adapt to to the new binding. Best of all, it's only one module, and an easy read.