We Keep Coding

What is the replacement for python IN package?

I am trying to use a code which was written for python 2 and may run with python 3.6.0, but it does not run with python 3.6.4. It imports the IN module, and uses IN.IP_RECVERR. I tried to google it, but it is a 'bit' hard to find anything about a module called IN (naming fail?). To demonstrate in REPL, that it works in python 2, but not in 3.6.4:
$ python2
Python 2.7.14 (default, Jan 5 2018, 10:41:29)
[GCC 7.2.1 20171224] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import IN
>>> IN.IP_RECVERR
11
>>>
$ python3
Python 3.6.4 (default, Jan 5 2018, 02:35:40)
[GCC 7.2.1 20171224] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import IN
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'IN'
>>>
What is the replacement for this IN module in newer versions of python 3?

This is presumably the private plat-linux/IN.py module, which was never intended to be used. There have been plans to remove these plat-* files for a few zillion years, but it looks like it finally happened in issue 28027 for 3.6. As mentioned in What's New in Python 3.6:
The undocumented IN, CDROM, DLFCN, TYPES, CDIO, and STROPTS modules have been removed. They had been available in the platform specific Lib/plat-*/ directories, but were chronically out of date, inconsistently available across platforms, and unmaintained. The script that created these modules is still available in the source distribution at Tools/scripts/h2py.py.
Most of the useful constants that are at least somewhat portable (as in you can expect them to be available and work the same on your old laptop's linux and your brand-new Galaxy's linux, if not on OS X or Solaris) have long been made available through other places in the stdlib.
I think this specific one you're looking for is an example of not completely useless, but not portable enough to put anywhere safe, because linux documents the existence of IP_RECVERR, but not its value. So, you really need the version from your own system's ip headers.
The way to do this safely, if you actually need the IN module, is to run Tools/scripts/h2py.py with the Python version you're using, on the specific platform you need. That will generate an IN.py from the appropriate headers on your system (or on your cross-compilation target), which you can then use on that system. If you want to distribute your code, you'd probably need to put a step to do that into the setup.py, so it'll be run at install time (and at wheel-building time for people who install pre-built wheels, but you may need to be careful to make sure the targets are specific enough).
If you don't need to be particularly portable, you just need to access the one value in a few scripts that you're only deploying on your laptop or your company's set of identical containers or the like, you may be better off hardcoding the values (with a nice scare comment explaining the details).

Python Version (sys.version) Meaning?

I've hit what should be a basic question... but my googles are failing me and I need a sanity check.
If I run the following in my Python shell:
>>> import sys
>>> sys.version
from two different Python environments, I get:
'2.7.8 (default, Nov 10 2014, 08:19:18) \n[GCC 4.9.2 20141101 (Red Hat 4.9.2-1)]'
and...
'2.7.8 (default, Apr 15 2015, 09:26:43) \n[GCC 4.9.2 20150212 (Red Hat 4.9.2-6)]'
Does that mean the two environments are actually running slightly different Python guts or is it enough that the '2.7.8' bit in that version string is the same so I can be confident these are 1:1 identical Python interpreters?
If I am guaranteed they are the same, then what's the significance of the date and other parts of that version output string?

All you need to compare is the first bit, the 2.7.8 string.
The differences you see are due to the compiler used to build the binary, and when the binary was built. That shouldn't really make a difference here.
The string is comprised of information you can find in machine-readable form elsewhere; specifically:
platform.python_version()
Returns the Python version as string 'major.minor.patchlevel'.
platform.python_build()
Returns a tuple (buildno, builddate) stating the Python build number and date as strings.
platform.python_compiler()
Returns a string identifying the compiler used for compiling Python.
For your sample strings, what differs is the date the binary was build (second value of the platform.python_build() tuple) and the exact revision of the GCC compiler used (from the platform.python_compiler() string). Only when there are specific problems with the compiler would this matter.
You should normally only care about the Python version information, which is more readily available as the sys.version_info tuple.

Why won't this ctypes code work with Python 3.3 but will work with Python 2.7?

So I'm trying to make a Python 3.3 program to change the Windows desktop background using the ctypes module. I've tested the following code in Python 2.7, and it worked perfectly. But it just won't work with Python 3.3! I'm using Windows 7. Here's the code:
import ctypes
SPI_SETDESKTOPWALLPAPER=20
ctypes.windll.user32.SystemParametersInfoA(SPI_SETDESKTOPWALLPAPER, 0,"C:/Users/Public/Pictures/Sample Pictures/Penguins.jpg", 3)

SystemParametersInfoA requires a 8-bit ANSI encoded input string as a parameter, which is known as mbcs encoding in Python.
You will have to use SystemParametersInfoW in python3. This is because SystemParametersInfoW takes in a UTF-16 wide string (which is wchar_t * in C) and the ctypes library automatically converts this passed unicode argument into c_wchar_p.
Refer the documentation for more details.

Print colorized output - working from console but not from script

I have weird problem that I cannot put my finger on. There is a program that I use (and contribute from time to time) that has colorized console output. Everything worked great until I reinstalled Windows. Now I cannot get colorized output.
This is the script that is used for colorizing.
I have managed to narrow down the problem to, more or less, simple situation, but I have no idea what is wrong.
This is console prompt that works as expected (string test is printed in red):
Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.insert(0, r'c:\bin\SV\tea\src')
>>> from tea.console.color import cprint, Color
>>> cprint('test\n', Color.red)
test
>>>
But when I run following script with same version of python I get output test but not in red color (there is no color, just default console color):
import sys
sys.path.insert(0, r'c:\bin\SV\tea\src')
from tea.console.color import cprint, Color
cprint('test\n', Color.red)
The same setup worked before I reinstalled my system.
I have checked, environment variables in interactive mode and script are the same.
I have tried this in standard windows command prompt and Console, program that I
usually use.
OS in question is Windows 8 and before reinstall this was also used on Windows 8.
Same code with same setup works at computer at work (Windows 7).
I have Python 2.7 and Python 3.3 installed (as I did before). I have tried to run script
with calling python interpreter directly (c:\Python27\python.exe) or with py -2,
but it does not help.
IPython and mercurial colorizes output as it should.
Any ideas what can I try to make this work?
Edit:
Maybe it was not clear, but script I use to colorize output is given in a link in question. Here it is once again:
https://bitbucket.org/alefnula/tea/src/dc14009a19d66f92463549332a321b29c71d47b8/src/tea/console/color.py?at=default

I have found the problem and solution.
I believe that the problem was the bug in x64 ctypes module. I had Python 2.7 x64 installed and with that version following line (from script that I linked in question):
ctypes.windll.kernel32.SetConsoleTextAttribute(std_out_handle, code)
returns error code 6 with description The handle is invalid. After some investigation, I deduced that problem might be x64 version of python, so I installed 32-bit version and everything works as expected.
Since this solves my problem, and I do not have the time for deeper analysis I will leave it at this, just wanted to give some kind of resolution for question.

List indexing efficiency (python 2 vs python 3)

In answering another question, I suggested to use timeit to test the difference between indexing a list with positive integers vs. negative integers. Here's the code:
import timeit
t=timeit.timeit('mylist[99]',setup='mylist=list(range(100))',number=10000000)
print (t)
t=timeit.timeit('mylist[-1]',setup='mylist=list(range(100))',number=10000000)
print (t)
I ran this code with python 2.6:
$ python2.6 test.py
0.587687015533
0.586369991302
Then I ran it with python 3.2:
$ python3.2 test.py
0.9212150573730469
1.0225799083709717
Then I scratched my head, did a little google searching and decided to post these observations here.
Operating system: OS-X (10.5.8) -- Intel Core2Duo
That seems like a pretty significant difference to me (a factor of over 1.5 difference). Does anyone have an idea why python3 is so much slower -- especially for such a common operation?
EDIT
I've run the same code on my Ubuntu Linux desktop (Intel i7) and achieved comparable results with python2.6 and python 3.2. It seems that this is an issue which is operating system (or processor) dependent (Other users are seeing the same behavior on Linux machines -- See comments).
EDIT 2
The startup banner was requested in one of the answers, so here goes:
Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
and:
Python 3.2 (r32:88452, Feb 20 2011, 10:19:59)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
UPDATE
I've just installed fresh versions of python2.7.3 and python3.2.3 from http://www.python.org/download/
In both cases, I took the
"Python x.x.3 Mac OS X 32-bit i386/PPC Installer (for Mac OS X 10.3 through 10.6 [2])"
since I am on OS X 10.5. Here are the new timings (which are reasonably consistent through multiple trials):
python 2.7
$python2.7 test.py
0.577006101608
0.590042829514
python 3.2.3
$python3.2 test.py
0.8882801532745361
1.034242868423462

This appears to be an artifact of some builds of Python 3.2. The best hypothesis at this point is that all 32-bit Intel builds have the slowdown, but no 64-bit ones do. Read on for further details.
You didn't run nearly enough tests to determine anything. Repeating your test a bunch of times, I got values ranging from 0.31 to 0.54 for the same test, which is a huge variation.
So, I ran your test with 10x the number, and repeat=10, using a bunch of different Python2 and Python3 installs. Throwing away the top and bottom results, averaging the other 8, and dividing by 10 (to get a number equivalent to your tests), here's what I saw:
1. 0.52/0.53 Lion 2.6
2. 0.49/0.50 Lion 2.7
3. 0.48/0.48 MacPorts 2.7
4. 0.39/0.49 MacPorts 3.2
5. 0.39/0.48 HomeBrew 3.2
So, it looks like 3.2 is actually slightly faster with [99], and about the same speed with [-1].
However, on a 10.5 machine, I got these results:
1. 0.98/1.02 MacPorts 2.6
2. 1.47/1.59 MacPorts 3.2
Back on the original (Lion) machine, I ran in 32-bit mode, and got this:
1. 0.50/0.48 Homebrew 2.7
2. 0.75/0.82 Homebrew 3.2
So, it seems like 32-bitness is what matters, and not Leopard vs. Lion, gcc 4.0 vs. gcc 4.2 or clang, hardware differences, etc. It would help to test 64-bit builds under Leopard, with different compilers, etc., but unfortunately my Leopard box is a first-gen Intel Mini (with a 32-bit Core Solo CPU), so I can't do that test.
As further circumstantial evidence, I ran a whole slew of other quick tests on the Lion box, and it looks like 32-bit 3.2 is ~50% slower than 2.x, while 64-bit 3.2 is maybe a little faster than 2.x. But if we really want to back that up, someone needs to pick and run a real benchmark suite.
Anyway, my best guess at this point is that when optimizing the 3.x branch, nobody put much effort into 32-bit i386 Mac builds. Which is actually a reasonable choice for them to have made.
Or, alternatively, they didn't even put much effort into 32-bit i386 period. That possibility might explain why the OP saw 2.x and 3.2 giving similar results on a linux box, while Otto Allmendinger saw 3.2 being similarly slower to 2.6 on a linux box. But since neither of them mentioned whether they were running 32-bit or 64-bit linux, it's hard to know whether that's relevant.
There are still lots of other different possibilities that we haven't ruled out, but this seems like the best one.

here is a code that illustrates at least part of the answer:
$ python
Python 2.7.3 (default, Apr 20 2012, 22:44:07)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> t=timeit.timeit('mylist[99]',setup='mylist=list(range(100))',number=50000000)
>>> print (t)
2.55517697334
>>> t=timeit.timeit('mylist[99L]',setup='mylist=list(range(100))',number=50000000)
>>> print (t)
3.89904499054
$ python3
Python 3.2.3 (default, May 3 2012, 15:54:42)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> t=timeit.timeit('mylist[99]',setup='mylist=list(range(100))',number=50000000)
>>> print (t)
3.9906489849090576
python3 does not have old int type.

Python 3 range() is the Python 2 xrange(). If you want to simulate the Python 2 range() in Python 3 code, you have to use list(range(num). The bigger the num is, the bigger difference will be observed with your original code.
Indexing should be independent on what is stored inside the list as the list stores only references to the target objects. The references are untyped and all of the same kind. The list type is therefore a homogeneous data structure -- technically. Indexing means to turn the index value into the start address + offset. Calculating the offset is very efficient with at most one subtraction. This is very cheap extra operation when compared with the other operations.