What is the value of None in memory? - python

None in Python is a reserved word, just a question crossed my mind about the exact value of None in memory. What I'm holding in my mind is this, None's representation in memory is either 0 or a pointer pointing to a heap. But neither the pointer pointing to an empty area in the memory will make sense neither the zero value. Because when I tested the following:
>>> None.__sizeof__()
16
It turns out that None consumes 16 bytes and that's actually too much for simply an empty value.
So what does None actually represents in memory?

None is singleton object which doesn't provide (almost) none methods and attributes and its only purpose is to signify the fact that there is no value for some specific operation.
As a real object it still needs to have some headers, some reflection information and things like that so it takes the minimum memory occupied by every python object.

None references some Python object of the NoneType, actually the only instance of its kind. Since it’s a Python object—like literally every other thing—it’s not just the raw content what is stored about also the additional object information like its type.
Every object has this overhead; that’s why integers start with a size of 28 for example. You cannot change this, it’s an implementation detail of Python and its type system.

None is a special constant in Python that represents the absence of a value or a null value. It is an object of its own datatype, the NoneType. We cannot create multiple None objects but can assign it to variables. These variables will be equal to one another. We must take special care that None does not imply False, 0 or any empty list, dictionary, string etc. For example:
>>> None == 0
False
>>> None == []
False
>>> None == False
False
>>> x = None
>>> y = None
>>> x == y
True
Void functions that do not return anything will return a None object automatically. None is also returned by functions in which the program flow does not encounter a return statement. For example:
def a_void_function():
a = 1
b = 2
c = a + b
x = a_void_function()
print(x)
Output:
None
This program has a function that does not return a value, although it does some operations inside. So when we print x, we get None which is returned automatically (implicitly). Similarly, here is another example:
def improper_return_function(a):
if (a % 2) == 0:
return True
x = improper_return_function(3)
print(x)
Output:
None
Although this function has a return statement, it is not reached in every case. The function will return True only when the input is even. So, if we give the function an odd number, None is returned implicitly.

It turns out that None consumes 16 bytes and that's actually too much for simply an empty value.
None ist not an empty value. It is an NoneType object which represents the absence of a value as you can read here: Python Built-in constants Doc. You can assign None to variables, but you can't assign to None, as it would raise an syntax error.

Related

Python None comparison with is keyword

So the is keyword returns true only if the two arguments point to the same object. My question is related to the snippet below.
This snippet
number = None
if number is None:
print("PEP 8 Style Guide prefers this pattern")
Outputs
>>PEP 8 Style Guide prefers this pattern
Does this mean when I assign number = None then it is only by reference, because is checks if it is the same object. I'm so confused why this happens?? Am I wrong?? Why was this design choice made?
This is because of two reasons.
1. Assignments in Python are by reference.
In the following code
x = object()
y = x
print(x is y) # True
print(id(x)) # 139957673835552
print(id(y)) # 139957673835552
Calling object() creates a new structure in memory, whose unique identifier can be accessed with the id() function.
You can imagine x and y being arrows pointing to the same object, and this is why their underlying identifier is the same in both cases.
As such, when assigning None to a variable, you're just saying "number is an alias, an arrow pointing to the object returned by writing None". You can check that
number = None
print(id(None), id(number))
will give you the same identifier two times. But wait! What if you do it for a big number like 100000?
number = 100000
print(id(100000), id(number)) # Different values!
This means that the same literal, written two times, can return different objects, bringing up the next reason.
2. The language guarantee for None
Note that no matter how many times you get the None identifier, you get the same one.
print(id(None)) # 139957682420224
print(id(None)) # 139957682420224
print(id(None)) # 139957682420224
This is because writing None doesn't create a new object, as in the first example, because the language specification guarantees that there's only one possible object of NoneType in memory. In other words, there's only one possible object returned by writing None, making comparisons with is work as expected. This is a good design choice: There's only one canonical way for saying that a variable (an arrow) points to nothingness.
In fact, using is is encouraged as the Pythonic way of checking that a variable is None.
You can also get the class of the object by writing
NoneType = type(None)
and notice how
NoneType() is None
is true.
Note: The uniqueness property is also satisfied by other literals, particularly small numbers, for performance reasons.
All assignments are by reference (see Facts and myths about Python names and values). However, the language guarantees that None is the only object of its type. That value is created at startup, and the literal None will always produce a reference to that value.
>>> a = None; b = None
>>> a is b
True
>>> a = None
>>> b = None
>>> a is b
True
Compare to a literal like 12345, which may or may not produce a reference to an existing value of type int.
>>> a = 12345; b = 12345
>>> a is b
True
>>> a = 12345
>>> b = 12345
>>> a is b
False
Why this produces different results isn't really important, other than to say that an implementation can create new objects from int literals if it prefers.

Changing the variable given in a function statement

My code so far is:
def errorCheckInt(n):
try:
n = int(n)
checkValue1 = True
except:
checkValue1 = False
It is supposed to take a variable input (which is n) and change it to an integer if possible. It will try to get a different input if it is a string.
However, it hasn't worked, and I think it's because you can't change the variable in the code.
This may not be very clear, so I'm giving an example:
testingNum = "2"
# def errorCheckInt here
errorCheckInt(testingNum)
# Should change testingNum to integer value 2, not string "2"
This would be followed by a while statement, checking whether it was successful (whether checkValue1 is True or False) then possibly asking for a new input and error checking that (depending on the input the user gives).
If this still isn't very clear, just comment a question and I may be able to tell you what I had in mind.
Thanks in advance!
It isn't clear to me what you mean when you say "you can't change the variable in the code". Honestly, I'm going to take a leap here (since I have no idea where you are at in terms of Python understanding).
There are two concepts I feel like you may be having trouble with. (1) is the idea of object mutability, and (2) how variables are used and defined in and out of scope inside of Python.
When you re-assign a value like this:
n = int(n)
You haven't actually modified the value held by the variable outside (testingNum). This is because variables in Python are just references to objects. This means that your variable will always point to the same object unless it is re-assigned explicitly. Where this gets confusing is when you're dealing with the concept of mutability. For example, lists, dictionaries and classes are mutable objects. If you do something like:
t = []
n = t
and then do:
n.extend([1,2,3,4])
you'll see that both n and t now look like:
[1,2,3,4]
This isn't really mysterious behavior when you finally understand what's going on. The integer 4 and the [1,2,3,4] are different objects. The only difference is that [1,2,3,4] was also the same object as the initial [] when it was assigned to t. On the other hand, when you re-assign an integer or a string to a variable, you've simply changed what object the variable is pointing to. That's all.
How does this apply to your example?
In essence, all you were missing was a return statement:
def errorCheckInt(n):
try:
n = int(n)
checkValue1 = True
except:
checkValue1 = False
return n
testingNum = "2"
newNum = errorCheckInt(testingNum) # You can catch it here with newNum.
Understanding the use of the return statement is tantamount. When you assign a reference to a variable inside a function, that variable falls out of scope as soon as you exit the function; in other words, if you try to call it afterwards, you'll get an error. To circumvent this, you need to catch the result of your function in a variable that's in the outer scope. This allows you to keep working with the data you've just calculated inside the function.
Note
A better way of doing what you're asking is to use isinstance. For example,
print isinstance(4, int)
# True
print isinstance('4', int)
# False
This will automatically check if the value is an instance of the int type. It's both clearer and cleaner.
Further clarification
An interesting fact of mutability is that when you pass references to mutable objects inside of functions, modifying the object with, say, [].append(4) will mean that any variable pointing to that object (even from outside) is getting the same update if you will. So in certain cases using the return statement isn't necessary, but being explicit is sometimes worth the effort.
When errorCheckInt is run, variable n is bound to the same value as testingNum. However with n =, you change the binding, not the value so n then points to a different value than testingNum. What you need is to pass a mutable type and change the binding inside it:
def errorCheckInt(n):
try:
n[0] = int(n[0])
checkValue1 = True
except:
checkValue1 = False
testingNum = [ "2" ]
errorCheckInt(testingNum)
# testingNum is now 2 and type(testingNum) is int

Is the data in xlsx cells string?

Am using This and doing
if sheet.row_values(rownum)[0] is not 'Value':
....print sheet.row_values(rownum)[0]
After I do that, I still get Value printed. Could it be that the sheet.row_values(rownum)[0] value is not a string? Or why is the If statement not working?
is is testing the identity of an object.
if I am to write:
a = 1
b = 1
a is b # not guaranteed to be true
This is because the variable itself may point at different data, which is your case.
Using the == equality operator would give you the desired results, because it would actually look at the contents, not the strings identifier.
Note when you are comparing a variable to None, the is operator is okay, because there is actually only ever one None object.

Use of True, False, and None as return values in Python functions

I think that I fully understand this, but I just want to make sure since I keep seeing people say to never ever test against True, False, or None.
They suggest that routines should raise an error rather than return False or None. Anyway, I have many situations where I simply want to know if a flag is set or not so my function returns True or False. There are other situations where I have a function return None if there was no useful result. From my thinking, neither is problematic so long as I realize that I should never use:
if foo == True
if foo == False
if foo == None
and should instead use:
if foo is True
if foo is False
if foo is None
since True, False, and None are all singletons and will always evaluate the way I expect when using "is" rather than "==". Am I wrong here?
Along the same lines, would it be more Pythonic to modify the functions that sometimes return None so that they raise an error instead?
Say I have an instance method called "get_attr()" that retrieves an attribute from some file. In the case where it finds that the attribute I requested does not exist, is it appropriate to return None? Would it be better to have them raise an error and catch it later?
The advice isn't that you should never use True, False, or None. It's just that you shouldn't use if x == True.
if x == True is silly because == is just a binary operator! It has a return value of either True or False, depending on whether its arguments are equal or not. And if condition will proceed if condition is true. So when you write if x == True Python is going to first evaluate x == True, which will become True if x was True and False otherwise, and then proceed if the result of that is true. But if you're expecting x to be either True or False, why not just use if x directly!
Likewise, x == False can usually be replaced by not x.
There are some circumstances where you might want to use x == True. This is because an if statement condition is "evaluated in Boolean context" to see if it is "truthy" rather than testing exactly against True. For example, non-empty strings, lists, and dictionaries are all considered truthy by an if statement, as well as non-zero numeric values, but none of those are equal to True. So if you want to test whether an arbitrary value is exactly the value True, not just whether it is truthy, when you would use if x == True. But I almost never see a use for that. It's so rare that if you do ever need to write that, it's worth adding a comment so future developers (including possibly yourself) don't just assume the == True is superfluous and remove it.
Using x is True instead is actually worse. You should never use is with basic built-in immutable types like Booleans (True, False), numbers, and strings. The reason is that for these types we care about values, not identity. == tests that values are the same for these types, while is always tests identities.
Testing identities rather than values is bad because an implementation could theoretically construct new Boolean values rather than go find existing ones, leading to you having two True values that have the same value, but they are stored in different places in memory and have different identities. In practice I'm pretty sure True and False are always reused by the Python interpreter so this won't happen, but that's really an implementation detail. This issue trips people up all the time with strings, because short strings and literal strings that appear directly in the program source are recycled by Python so 'foo' is 'foo' always returns True. But it's easy to construct the same string 2 different ways and have Python give them different identities. Observe the following:
>>> stars1 = ''.join('*' for _ in xrange(100))
>>> stars2 = '*' * 100
>>> stars1 is stars2
False
>>> stars1 == stars2
True
EDIT: So it turns out that Python's equality on Booleans is a little unexpected (at least to me):
>>> True is 1
False
>>> True == 1
True
>>> True == 2
False
>>> False is 0
False
>>> False == 0
True
>>> False == 0.0
True
The rationale for this, as explained in the notes when bools were introduced in Python 2.3.5, is that the old behaviour of using integers 1 and 0 to represent True and False was good, but we just wanted more descriptive names for numbers we intended to represent truth values.
One way to achieve that would have been to simply have True = 1 and False = 0 in the builtins; then 1 and True really would be indistinguishable (including by is). But that would also mean a function returning True would show 1 in the interactive interpreter, so what's been done instead is to create bool as a subtype of int. The only thing that's different about bool is str and repr; bool instances still have the same data as int instances, and still compare equality the same way, so True == 1.
So it's wrong to use x is True when x might have been set by some code that expects that "True is just another way to spell 1", because there are lots of ways to construct values that are equal to True but do not have the same identity as it:
>>> a = 1L
>>> b = 1L
>>> c = 1
>>> d = 1.0
>>> a == True, b == True, c == True, d == True
(True, True, True, True)
>>> a is b, a is c, a is d, c is d
(False, False, False, False)
And it's wrong to use x == True when x could be an arbitrary Python value and you only want to know whether it is the Boolean value True. The only certainty we have is that just using x is best when you just want to test "truthiness". Thankfully that is usually all that is required, at least in the code I write!
A more sure way would be x == True and type(x) is bool. But that's getting pretty verbose for a pretty obscure case. It also doesn't look very Pythonic by doing explicit type checking... but that really is what you're doing when you're trying to test precisely True rather than truthy; the duck typing way would be to accept truthy values and allow any user-defined class to declare itself to be truthy.
If you're dealing with this extremely precise notion of truth where you not only don't consider non-empty collections to be true but also don't consider 1 to be true, then just using x is True is probably okay, because presumably then you know that x didn't come from code that considers 1 to be true. I don't think there's any pure-python way to come up with another True that lives at a different memory address (although you could probably do it from C), so this shouldn't ever break despite being theoretically the "wrong" thing to do.
And I used to think Booleans were simple!
End Edit
In the case of None, however, the idiom is to use if x is None. In many circumstances you can use if not x, because None is a "falsey" value to an if statement. But it's best to only do this if you're wanting to treat all falsey values (zero-valued numeric types, empty collections, and None) the same way. If you are dealing with a value that is either some possible other value or None to indicate "no value" (such as when a function returns None on failure), then it's much better to use if x is None so that you don't accidentally assume the function failed when it just happened to return an empty list, or the number 0.
My arguments for using == rather than is for immutable value types would suggest that you should use if x == None rather than if x is None. However, in the case of None Python does explicitly guarantee that there is exactly one None in the entire universe, and normal idiomatic Python code uses is.
Regarding whether to return None or raise an exception, it depends on the context.
For something like your get_attr example I would expect it to raise an exception, because I'm going to be calling it like do_something_with(get_attr(file)). The normal expectation of the callers is that they'll get the attribute value, and having them get None and assume that was the attribute value is a much worse danger than forgetting to handle the exception when you can actually continue if the attribute can't be found. Plus, returning None to indicate failure means that None is not a valid value for the attribute. This can be a problem in some cases.
For an imaginary function like see_if_matching_file_exists, that we provide a pattern to and it checks several places to see if there's a match, it could return a match if it finds one or None if it doesn't. But alternatively it could return a list of matches; then no match is just the empty list (which is also "falsey"; this is one of those situations where I'd just use if x to see if I got anything back).
So when choosing between exceptions and None to indicate failure, you have to decide whether None is an expected non-failure value, and then look at the expectations of code calling the function. If the "normal" expectation is that there will be a valid value returned, and only occasionally will a caller be able to work fine whether or not a valid value is returned, then you should use exceptions to indicate failure. If it will be quite common for there to be no valid value, so callers will be expecting to handle both possibilities, then you can use None.
Use if foo or if not foo. There isn't any need for either == or is for that.
For checking against None, is None and is not None are recommended. This allows you to distinguish it from False (or things that evaluate to False, like "" and []).
Whether get_attr should return None would depend on the context. You might have an attribute where the value is None, and you wouldn't be able to do that. I would interpret None as meaning "unset", and a KeyError would mean the key does not exist in the file.
If checking for truth:
if foo
For false:
if not foo
For none:
if foo is None
For non-none:
if foo is not None
For getattr() the correct behaviour is not to return None, but raise an AttributError error instead - unless your class is something like defaultdict.
Concerning whether to raise an exception or return None: it depends on the use case. Either can be Pythonic.
Look at Python's dict class for example. x[y] hooks into dict.__getitem__, and it raises a KeyError if key is not present. But the dict.get method returns the second argument (which is defaulted to None) if key is not present. They are both useful.
The most important thing to consider is to document that behaviour in the docstring, and make sure that your get_attr() method does what it says it does.
To address your other questions, use these conventions:
if foo:
# For testing truthiness
if not foo:
# For testing falsiness
if foo is None:
# Testing .. Noneliness ?
if foo is not None:
# Check explicitly avoids common bugs caused by empty sequences being false
Functions that return True or False should probably have a name that makes this obvious to improve code readability:
def is_running_on_windows():
return os.name == 'nt'
In Python 3 you can "type-hint" that:
>>> def is_running_on_windows() -> bool:
... return os.name == 'nt'
...
>>> is_running_on_windows.__annotations__
{'return': bool}
You can directly check that a variable contains a value or not, like if var or not var.
In the examples in PEP 8 (Style Guide for Python Code) document, I have seen that foo is None or foo is not None are being used instead of foo == None or foo != None.
Also using if boolean_value is recommended in this document instead of if boolean_value == True or if boolean_value is True. So I think if this is the official Python way. We Python guys should go on this way, too.
One thing to ensure is that nothing can reassign your variable. If it is not a Boolean in the end, relying on truthiness will lead to bugs. The beauty of conditional programming in dynamically typed languages :).
The following prints "no".
x = False
if x:
print 'yes'
else:
print 'no'
Now let's change x.
x = 'False'
Now the statement prints "yes", because the string is truthy.
if x:
print 'yes'
else:
print 'no'
This statement, however, correctly outputs "no".
if x == True:
print 'yes'
else:
print 'no'
In the case of your fictional getattr function, if the requested attribute always should be available but isn't then throw an error. If the attribute is optional then return None.
For True, not None:
if foo:
For false, None:
if not foo:

Python things which are neither True nor False

I just found this :
a = (None,)
print (a is True)
print (a is False)
print (a == True)
print (a == False)
print (a == None)
print (a is None)
if a : print "hello"
if not a : print "goodbye"
which produces :
False
False
False
False
False
False
hello
So a neither is, nor equals True nor False, but acts as True in an if statement.
Why?
Update :
actually, I've just realized that this isn't as obscure as I thought. I get the same result for a=2, as well (though not for a=0 or a=1, which are considered equal to False and True respectively)
I find almost all the explanations here unhelpful, so here is another try:
The confusion here is based on that testing with "is", "==" and "if" are three different things.
"is" tests identity, that is, if it's the same object. That is obviously not true in this case.
"==" tests value equality, and obviously the only built in objects with the values of True and False are the object True and False (with the exception of the numbers 0 and 1, of any numeric type).
And here comes the important part:
'if' tests on boolean values. That means that whatever expression you give it, it will be converted to either True or False. You can make the same with bool(). And bool((None,)) will return True. The things that will evaluate to False is listed in the docs (linked to by others here)
Now maybe this is only more clear in my head, but at least I tried. :)
a is a one-member tuple, which evaluates to True. is test identity of the object, therefore, you get False in all those test. == test equality of the objects, therefore, you get False again.
in if statement a __bool__ (or __nonzero__) used to evaluate the object, for a non-empty tuple it should return True, therefore you get True. hope that answers your question.
edit: the reason True and False are equal to 1 and 0 respectively is because bool type implemented as a subclass of int type.
Things in python don't have to be one of True or False.
When they're used as a text expression for if/while loops, they're converted to booleans. You can't use is or == to test what they evaluate to. You use bool( thing )
>>> a = (None,)
>>> bool(a)
True
Also note:
>>> 10 == True
False
>>> 10 is True
False
>>> bool(10)
True
TL;DR:
if and == are completely different operations. The if checks the truth value of a variable while == compares two variables. is also compares two variables but it compares if both reference the same object.
So it makes no sense to compare a variable with True, False or None to check it's truth value.
What happens when if is used on a variable?
In Python a check like if implicitly gets the bool of the argument. So
if something:
will be (under the hood) executed like:
if bool(something):
Note that you should never use the latter in your code because it's considered less pythonic and it's slower (because Python then uses two bools: bool(bool(something))). Always use the if something.
If you're interested in how it's evaluated by CPython 3.6:
Note that CPython doesn't exactly use hasattr here. It does check if the type of x implements the method but without going through the __getattribute__ method (hasattr would use that).
In Python2 the method was called __nonzero__ and not __bool__
What happens when variables are compared using ==?
The == will check for equality (often also called "value equality"). However this equality check doesn't coerce the operands (unlike in other programming languages). The value equality in Python is explicitly implemented. So you can do:
>>> 1 == True # because bool subclasses int, True is equal to 1 (and False to 0)
True
>>> 1.0 == True # because float implements __eq__ with int
True
>>> 1+1j == True # because complex implements __eq__ with int
True
However == will default to reference comparison (is) if the comparison isn't implemented by either operand. That's why:
>>> (None, ) == True
False
Because tuple doesn't "support" equality with int and vise-versa. Note that even comparing lists to tuples is "unsupported":
>>> [None] == (None, )
False
Just in case you're interested this is how CPython (3.6) implements equality (the orange arrows indicate if an operation returned the NotImplemented constant):
That's only roughly correct because CPython also checks if the type() of value1 or value2 implements __eq__ (without going through the __getattribute__ method!) before it's called (if it exists) or skipped (if it doesn't exist).
Note that the behavior in Python2 was significantly more lengthy (at least if the methods returned NotImplemented) and Python 2 also supported __cmp__,
What happens when variables are compared using is?
is is generally referred to as reference equality comparison operator. It only returns True if both variables refer to exactly the same object. In general variables that hold the same value can nevertheless refer to different objects:
>>> 1 is 1. # same value, different types
False
>>> a = 500
>>> a is 500 # same value, same type, different instances
False
Note that CPython uses cached values, so sometimes variables that "should" be different instances are actually the same instance. That's why I didn't use 500 is 500 (literals with the same value in the same line are always equal) and why I couldn't use 1 as example (because CPython re-uses the values -5 to 256).
But back to your comparisons: is compares references, that means it's not enough if both operands have the same type and value but they have to be the same reference. Given that they didn't even have the same type (you're comparing tuple with bool and NoneType objects) it's impossible that is would return True.
Note that True, False and None (and also NotImplemented and Ellipsis) are constants and singletons in CPython. That's not just an optimization in these cases.
(None,) is a tuple that contains an element, it's not empty and therefore does not evaluate to False in that context.
Because a=(None,) is a tuple containing a single element None
Try again with a=None and you will see there is a different result.
Also try a=() which is the empty tuple. This has a truth value of false
In Python every type can be converted to bool by using the bool() function or the __nonzero__ method.
Examples:
Sequences (lists, strings, ...) are converted to False when they are empty.
Integers are converted to False when they are equal to 0.
You can define this behavior in your own classes by overriding __nonzero__().
[Edit]
In your code, the tuple (None,) is converted using bool() in the if statements. Since it's non-empty, it evaluates to True.

Categories