I have learning python for a year. In the lecture slides, I see that the compiler usually allocate 4 bytes to store items in a list.
Why can't the compiler allocate memory dynamically? For example, if I have a small value, why don't the compiler just assign 1 or 2 bytes to store it, would it be more memory efficient?
What is mean by 32 bit-integer where one bit is a sign bit and 31 bits represent number? One byte is equal to 8 bits and an integer is 32 bits, so why would they represent it with 8 bytes where they waste 4 bytes btw in some computers you see 64-bit , they can use 8 bytes and it has only effect how greater number you can represent. The more bits ==> the more greater number.It simply goes about your processor and it's registers.
It is not true that elements in a list or array take 4 bytes.
In languages with strong static types (that is, in which you have to declare the type of a variable before using it), an array takes exactly the size of the array times the size of each variable of that type. In C, char types usually uses one byte, and int types use an amount of bytes that depends on the system's architecture.
In languages with dynamic types, like Python, all variables are implemented as objects containing pointers (to a location in memory where data is actually stored) and labels (indicating the type of the variable, which will be used in runtime to determine the variable's behavior when performing operations).
The 4-byte issue probably refers to the fact that many architectures use 4 bytes for integers, but this is coincidental and it is a stretch to consider it a rule.
Related
I have started learning Data Structures and Algorithms.
Please help me with my doubt.
Is it ok to say that lists, tuples, and dictionaries are a type of referential array?
I was going through the example in the book where it was written that we want a medical information system to keep track of the patients currently assigned to beds in a certain hospital. If we assume that the hospital has 200 beds, and conveniently that those beds are numbered from 0 to 199, we might consider using an array-based structure to maintain the names of the patients currently assigned to those beds. For example, in Python, we might use a list of names, such as:
[ Rene , Joseph , Janet , Jonas , Helen , Virginia , ... ]
To represent such a list with an array, Python must adhere to the requirement that each cell of the array use the same number of bytes. Yet the elements are strings, and strings naturally have different lengths. Python could attempt to reserve enough space for each cell to hold the maximum length string (not just of currently stored strings, but of any string we might ever want to store), but that would be wasteful.
Instead, Python represents a list or tuple instance using an internal storage mechanism of an array of object references. At the lowest level, what is stored is a consecutive sequence of memory addresses at which the elements of the sequence reside. A high-level diagram of such a list is shown in Figure
https://i.stack.imgur.com/7FffP.jpg
What I understood is that python stores the memory address of "Rene" or "Joseph". But memory addresses will also change with respect to number of characters in the name like each Unicode takes 2 bytes of space.
Now it was also written that "Although the relative size of the individual elements may vary, the number of bits used to store the memory address of each element is fixed (e.g., 64-bits per address i.e 8 bytes). What if the character is very long and it's not able to store the memory address in 64 bits?
Is it ok to say that lists, tuples, and dictionaries are a type of referential array?
Lists are (though it might matter that they have dynamic resizing built in), but tuples and dictionaries have a lot more going on behind the scenes.
What I understood is that python stores the memory address of "Rene" or "Joseph". But memory addresses will also change with respect to number of characters in the name like each Unicode takes 2 bytes of space.
The memory address is an integer describing where to find the data, and the data itself is a string of bytes like b'Rene'. I'm not sure what CPython uses to figure out where a data structure ends, but common approaches include a null terminator (when given an address, all bytes that you find are part of the data structure till you hit a null byte), ensuring every object is the same size so that you can skip ahead (not feasible for strings), or having the first byte(s) describe the length of the rest of the object. The address itself has a fixed size, e.g., 64 bits.
In the following illustration, the "address" used in the referential array would be 3, and you would know that the last byte of the string happened at address 6 by walking till you found the null byte (the other approaches I mentioned work similarly). The important thing is that 3 is the only number the referential array needs to store to be able to reference b'Rene'.
|3|4|5|6|7 |
|R|e|n|e|\0|
What if the character is very long and it's not able to store the memory address in 64 bits?
This was actually the reason 32-bit systems couldn't use more than 4GB of RAM -- they only had 32-bit identifiers to access the RAM (without additional techniques like assigning each process a RAM offset so that each process has a 32-bit memory space). Modern 64-bit systems allow 16 exabytes of RAM to be indexed without needing any other special techniques -- one of the primary motivations for the 32 -> 64 bit migration.
In any case though, memory limits are a real thing, and if we start needing to address more memory than that we'll need to use more complicated strategies than fixed-width integer references, or we'll need to jump up again and use bigger integers.
Large numpy array (over 4GB) with nyp file and memmap function
I was using numpy package for array calculation where I read https://docs.scipy.org/doc/numpy/neps/npy-format.html
In "Format Specification: Version 2.0" it said that, for .npy file, "version 2.0 format extends the header size to 4 GiB".
My question was that:
What was header size? Did that mean I could only save numpy.array of sizeat most 4GB array into the npy file? How large could a single array go?
I also read https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.memmap.html
where it stated that "Memory-mapped files cannot be larger than 2GB on 32-bit systems"
did it mean numpy.memmap's limitation was based on the memory of the system? Was there anyway to avoid such limitation?
Further, I read that we could chose the dtype of the array, where the best resolution was "complex128". Was there any way to "use" and "save" elements with more accuracy on a 64 bit computer?(more accurate than complex128 or float64)
The previous header size field was 16 bits wide, allowing headers smaller than 64KiB. Because the header describes the structure of the data, and doesn't contain the data itself, this is not a huge concern for most people. Quoting the notes, "This can be exceeded by structured arrays with a large number of columns." So to answer the first question, header size was under 64KiB but the data came after, so this wasn't the array size limit. The format didn't specify a data size limit.
Memory map capacity is dependent on operating system as well as machine architecture. Nowadays we've largely moved to flat but typically virtual address maps, so the program itself, stack, heap, and mapped files all compete for the same space, in total 4GiB for 32 bit pointers. Operating systems frequently partition this in quite large chunks, so some systems might only allow 2GiB total for user space, others 3GiB; and often you can map more memory than you can allocate otherwise. The memmap limitation is more closely tied to the operating system in use than the physical memory.
Non-flat address spaces, such as using distinct segments on OS/2, could allow larger usage. The cost is that a pointer is no longer a single word. PAE, for instance, supplies a way for the operating system to use more memory but still leaves processes with their own 32 bit limits. Typically it's easier nowadays to use a 64 bit system, allowing memory spaces up to 16 exabytes. Because data sizes have grown a lot, we also handle it in larger pieces, such as 4MiB or 16MiB allocations rather than the classic 4KiB pages or 512B sectors. Physical memory typically has more practical limits.
Yes, there are elements with more precision than 64 bit floating point; in particular, 64 bit integers. This effectively uses a larger mantissa by sacrificing all of the exponent. Complex128 is two 64 bit floats, and doesn't have higher precision but a second dimension. There are types that can grow arbitrarily precise, such as Python's long integers (long in python 2, int in python 3) and fractions, but numpy generally doesn't delve into those because they also have matching storage and computation costs. A basic property of the arrays is that they can be addressed using index calculations since the element size is consistent.
This is mostly a question out of curiosity. I noticed that the numpy test suite contains tests for 128 bit integers, and the numerictypes module refers to int128, float256 (octuple precision?), and other types that don't seem to map to numpy dtypes on my machine.
My machine is 64bit, yet I can use quadruple 128bit floats (but not really). I suppose that if it's possible to emulate quadruple floats in software, one can theoretically also emulate octuple floats and 128bit ints. On the other hand, until just now I had never heard of either 128bit ints or octuple precision floating point before. Why is there a reference to 128bit ints and 256bit floats in numpy's numerictypes module if there are no corresponding dtypes, and how can I use those?
This is a very interesting question and probably there are reasons related to python, to computing and/or to hardware. While not trying to give a full answer, here is what I would go towards...
First note that the types are defined by the language and can be different from your hardware architecture. For example you could even have doubles with an 8-bits processor. Of course any arithmetic involves multiple CPU instructions, making the computation much slower. Still, if your application requires it, it might be worth it or even required (better being late than wrong, especially if say you are running a simulation for a say bridge stability...) So where is 128bit precision required? Here's the wikipedia article on it...
One more interesting detail is that when we say a computer is say 64-bit, this is not fully describing the hardware. There are a lot of pieces that can each be (and at least have been at times) different bits: The computational registers in the CPU, the memory addressing scheme / memory registers and the different buses with most important the buss from CPU to memory.
-The ALU (arithmetic and logic unit) has registers that do calculations. Your machines are 64bit (not sure if that also mean they could do 2 32bit calculations at a similar time) This is clearly the most relevant quantity for this discussion. Long time ago, it used to be the case you could go out and buy a co-processor to speed that for calculations of higher precision...
-The Registers that hold memory addresses limit the memory the computer can see (directly) that is why computers that had 32bit memory registers could only see 2^32 bytes (or approx 4 GB) Notice that for 16bits, this becomes 65K which is very low. The OS can find ways around this limit, but not for a single program, so no program in a 32bit computer can normally have more than 4GB memmory.
-Notice that those limits are about bytes, not bits. That is because when referring and loading from memory we load bytes. In fact, the way this is done, loading a byte (8 bits) or 8 (64 bits == buss length for your computer) takes the same time. I ask for an address, and then get at once all bits through the bus.
It can be that in an architecture all these quantities are not the same number of bits.
NumPy is amazingly powerful and can handle numbers much bigger than the internal CPU representation (e.g. 64 bit).
In case of dynamic type it stores the number in an array. It can extend the memory block too, that is why you can have an integer with 500 digits. This dynamic type is called bignum. In older Python versions it was the type long. In newer Python (3.0+) there is only long, which is called int, which supports almost arbitrarily number of digits (-> bignum).
If you specify a data type (int32 for example), then you specify bit length and bit format, i.e. which bits in memory stands for what. Example:
dt = np.dtype(np.int32) # 32-bit integer
dt = np.dtype(np.complex128) # 128-bit complex floating-point number
Look in: https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
I need to implement a Dynamic Programming algorithm to solve the Traveling Salesman problem in time that beats Brute Force Search for computing distances between points. For this I need to index subproblems by size and the value of each subproblem will be a float (the length of the tour). However holding the array in memory will take about 6GB RAM if I use python floats (which actually have double precision) and so to try and halve that amount (I only have 4GB RAM) I will need to use single precision floats. However I do not know how I can get single precision floats in Python (I am using Python 3). Could someone please tell me where I can find them (I was not able to find much on this on the internet). Thanks.
EDIT: I notice that numpy also has a float16 type which will allow for even more memory savings. The distances between points are around 10000 and there are 25 unique points and my answer needs to be to the nearest integer. Will float16 provide enought accuracy or do I need to use float32?
As a first step, you should use a NumPy array to store your data instead of a Python list.
As you correctly observe, a Python float uses double precision internally, and the double-precision value underlying a Python float can be represented in 8 bytes. But on a 64-bit machine, with the CPython reference implementation of Python, a Python float object takes a full 24 bytes of memory: 8 bytes for the underlying double-precision value, 8 bytes for a pointer to the object type, and 8 bytes for a reference count (used for garbage collection). There's no equivalent of Java's "primitive" types or .NET's "value" types in Python - everything is boxed. That makes the language semantics simpler, but means that objects tend to be fatter.
Now if we're creating a Python list of float objects, there's the added overhead of the list itself: one 8-byte object pointer per Python float (still assuming a 64-bit machine here). So in general, a list of n Python float objects is going to cost you over 32n bytes of memory. On a 32-bit machine, things are a little better, but not much: our float objects are going to take 16 bytes each, and with the list pointers we'll be using 20n bytes of memory for a list of floats of length n. (Caveat: this analysis doesn't quite work in the case that your list refers to the same Python float object from multiple list indices, but that's not a particularly common case.)
In contrast, a NumPy array of n double-precision floats (using NumPy's float64 dtype) stores its data in "packed" format in a single data block of 8n bytes, so allowing for the array metadata the total memory requirement will be a little over 8n bytes.
Conclusion: just by switching from a Python list to a NumPy array you'll reduce your memory needs by about a factor of 4. If that's still not enough, then it might make sense to consider reducing precision from double to single precision (NumPy's float32 dtype), if that's consistent with your accuracy needs. NumPy's float16 datatype takes only 2 bytes per float, but records only about 3 decimal digits of precision; I suspect that it's going to be close to useless for the application you describe.
You could try the c_float type from the ctypes standard library. Alternatively, if you are capable of installing additional packages you might try the numpy package. It includes the float32 type.
Observe the following (in Python verson 2.7.3):
>>> import sys
>>> sys.getsizeof(1)
24
>>> sys.getsizeof(10000)
24
Why does the larger integer only take the same amount of memory (memory of the heap as afaik) of the smaller integer?
Compare this the following
>>> sys.getsizeof("1")
38
>>> sys.getsizeof("100000")
42
If you do sys.getsizeof("1") vs sys.getsizeof("100000") the larger string takes up more memory. So why would it be different for integers?
Because integers are not stored character by character.
In Python, an integer starts as large as is the platform's native word (plus the extra space needed for the interpreter's internal bookkeeping) because arithmetic is way faster - if you use native types, the processor can directly do the arithmetic on them.
When you get outside the native word boundaries (-2**31 - (2**31-1) on 32 bit machines), Python automatically switches to arbitrary precision arithmetic, where operations are "emulated" in software (of course building on the usual primitives provided by the hardware).
This still won't show you a "string-like" increase because the used space increases in larger chunks (again, typically for efficiency reasons).
Also, looking for an 1:1 decimal digits-integer size relation is misleading, since integers are stored in binary, and a decimal digit "takes" ~3.32 binary digits; so, the size "jumps" won't be on "decimal digits" boundaries, but more on powers of two boundaries.
Strings and integers are indeed comparable in that they can be considered as an arbitrarily long sequence of "digits" or "characters". And when you use large enough integers (like 2^70, 5e100, ...) you will see memory usage go up1. When you add one "character"/"digit" to either, memory usage goes up. For strings, a "character" is roughly what you expect (it gets more complicated with unicode, but whatever).
For integers, however, one "digit" is quite large: Several dozen bits (instead of a single decimal digit, let alone a single bit). This is partly because memory is divided into bits and groups of bits (rather then decimal digits), partly you can't allocate individual bits (at most individual bytes, but even that is wasteful), partly because it's more effective to as many bits as the CPU can work on "natively" (4 to 8 byte usually), and partly to simplify the code.
There's the additional complication that Python 2 has two integer types, int and long. In the context of the above explanation, ints are a weird exception in that they allow you to use a larger single digit (e.g. 64 bits instead of 30), but only as long as the number fits into that single digit. The general principle still applies.
1 Though these integers will have the class long rather than int.