We Keep Coding

Python: accidentally created a reference but not sure how

I imagine this is one in a very long list of questions from people who have inadvertantly created references in python, but I've got the following situation. I'm using scipy minimize to set the sum of the top row of an array to 5 (as an example).
class problem_test:
def __init__(self):
test_array = [[1,2,3,4,5,6,7],
[4,5,6,7,8,9,10]]
def set_top_row_to_five(x, array):
array[0] = array[0] + x
return abs(sum(array[0]) - 5)
adjustment = spo.minimize(set_top_row_to_five,0,args=(test_array))
print(test_array)
print(adjustment.x)
ptest = problem_test()
However, the optimization is altering the original array (test_array):
[array([-2.03, -1.03, -0.03, 0.97, 1.97, 2.97, 3.97]), [4, 5, 6, 7, 8, 9, 10]]
[-0.00000001]
I realize I can solve this using, for example, deepcopy, but I'm keen to learn why this is happening so I don't do the same in future by accident.
Thanks in advance!

Names are references to objects. What is to observe is whether the objects (also passed in an argument) is modified itself or a new object is created. An example would be:
>>> l1 = list()
>>> l2 = l1
>>> l2.append(0) # this modifies object currently reference to by l1 and l2
>>> print(l1)
[0]
Whereas:
>>> l1 = list()
>>> l2 = list(l1) # New list object has been created with initial values from l1
>>> l2.append(0)
>>> print(l1)
[]
Or:
>>> l1 = list()
>>> l2 = l1
>>> l2 = [0] # New list object has been created and assigned to l2
>>> l2.append(0)
>>> print(l1)
[]
Similarly assuming l = [1, 2, 3]:
>>> def f1(list_arg):
... return list_arg.reverse()
>>> print(f1, l)
None [3, 2, 1]
We have just passed None returned my list.reverse method through and reversed l (in place). However:
>>> def f2(list_arg):
... ret_list = list(list_arg)
... ret_list.reverse()
... return ret_list
>>> print(f2(l), l)
[3, 2, 1] [1, 2, 3]
Function returns a new reversed object (initialized) from l which remained unchanged (NOTE: in this exampled built-in reversed or slicing would of course make more sense.)
When nested, one must not forget that for instance:
>>> l = [1, 2, 3]
>>> d1 = {'k': l}
>>> d2 = dict(d1)
>>> d1 is d2
False
>>> d1['k'] is d2['k']
True
Dictionaries d1 and d2 are two different objects, but their k item is only one (and shared) instance. This is the case when copy.deepcopy might come in handy.
Care needs to be taken when passing objects around to make sure they are modified or copy is used as wanted and expected. It might be helpful to return None or similar generic value when making in place changes and return the resulting object when working with a copy so that the function/method interface itself hints what the intention was and what is actually going on here.
When immutable objects (as the name suggests) are being "modified" a new object would actually be created and assigned to a new or back to the original name/reference:
>>> s = 'abc'
>>> print('0x{:x} {}'.format(id(s), s))
0x7f4a9dbbfa78 abc
>>> s = s.upper()
>>> print('0x{:x} {}'.format(id(s), s))
0x7f4a9c989490 ABC
Note though, that even immutable type could include reference to a mutable object. For instance for l = [1, 2, 3]; t1 = (l,); t2 = t1, one can t1[0].append(4). This change would also be seen in t2[0] (for the same reason as d1['k'] and d2['k'] above) while both tuples themselves remained unmodified.
One extra caveat (possible gotcha). When defining default argument values (using mutable types), that default argument, when function is called without passing an object, behaves like a "static" variable:
>>> def f3(arg_list=[]):
... arg_list.append('x')
... print(arg_list)
>>> f3()
['x']
>>> f3()
['x', 'x']
Since this is often not a behavior people assume at first glance, using mutable objects as default argument value is usually better avoided.
Similar would be true for class attributes where one object would be shared between all instances:
>>> class C(object):
... a = []
... def m(self):
... self.a.append('x') # We actually modify value of an attribute of C
... print(self.a)
>>> c1 = C()
>>> c2 = C()
>>> c1.m()
['x']
>>> c2.m()
['x', 'x']
>>> c1.m()
['x', 'x', 'x']
Note what the behavior would be in case of class immutable type class attribute in a similar example:
>>> class C(object):
... a = 0
... def m(self):
... self.a += 1 # We assign new object to an attribute of self
... print(self.a)
>>> c1 = C()
>>> c2 = C()
>>> c1.m()
1
>>> c2.m()
1
>>> c1.m()
2
All the fun details can be found in the documentation: https://docs.python.org/3.6/reference/datamodel.html

Modifying a list outside of recursive function in Python

There is a multidimensional list with not clear structure:
a=[[['123', '456'], ['789', '1011']], [['1213', '1415']], [['1617', '1819']]]
And there is a recursive function, which operates the list:
def get_The_Group_And_Rewrite_It(workingList):
if isinstance(workingList[0],str):
doSomething(workingList)
else:
for i in workingList:
get_The_Group_And_Rewrite_It(i)
Through time the get_The_Group_And_Rewrite_It() should get a list, for instance, ['123','456'], and as soon as it get it, doSomething function should rewrite it with ['abc'] in entire list.
The same with other lists of format[str,str,...]. In the end I should get something like
a=[[['abc'], ['abc']], [['abc']], [['abc']]]
I see it would be easy in C++, using *links, but how to do that in Python?

For this case, you can use slice assignment:
>>> a = [[['123', '456']]]
>>> x = a[0][0]
>>> x[:] = ['abc']
>>> a
[[['abc']]]
>>> def f(workingList):
... if isinstance(workingList[0],str):
... workingList[:] = ['abc']
... else:
... for i in workingList:
... f(i)
...
>>> a=[[['123', '456'], ['789', '1011']], [['1213', '1415']], [['1617', '1819']]]
>>> f(a)
>>> a
[[['abc'], ['abc']], [['abc']], [['abc']]]

Set class attribute values in a generator

I have a list of lists. Each sublist contains objects of a custom class. What I want to do is set a certain attribute of each class object to 0. The simple way to do this would be a double for loop or similar:
for subl in L:
for myObj in subL:
myObj.attr = 0
Alternatively, I could use itertools.chain:
for myObj in itertools.chain.from_iterable(L):
myObj.attr = 0
However, I wonder if I could set everything in one line. Could I perhaps use a generator-like structure to do this? Something along the lines of:
(myObj.attr=0 for subl in L for myObj in subl)
Now that won't really work, and will raise a SyntaxError, but is something even remotely similar possible?

This is an abuse of generator expressions, but:
any(setattr(obj, "attr", 0) for sub in L for obj in sub)
Or, perhaps slightly faster since there's no testing of each object:
from collections import deque
do = deque(maxlen=0).extend
do(setattr(obj, "attr", 0) for sub in L for obj in sub)

See this example:
class C:
def __init__(self):
self.a = None
def f(self, para):
self.a = para
list1 = [C() for e in range(3)]
list2 = [C() for e in range(3)]
list3 = [list1, list2]
[c.f(5) for l in list3 for c in l]
for e in list3:
for c in e:
print c.a
Conclusion
You could create a method to set the attribute. It will look something like:
[myObj.setattr(0) for subl in L for myObj in subl]
Note the brackets.

Here is a simple solution that popped out in my head.
Using the built-in setattr, your suggestion - itertools.chain.from_iterable -, and an abuse of list comprehension:
class Foo():
def __init__(self):
my_attr = 10
A = Foo()
B = Foo()
C = Foo()
D = Foo()
obj_list = [[A, B], [C, D]]
a = [setattr(obj, "my_attr", 0) for obj in itertools.chain.from_iterable(obj_list)]
Result:
>>> a
[None, None, None, None]
>>> A.my_attr
0
>>> B.my_attr
0
>>> C.my_attr
0
>>> D.my_attr
0
I found setattr to be very useful for cases like this, it's simple, short, and effective.
Hope this helps!

Scrapy Only Returning First Result in Loop

I have a loop (as shown below) that executes twice (indexes 1->3), but Scrapy only returns the first trackname in both results. But the print item line shows different values for str_selector so I know the loop works but Scrapy isn't seeing the changing value of x.
Any idea what mistake I have made?
items = []
item = scrapyItem()
for x in range (1,3):
str_selector = '//tr[#name="tracks-grid-browse_track_{0}"]/td[contains(#class,"secondColumn")]/a/text()'.format(x)
item['trackname'] = hxs.select(str_selector).extract()
print item
items.append(item)
return items

It's just that you should build a new item for each iteration, instead of keeping the same: you add in items the same object, which is mutable (as for all user-defined classes by default in python) and so when you update item['trackname'], all items contained are updated !
Here is some code to illustrate:
>>> class C(object):
# Basic user-defined class
def __init__(self):
self.test = None
>>> c = C()
>>> items = []
>>> for x in range (1,3):
c.test = x
print c, c.test
items.append(c)
<__main__.C object at 0x01CEB130> 1
<__main__.C object at 0x01CEB130> 2
>>> items # All objects contained are the same !!!
[<__main__.C object at 0x01CEB130>, <__main__.C object at 0x01CEB130>]
>>> for c in items:
print c.test
2
2
Now create a new object each time:
>>> items = []
>>> for x in range (1,3):
c = C()
c.test = x
print c, c.test
items.append(c)
<__main__.C object at 0x01CEB110> 1
<__main__.C object at 0x011F2270> 2
Objects are now different !
>>> for c in items:
print c.test
1
2

what actually you are doing right now is creating an item object and changing its value in loop, you need to create item in loop.
items = []
#item = scrapyItem()
for x in range (1,3):
item = scrapyItem()
str_selector = '//tr[#name="tracks-grid-browse_track_{0}"]/td[contains(#class,"secondColumn")]/a/text()'.format(x)
item['trackname'] = hxs.select(str_selector).extract()
print item
items.append(item)
return items

Remove object from a list of objects in python

In Python, how can I remove an object from a list of objects? Like this:
x = object()
y = object()
array = [x, y]
# Remove x
I've tried array.remove() but it only works with a value, not a specific location in the array. I need to be able to delete the object by addressing its position (remove array[0]).

There are various ways to delete an object from a list:
my_list = [1,2,4,6,7]
del my_list[1] # Removes index 1 from the list
print my_list # [1,4,6,7]
my_list.remove(4) # Removes the integer 4 from the list, not the index 4
print my_list # [1,6,7]
my_list.pop(2) # Removes index 2 from the list
In your case the appropriate method to use is pop, because it takes the index to be removed:
x = object()
y = object()
array = [x, y]
array.pop(0)
# Using the del statement
del array[0]

del array[0]
where 0 is the index of the object in the list (there is no array in python)

If you want to remove multiple object from a list. There are various ways to delete an object from a list
Try this code. a is list with all object, b is list object you want to remove.
example :
a = [1,2,3,4,5,6]
b = [2,3]
for i in b:
if i in a:
a.remove(i)
print(a)
the output is [1,4,5,6]
I hope, it will work for you

You could try this to dynamically remove an object from an array without looping through it? Where e and t are just random objects.
>>> e = {'b':1, 'w': 2}
>>> t = {'b':1, 'w': 3}
>>> p = [e,t]
>>> p
[{'b': 1, 'w': 2}, {'b': 1, 'w': 3}]
>>>
>>> p.pop(p.index({'b':1, 'w': 3}))
{'b': 1, 'w': 3}
>>> p
[{'b': 1, 'w': 2}]
>>>

You can remove a string from an array like this:
array = ["Bob", "Same"]
array.remove("Bob")

if you wanna remove the last one just do your_list.pop(-1)
if you wanna remove the first one your_list.pop(0) or any index you wish to remove

If you know the array location you can can pass it into itself. If you are removing multiple items I suggest you remove them in reverse order.
#Setup array
array = [55,126,555,2,36]
#Remove 55 which is in position 0
array.remove(array[0])

If we have an object x (e.g. an instance of some class X) that's in a list of other objects of the same type, we can simply directly remove it using list.remove(), provided the class has __eq__ properly implemented (this is already the case for built-in object types, but for custom classes always remember to implement the eq dunder function):
class X():
def __init__(self, value=None):
self.a = value
def __eq__(self, other):
if not hasattr(other, 'a'):
return False
return self.a == other.a
x = X(4)
y = X(10)
z = X(9)
my_list = [x,y,z]
print([e.a for e in a])
# prints [4, 10, 9]
my_list.remove(x)
print([e.a for e in a])
# prints [10, 9]