We Keep Coding

python panda apply compare to external list and remove part of list

I have a parking lot with cars of different models (nr) and the cars are so closely packed that in order for one to get out one might need to move some others. A little like a 15Puzzle, only I can take one or more cars out of the parking lot. Ordered_car_List includes the cars that will be picked up today, and they need to be taken out of the parking lot with as few non-ordered cars as possible moved. There are more columns to this panda, but this is what I can't figure out.
I have a Program that works good for small sets of data, but it seems that this is not the way of the PANDAS :-)
I have this:
cars = pd.DataFrame({'x': [1,1,1,1,1,2,2,2,2],
'y': [1,2,3,4,5,1,2,3,4],
'order_number':[6,6,7,6,7,9,9,10,12]})
cars['order_number_no_dublicates_down'] = None
Ordered_car_List = [6,9,9,10,28]
i=0
while i < len(cars):
temp_val = cars.at[i, 'order_number']
if temp_val in Ordered_car_List:
cars.at[i, 'order_number_no_dublicates_down'] = temp_val
Ordered_car_List.remove(temp_val)
i+=1
If I use cars.apply(lambda..., how can I change the Ordered_car_List in each iteration?
Is there another approach that I can take?
I found this page, and it made me want to be faster. The Lambda approach is in the middle when it comes to speed, but it still is so much faster than what I am doing now.
https://towardsdatascience.com/how-to-make-your-pandas-loop-71-803-times-faster-805030df4f06

Updating cars
We can vectorize this based on two counters:
cumcount() to cumulatively count each unique value in cars['order_number']
collections.Counter() to count each unique value in Ordered_car_List
cumcount = cars.groupby('order_number').cumcount().add(1)
maxcount = cars['order_number'].map(Counter(Ordered_car_List))
# order_number cumcount maxcount
# 0 6 1 1
# 1 6 2 1
# 2 7 1 0
# 3 6 3 1
# 4 7 2 0
# 5 9 1 2
# 6 9 2 2
# 7 10 1 1
# 8 12 1 0
So then we only want to keep cars['order_number'] where cumcount <= maxcount:
either use DataFrame.loc[]
cars.loc[cumcount <= maxcount, 'nodup'] = cars['order_number']
or Series.where()
cars['nodup'] = cars['order_number'].where(cumcount <= maxcount)
or Series.mask() with the condition inverted
cars['nodup'] = cars['order_number'].mask(cumcount > maxcount)
Updating Ordered_car_List
The final Ordered_car_List is a Counter() difference:
Used_car_List = cars.loc[cumcount <= maxcount, 'order_number']
# [6, 9, 9, 10]
Ordered_car_List = list(Counter(Ordered_car_List) - Counter(Used_car_List))
# [28]
Final output
cumcount = cars.groupby('order_number').cumcount().add(1)
maxcount = cars['order_number'].map(Counter(Ordered_car_List))
cars['nodup'] = cars['order_number'].where(cumcount <= maxcount)
# x y order_number nodup
# 0 1 1 6 6.0
# 1 1 2 6 NaN
# 2 1 3 7 NaN
# 3 1 4 6 NaN
# 4 1 5 7 NaN
# 5 2 1 9 9.0
# 6 2 2 9 9.0
# 7 2 3 10 10.0
# 8 2 4 12 NaN
Used_car_List = cars.loc[cumcount <= maxcount, 'order_number']
Ordered_car_List = list(Counter(Ordered_car_List) - Counter(Used_car_List))
# [28]
Timings
Note that your loop is still very fast with small data, but the vectorized counter approach just scales much better:

How to parse tables from .txt files using Pandas

I'm trying to extract tables from log files which are in .txt format. The file is loaded using read_csv() from pandas.
The log file looks like this:
aaa
bbb
ccc
=====================
A B C D E F
=====================
1 2 3 4 5 6
7 8 9 1 2 3
4 5 6 7 8 9
1 2 3 4 5 6
---------------------
=====================
G H I J
=====================
1 3 4
5 6 7
---------------------
=====================
K L M N O
=====================
1 2 3
4 5 6
7 8 9
---------------------
xxx
yyy
zzz
Here are some points about the log file:
Files start and end with some lines of comment which can be ignored.
In the example above there are three tables.
Headers for each table are located between lines of "======..."
The end of each table is signified by a line of "------..."
My code as of now:
import pandas as pd
import itertools
df = pd.read_csv("xxx.txt", sep="\n", header=None)
# delimiters for header and end-of-table
h_dl = "=" * 21
r_dl = "-" * 21
for i in range(len(df.index)-2):
# if loop to find lines which are table headers & convert to list
if (df.iloc[i].any() == h_dl) & (df.iloc[i+2].any() == h_dl):
h = df.iloc[i+1].str.split().tolist()
h = list(itertools.chain(*h))
# while loop to find lines which are table rows & append to one list
x = 3
r = []
while True:
if df.iloc[i+x].any() == r_dl:
break
r.append(df.iloc[i+x].str.split().tolist())
x += 1
r = list(itertools.chain(*r))
# create pandas dataframe with header and rows obtained above
t = pd.DataFrame(data=r, columns=h)
This code returns AssertionError: 14 columns passed, passed data had 15 columns. I know that this is due to the fact that for the table rows, I am using .str.split() which by default splits on whitespace. Since there are some columns for which there are missing values, the number of elements in table headers and number of elements in table rows does not match for the second and htird table. I am struggling to get around this, since the number of whitespace characters to signify missing values is different for each table.
My question is: is there a way to account for missing values in some of the columns, so that I can get a DataFrame as output where there are either null or NaN for missing values as appropriate?

With usage of Victor Ruiz method I added if options to handle different header sizes.
=^..^=
Description in code:
import re
import pandas as pd
import itertools
df = pd.read_csv("stack.txt", sep="\n", header=None)
# delimiters for header and end-of-table
h_dl = "=" * 21
r_dl = "-" * 21
for i in range(len(df.index)-2):
# if loop to find lines which are table headers & convert to list
if (df.iloc[i].any() == h_dl) & (df.iloc[i+2].any() == h_dl):
h = df.iloc[i+1].str.split().tolist()
h = list(itertools.chain(*h))
# get header string
head = df.iloc[i+1].to_string()
# get space distance in header
space_range = 0
for result in re.findall('([ ]*)', head):
if len(result) > 0:
space_range = len(result)
x = 3
r = []
while True:
if df.iloc[i+x].any() == r_dl:
break
# strip line
line = df.iloc[i+x].to_string()[5::]
# collect items based on elements distance
items = []
for result in re.finditer('(\d+)([ ]*)', line):
item, delimiter = result.groups()
items.append(item)
if len(delimiter) > space_range*2+1:
items.append('NaN')
items.append('NaN')
if len(delimiter) < space_range*2+2 and len(delimiter) > space_range:
items.append('NaN')
r.append([items])
x += 1
r = list(itertools.chain(*r))
# create pandas dataframe with header and rows obtained above
t = pd.DataFrame(data=r, columns=h)
Output:
A B C D E F
0 1 2 3 4 5 6
1 7 8 9 1 2 3
2 4 5 6 7 8 9
3 1 2 3 4 5 6
G H I J
0 1 NaN 3 4
1 5 NaN 6 7
K L M N O
0 1 NaN NaN 2 3
1 4 5 NaN NaN 6
2 7 8 NaN 9 None

Maybe this can help you.
Suppose we have the next line of text:
1 3 4
The problem is to identify how much spaces delimits two consecutive items without considering that there is a missing value between them.
Let consider that 5 spaces is a delimiter, and more than 5 is a missing value.
You can use regex to parse the items:
from re import finditer
line = '1 3 4'
items = []
for result in finditer('(\d+)([ ]*)', line):
item, delimiter = result.groups()
items.append(item)
if len(delimiter) > 5:
items.append(nan)
print(items)
Output is:
['1', nan, '3', '4']
A more complex situation would be if it can appear two or more consecutive missing values (the code above will just inyect only one nan)

How to delete a matrix cell's neighbors which are the same value with it

I have a matrix as shown below (taken from a txt file with an argument), and every cell has neighbors. Once you pick a cell, that cell and all neighboring cells that containing the same number will disappear.
1 0 4 7 6 8
0 5 4 4 5 5
2 1 4 4 4 6
4 1 3 7 4 4
I've tried to do this with using recursion. I separated function four parts which are up(), down() , left() and right(). But I got an error message: RecursionError: maximum recursion depth exceeded in comparison
cmd=input("Row,column:")
cmdlist=command.split(",")
row,column=int(cmdlist[0]),int(cmdlist[1])
num=lines[row-1][column-1]
def up(x,y):
if lines[x-2][y-1]==num and x>1:
left(x,y)
right(x,y)
lines[x-2][y-1]=None
def left(x,y):
if lines[x-1][y-2]==num and y>1:
up(x,y)
down(x,y)
lines[x-1][y-2]=None
def right(x,y):
if lines[x-1][y]==num and y<len(lines[row-1]):
up(x,y)
down(x,y)
lines[x-1][y]=None
def down(x,y):
if lines[x][y-1]==num and x<len(lines):
left(x,y)
right(x,y)
lines[x][y-1]=None
up(row,column)
down(row,column)
for i in lines:
print(str(i).strip("[]").replace(",","").replace("None"," "))
When I give the input (3,3) which represents the number of "4", the output must be like this:
1 0 7 6 8
0 5 5 5
2 1 6
4 1 3 7
I don't need fixed code, just the main idea will be enough. Thanks a lot.

Recursion error happens when your recursion does not terminate.
You can solve this without recursing using set's of indexes:
search all indexes that contain the looked for number into all_num_idx
add the index you are currently at (your input) to a set tbd (to be deleted)
loop over the tbd and add all indexed from all_num_idx that differ only in -1/+1 in row or col to any index thats already in the set
do until tbd does no longer grow
delete all indexes from tbd:
t = """4 0 4 7 6 8
0 5 4 4 5 5
2 1 4 4 4 6
4 1 3 7 4 4"""
data = [k.strip().split() for k in t.splitlines()]
row,column=map(int,input("Row,column:").strip().split(";"))
num = data[row][column]
len_r =len(data)
len_c = len(data[0])
all_num_idx = set((r,c) for r in range(len_r) for c in range(len_c) if data[r][c]==num)
tbd = set( [ (row,column)] ) # inital field
tbd_size = 0 # different size to enter while
done = set() # we processed those already
while len(tbd) != tbd_size: # loop while growing
tbd_size=len(tbd)
for t in tbd:
if t in done:
continue
# only 4-piece neighbourhood +1 or -1 in one direction
poss_neighbours = set( [(t[0]+1,t[1]), (t[0],t[1]+1),
(t[0]-1,t[1]), (t[0],t[1]-1)] )
# 8-way neighbourhood with diagonals
# poss_neighbours = set((t[0]+a,t[1]+b) for a in range(-1,2) for b in range(-1,2))
tbd = tbd.union( poss_neighbours & all_num_idx)
# reduce all_num_idx by all those that we already addded
all_num_idx -= tbd
done.add(t)
# delete the indexes we collected
for r,c in tbd:
data[r][c]=None
# output
for line in data:
print(*(c or " " for c in line) , sep=" ")
Output:
Row,column: 3,4
4 0 7 6 8
0 5 5 5
2 1 6
4 1 3 7
This is a variant of a "flood-fill-algorythm" flooding only cells of a certain value. See https://en.wikipedia.org/wiki/Flood_fill

Maybe you should replace
def right(x,y):
if lines[x-1][y]==num and y<len(lines[row-1]):
up(x,y)
down(x,y)
lines[x-1][y]=None
by
def right(x,y):
if lines[x-1][y]==num and y<len(lines[row-1]):
lines[x-1][y]=None
up(x - 1,y)
down(x - 1,y)
right(x - 1, y)
and do the same for all the other functions.
Putting lines[x-1][y]=None ensure that your algorithm stops and changing the indices ensure that the next step of your algorithm will start from the neighbouring cell.

Sampling a matrix with conditions (no zeros or repeated columns)

In case you are interested in the background of the question, I'm thinking how to solve this post- incidentally, if you solve it there, I'll just erase this question. Ideally, I'd like to get an analytical or algebraic solution (constrained non-capturing rook problem), but short of that I'd like a simulation. Incidentally, I posted a related question without as much detail, in case it is easier to tackle.
But you don't have to leave this page. Basically there are pairings of two lists of soccer teams, and some pairings are good, while others are forbidden by the rules. This is the matrix:
So to generate multiple samplings to match the teams on the row names (to the left) with the column names of opposing teams (at the top), I have to come up with a conditional sampling procedure, but I have no clue how to.
This is what I have attempted so far:
BCN = c(0,2,3,4,0,0,7,8)
ATL = c(0,0,3,4,5,0,7,8)
DOR = c(0,0,3,4,5,6,7,0)
MON = c(1,2,3,0,5,6,7,0)
ARS = c(1,2,3,0,0,6,7,8)
LEI = c(1,2,3,4,0,6,0,8)
JUV = c(1,2,3,4,5,0,7,8)
NAP = c(1,2,0,4,5,6,7,8)
chessboard = t(as.matrix(data.frame(BCN, ATL, DOR, MON, ARS, LEI, JUV, NAP)))
colnames(chessboard) = c("MAD", "BYN", "BEN", "PSG", "MCY", "SEV", "OPO", "LEV")
chessboard
MAD BYN BEN PSG MCY SEV OPO LEV
BCN 0 2 3 4 0 0 7 8
ATL 0 0 3 4 5 0 7 8
DOR 0 0 3 4 5 6 7 0
MON 1 2 3 0 5 6 7 0
ARS 1 2 3 0 0 6 7 8
LEI 1 2 3 4 0 6 0 8
JUV 1 2 3 4 5 0 7 8
NAP 1 2 0 4 5 6 7 8
match = function(){
vec = rep(0,8)
for(i in 1:8){
tryCatch({vec[i] = as.numeric(sample(as.character(chessboard[i,][!(chessboard[i,] %in% vec) & chessboard[i,] > 0]),1))
last=chessboard[8,][!(chessboard[8,] %in% vec) & chessboard[i,] > 0]
},error=function(e){})
}
vec
}
match()
set.seed(0)
nsim = 100000
matches = t(replicate(nsim, match()))
matches = subset(matches, matches[,8]!=0)
colnames(matches) = c("BCN", "ATL", "DOR", "MON", "ARS", "LEI", "JUV", "NAP")
head(matches)
table = apply(matches, 2, function(x) table(x)/nrow(matches))
table
$BCN
x
2 3 4 7 8
0.1969821 0.2125814 0.1967272 0.1967166 0.1969927
$ATL
x
3 4 5 7 8
0.2016226 0.1874462 0.2357732 0.1875737 0.1875843
$DOR
x
3 4 5 6 7
0.1773264 0.1686188 0.2097673 0.2787270 0.1655605
$MON
x
1 2 3 5 6 7
0.2567882 0.2031199 0.1172017 0.1341921 0.1789617 0.1097365
$ARS
x
1 2 3 6 7 8
0.2368882 0.1907169 0.1104480 0.1651358 0.1026112 0.1941999
$LEI
x
1 2 3 4 6 8
0.2129743 0.1717302 0.1019210 0.1856410 0.1511081 0.1766255
$JUV
x
1 2 3 4 5 7 8
0.15873252 0.12940289 0.07889902 0.14203948 0.22837179 0.12845781 0.13409648
$NAP
x
1 2 4 5 6 7 8
0.1346168 0.1080481 0.1195272 0.1918956 0.2260675 0.1093436 0.1105011

Maybe try this:
matches = setNames(as.list(rep(NA,8)), rownames(mat))
set.seed(1)
# For each row, sample a column, then drop that column.
# 'sample.int' will automatically renormalize the probabilities.
for (i in sample.int(8)) {
team_i = rownames(mat)[i]
j = sample.int(ncol(mat), 1, prob=mat[i,])
matches[[team_i]] = colnames(mat)[j]
mat = mat[,-j,drop=FALSE]
}
> matches
# $Barcelona
# [1] "Oporto"
#
# $Atletico
# [1] "Benfica"
#
# $Dortmund
# [1] "Paris"
#
# $Juventus
# [1] "City"
#
# $Arsenal
# [1] "Sevilla"
#
# $Napoli
# [1] "Leverkusen"
#
# $Monaco
# [1] "Bayern"
#
# $Leicester
# [1] "Madrid"
Might be a good idea to add restrictions so you don't end up with a row of zeros.

Keeping Python from spacing after breaking a line when printing a List

(yes, I've searched all around for a solution, and, if did I see it, I wasn't able to relate to my issue. I'm new to Python, sorry!)
I've got a work to do, and it says to me:
"User will input X and Y. Show a sequence from 1 to Y, with only X elements each line."
e.g
2 4 as entrance
1 2
3 4
e.g 2 6
1 2
3 4
5 6
Okay... So, I thought on doing this:
line, final = input().split()
line = int(line)
final = int(final)
List = []
i = 0
total = (final // line)
spot = 0
correction = 0
k = 1
if i != final:
List = list(range(1, final + 1, 1))
i += 1
while k != total:
spot = line * k + correction
correction += 1
k += 1
list.insert(List, spot, '\n')
print(*List)
Ok. So I managed to build my List from 1 to the "final" var.
Also managed to find on which spots (therefore, var "spot") my new line would be created. (Had to use a correction var and some math to reach it, but it's 10/10)
So far, so good.
The only problem is this work is supposed to be delivered on URI Online Judge, and it DEMANDS that my result shows like this:
2 10 as entrance
1 2
3 4
5 6
7 8
9 10
And, using the code I just posted, I get this as a result:
1 2
3 4
5 6
7 8
9 10
Thus, it says my code is wrong. I've tried everything to remove those spaces (I think). Using sys won't work since it only prints one argument. Tried using join (but I could have done it wrong, as I'm new anyway)
Well, I've tried pretty much anything. Hope anyone can help me.
Thanks in advance :)

You have built a list that includes each necessary character, including the linefeed. Therefore, you have a list like this:
[1, 2, '\n', 3, 4, '\n'...]
When you unpack arguments to print(), it puts a separator between each argument, defaulting to a space. So, it prints 1, then a space, then 2, then a space, then a linefeed, then a space... And that is why you have a space at the beginning of each line.
Instead of inserting linefeeds into a list, chunk that list with iter and next:
>>> def chunks(x, y):
... i = iter(range(1, y+1))
... for row in range(y//x):
... print(*(next(i) for _ in range(x)))
... t = tuple(i)
... if t:
... print(*t)
...
>>> chunks(2, 6)
1 2
3 4
5 6
>>> chunks(2, 7)
1 2
3 4
5 6
7

The problem with the approach you're using is a result of a space being printed after each "\n" character in the series. While the idea was quite clever, unfortunately, I think this means you will have to take a different approach from inserting the newline character into the list.
Try this approach: (EDITED)
x, y = input().split()
x, y = int(x), int(y)
for i in range(1, y+1):
if i % x == 0 or i == y:
print(i)
else:
print(i, end=" ")
Output for 3 11
1 2 3
4 5 6
7 8 9
10 11
Output for 2 10
1 2
3 4
5 6
7 8
9 10

Use itertools to take from an iterable in chunks:
>>> import itertools
>>> def print_stuff(x,y):
... it = iter(range(1, y + 1))
... chunk = list(itertools.islice(it,X))
... while chunk:
... print(*chunk)
... chunk = list(itertools.islice(it,X))
...
>>> print_stuff(2,4)
1 2
3 4
>>>
And here:
>>> print_stuff(2,10)
1 2
3 4
5 6
7 8
9 10
>>>

I split user input into two string then convert them into int and comapre if y greater than x by 2 because this is minimum for drawing your sequence
Then i make a list from 1 to y
And iterate over it 2 element for each iteration printing them
x,y=input().split()
if int(y)>int(x)+2:
s=range(1,int(y)+1)
for i in range(0,len(s),2):
print(' '.join(str(d) for d in s[i:i+2]))
result:
1 2
3 4
5 6
7 8
9 10