I have a socket that take 60 numbers from another computer in 6 columns and 10 rows. I orderd them with spilit and output is completely right. about first column, I want to take each number separately for calculating moving average filter on them.
Codes:
import socket
import numpy as np
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('192.168.0.1', 2015))
column1 = []
column2 = []
column3 = []
column4 = []
column5 = []
column6 = []
for message in range(10):
message = sock.recv(1024)
a1 = column1.append(message.split()[0])
a2 = column2.append(message.split()[1])
a3 = column3.append(message.split()[2])
a4 = column4.append(message.split()[3])
a5 = column5.append(message.split()[4])
a6 = column6.append(message.split()[5])
b1 =message.split()[0]
b2 = message.split()[1]
b3 = message.split()[2]
b4 = message.split()[3]
b5 = message.split()[4]
b6 = message.split()[5]
print b1
print b2
print b3
print b4
print b5
print b6
if I only print b1, output will be 10 numbers that I want to have tham separately for next function (moving average filter). I need help to make them separate.
I tried a for loop for b1[i] but gives me only first digit of b1.
First, you want to use a list of columns:
columns = [[] for _ in range(6)]
Then you can split the message into a single list:
for message in range(10):
message = sock.recv(1024)
splits = message.split(None, 5) # split into six pieces at most
which you can then append to the list of lists you created before:
for index, item in enumerate(splits):
columns[index].append(item)
Now if you only wish to print the first of those appended numbers, do
print columns[0][0] # first item of first list
The following should get you started. I have created some random data in the format 6 columns by 10 rows. It then splits the raw data into rows, splits each row into columns and then transposes them to get the data per columns.
Each entry in the first column is then displayed with a moving average of the last 3 entries. deque is used to implement an efficient mini queue of the last entries to calculate the moving average with.
import collections
message = """89 39 59 88 46 1 87 21 2 34
59 40 68 74 29 29 26 30 93 38
84 60 44 98 41 29 8 60 61 83
36 44 56 8 50 94 99 1 30 52
5 27 53 85 67 69 38 67 69 26
92 17 4 13 74 89 30 49 44 20"""
rows = message.splitlines()
data = []
for row in rows:
data.append(row.split())
columns = zip(*data)
total = 0
moving = collections.deque()
# Display the moving average for the first column
for entry in columns[0]:
value = int(entry)
moving.append(value)
total += value
if len(moving) > 3: # Length of moving average
total -= moving.popleft()
print "%3d %.1f" % (value, total/float(len(moving)))
For this data, it will display the following output:
89 89.0
59 74.0
84 77.3
36 59.7
5 41.7
92 44.3
Tested using Python 2.7
Related
I need to generate a list of data. The data is randomised based on a seed. As the list has potentially no limit to size, I am thinking of using pagination to send the data back to requester. The list has to be replicable with a given seed by requester.
Unlike getting data from a database where I can specify offset and number of records to retrieve, the random list needs to be created each time ? How do I avoid having to start from the beginning to get to the nth page (for instance) ? eg
import numpy as np
np.random.seed(0)
for i in range(20):
print(f'{i+1}\t=\t{np.random.randint(100)}')
1 = 44
2 = 47
3 = 64
4 = 67
5 = 67
6 = 9
7 = 83
8 = 21
9 = 36
10 = 87
11 = 70
12 = 88
13 = 88
14 = 12
15 = 58
16 = 65
17 = 39
18 = 87
19 = 46
20 = 88
I my page size = 10, how to avoid generating 1-10 by the time I'm generating 11-20 for the 2nd page ?
Thanks.
In my first table I have columns: indeks, il, start and stop. The last two define a range. I need to list (in a new table) all numbers in the range from start to stop, but also save indeks and the other values belonging to the range.
This table shows what kind of data I have (sample):
ID
Indeks
Start
Stop
il
0
A1
1
3
25
1
B1
31
55
5
2
C1
36
900
865
3
D1
900
2500
20
...
...
...
...
...
And this is the table I want to get:
Indeks
Start
Stop
il
kod
A1
1
3
25
1
A1
1
3
25
2
A1
1
3
25
3
B1
31
55
5
31
B1
31
55
5
32
B1
31
55
5
33
...
...
...
...
...
B1
31
55
5
53
B1
31
55
5
54
B1
31
55
5
55
C1
36
900
865
36
C1
36
900
865
37
C1
36
900
865
38
...
...
...
...
...
C1
36
900
865
898
C1
36
900
865
899
C1
36
900
865
900
...
...
...
...
...
EDITET
lidy=pd.read_excel('path' )
lid=pd.DataFrame(lidy)
output = []
for i in range (0,len(lid)):
for j in range (lid.iloc[i,1],lid.iloc[i,2]+1):
y=((lid.iloc[i,0], j))output.append(y)
print(output)
OR
lidy=pd.read_excel('path' )
lid=pd.DataFrame(lidy)
for i in range (0,len(lid)):
for j in range (lid.iloc[i,1],lid.iloc[i,2]+1):
y=((lid.iloc[i,0], j))
print(y)
Two options:
(1 - preferred) Use Pandas (in combination with openpyxl as engine): The Excel-file I'm using is named data.xlsx, and sheet Sheet1 contains your data. Then this
import pandas as pd
df = pd.read_excel("data.xlsx", sheet_name="Sheet1")
df["kod"] = df[["Start", "Stop"]].apply(
lambda row: range(row.iat[0], row.iat[1] + 1), axis=1
)
df = df.iloc[:, 1:].explode("kod", ignore_index=True)
with pd.ExcelWriter("data.xlsx", mode="a", if_sheet_exists="replace") as writer:
df.to_excel(writer, sheet_name="Sheet2", index=False)
should produce the required output in sheet Sheet2. The work is done by putting the required range()s in the new column kod, and then .explode()-ing it.
(2) Use only openpyxl:
from openpyxl import load_workbook
wb = load_workbook(filename="data.xlsx")
ws = wb["Sheet1"]
rows = ws.iter_rows(values_only=True)
# Reading the required column names
data = [list(next(rows)[1:]) + ["kod"]]
for row in rows:
# Read the input data (a row)
base = list(row[1:])
# Create the new data via iterating over the the given range
data.extend(base + [n] for n in range(base[1], base[2] + 1))
if "Sheet2" in wb.sheetnames:
del wb["Sheet2"]
ws_new = wb.create_sheet(title="Sheet2")
for row in data:
ws_new.append(row)
wb.save("data.xlsx")
I have a CSV file which has 255 columns and 16,000 rows of data, and I want to add a list of data which contains 16,000 data to the first column of my CSV file.
The code I tried to use is
# Append the name of the file to List
path = 'C:/Users/User/Desktop/Guanlin_CNN1D/CNN1D/0.3 15 and 105 circle cropped'
list = os.listdir(path)
List = []
for a in list:
List.append(str(a))
## Load the to-be-added CSV file
data = pd.read_csv('C:/Users/User/Desktop/Guanlin_CNN1D/CNN1D/0.3 15 and 105 for toolpath recreatation.csv',sep=',', engine='python' ,header=None)
tempdata = pd.DataFrame(data)
features = tempdata.values[:, 1:]
file_num = tempdata.values[:, 0]
# add the List to first columns of CSV file
Temp = {List,file_num,features}
temp = pd.DataFrame(Temp)
temp
The result shows
TypeError: unhashable type: 'list'
How to rewrite the code?
Thanks in advance!
I think you simply need to use the dataframe insert method. It looks like you are trying to create a new dataframe but I think it is not necessary. Below example inserts a new column at the zeroth position. It looks like you were trying to make a new dataframe from a dict; this link has some easy examples on way to populate a dataframe with lists and dicts. I think the number of rows and columns should not be a concern for you in this case.
import numpy as np
import pandas as pd
np.random.seed(0)
df = pd.DataFrame(np.random.randint(0, 100, size=(5, 5)), columns=list('ABCDE'))
print(df)
df.insert(0,column='newcol', value=np.random.randint(0, 100, size=(5)))
print()
print(df)
df.to_csv( r'data.csv', index=False, header=True)
will produce this output
A B C D E
0 44 47 64 67 67
1 9 83 21 36 87
2 70 88 88 12 58
3 65 39 87 46 88
4 81 37 25 77 72
newcol A B C D E
0 9 44 47 64 67 67
1 20 9 83 21 36 87
2 80 70 88 88 12 58
3 69 65 39 87 46 88
4 79 81 37 25 77 72
Here I have a dataset with three inputs. Three inputs x1,x2,x3. Here I want to read just x2 column and in that column data stepwise row by row.
Here I wrote a code. But it is just showing only letters.
Here is my code
data = pd.read_csv('data6.csv')
row_num =0
x=[]
for col in data:
if (row_num==1):
x.append(col[0])
row_num =+ 1
print(x)
result : x1,x2,x3
What I expected output is:
expected output x2 (read one by one row)
65
32
14
25
85
47
63
21
98
65
21
47
48
49
46
43
48
25
28
29
37
Subset of my csv file :
x1 x2 x3
6 65 78
5 32 59
5 14 547
6 25 69
7 85 57
8 47 51
9 63 26
3 21 38
2 98 24
7 65 96
1 21 85
5 47 94
9 48 15
4 49 27
3 46 96
6 43 32
5 48 10
8 25 75
5 28 20
2 29 30
7 37 96
Can anyone help me to solve this error?
If you want list from x2 use:
x = data['x2'].tolist()
I am not sure I even get what you're trying to do from your code.
What you're doing (after fixing the indentation to make it somewhat correct):
Iterate through all columns of your dataframe
Take the first character of the column name if row_num is equal to 1.
Based on this guess:
import pandas as pd
data = pd.read_csv("data6.csv")
row_num = 0
x = []
for col in data:
if row_num == 1:
x.append(col[0])
row_num = +1
print(x)
What you probably want to do:
import pandas as pd
data = pd.read_csv("data6.csv")
# Make a list containing the values in column 'x2'
x = list(data['x2'])
# Print all values at once:
print(x)
# Print one value per line:
for val in x:
print(val)
When you are using pandas you can use it. You can try this to get any specific column values by using list to direct convert into a list.For loop not needed
import pandas as pd
data = pd.read_csv('data6.csv')
print(list(data['x2']))
For index.csv file, its fourth column has ten numbers ranging from 1-5. Each number can be regarded as an index, and each index corresponds with an array of numbers in filename.csv.
The row number of filename.csv represents the index, and each row has three numbers. My question is about using a nesting loop to transfer the numbers in filename.csv to index.csv.
from numpy import genfromtxt
import numpy as np
import csv
import collections
data1 = genfromtxt('filename.csv', delimiter=',')
data2 = genfromtxt('index.csv', delimiter=',')
out = np.zeros((len(data2),len(data1)))
for row in data2:
for ch_row in range(len(data1)):
if (row[3] == ch_row + 1):
out = row.tolist() + data1[ch_row].tolist()
print(out)
writer = csv.writer(open('dn.csv','w'), delimiter=',',quoting=csv.QUOTE_ALL)
writer.writerow(out)
For example, the fourth column of index.csv contains 1,2,5,3,4,1,4,5,2,3 and filename.csv contains:
# filename.csv
20 30 50
70 60 45
35 26 77
93 37 68
13 08 55
What I need is to write the indexed row from filename.csv to index.csv and store these number in 5th, 6th and 7th column:
# index.csv
# 4 5 6 7
... 1 20 30 50
... 2 70 60 45
... 5 13 08 55
... 3 35 26 77
... 4 93 37 68
... 1 20 30 50
... 4 93 37 68
... 5 13 08 55
... 2 70 60 45
... 3 35 26 77
If I do "print(out)", it comes out a correct answer. However, when I input "out" in the shell, there are only one row appears like [1.0, 1.0, 1.0, 1.0, 20.0, 30.0, 50.0]
What I need is to store all the values in the "out" variables and write them to the dn.csv file.
This ought to do the trick for you:
Code:
from csv import reader, writer
data = list(reader(open("filename.csv", "r"), delimiter=" "))
out = writer(open("output.csv", "w"), delimiter=" ")
for row in reader(open("index.csv", "r"), delimiter=" "):
out.writerow(row + data[int(row[3])])
index.csv:
0 0 0 1
0 0 0 2
0 0 0 3
filename.csv:
20 30 50
70 60 45
35 26 77
93 37 68
13 08 55
This produces the output:
0 0 0 1 70 60 45
0 0 0 2 35 26 77
0 0 0 3 93 37 68
Note: There's no need to use numpy here. The stadard library csv module will do most of the work for you.
I also had to modify your sample datasets a bit as what you showed had indexes out of bounds of the sample data in filename.csv.
Please also note that Python (like most languages) uses 0th indexes. So you may have to fiddle with the above code to exactly fit your needs.
with open('dn.csv','w') as f:
writer = csv.writer(f, delimiter=',',quoting=csv.QUOTE_ALL)
for row in data2:
idx = row[3]
out = [idx] + [x for x in data1[idx-1]]
writer.writerow(out)