We Keep Coding

ValueError: Expected object or value in pandas using lines=True

my test_data.json file format is
[{"Gender": "Male", "HeightCm": 171, "WeightKg": 96 },
{ "Gender": "Male", "HeightCm": 161, "WeightKg": 85 },
{ "Gender": "Male", "HeightCm": 180, "WeightKg": 77 },
{ "Gender": "Female", "HeightCm": 166, "WeightKg": 62},
{"Gender": "Female", "HeightCm": 150, "WeightKg": 70},
{"Gender": "Female", "HeightCm": 167, "WeightKg": 82}]
when I read the json file in panda using
basePath = os.path.dirname(os.path.abspath(__file__))
reader=pd.read_json(basePath+'/test_data.json',orient="records")
It converted json file in the perfect dataframe i.e.
Gender HeightCm WeightKg
0 Male 171 96
1 Male 161 85
2 Male 180 77
3 Female 166 62
4 Female 150 70
5 Female 167 82
but now when i put the lines=True in the pd.read_json() i.e.
basePath = os.path.dirname(os.path.abspath(__file__))
reader=pd.read_json(basePath+'/test_data.json',orient="records",lines=True)
its generating the error
Traceback (most recent call last):
File "test1.py", line 5, in <module>
reader=pd.read_json(basePath+'/test_data.json',orient="records",lines=True)
File "/home/king_leo/.local/lib/python3.8/site-packages/pandas/util/_decorators.py", line 207, in wrapper
return func(*args, **kwargs)
File "/home/king_leo/.local/lib/python3.8/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/home/king_leo/.local/lib/python3.8/site-packages/pandas/io/json/_json.py", line 614, in read_json
return json_reader.read()
File "/home/king_leo/.local/lib/python3.8/site-packages/pandas/io/json/_json.py", line 746, in read
obj = self._get_object_parser(self._combine_lines(data_lines))
File "/home/king_leo/.local/lib/python3.8/site-packages/pandas/io/json/_json.py", line 770, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "/home/king_leo/.local/lib/python3.8/site-packages/pandas/io/json/_json.py", line 885, in parse
self._parse_no_numpy()
File "/home/king_leo/.local/lib/python3.8/site-packages/pandas/io/json/_json.py", line 1164, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None
ValueError: Expected object or value

How to make dataframe from list of list

I have searched but till not able to figure out how to make data frame from below:
0 ([179, 142, 176, 177, 176, 180, 180, 180, 180,...
1 ([353, 314, 349, 349, 344, 359, 359, 359, 359,...
2 ([535, 504, 535, 535, 535, 540, 540, 540, 540,...
3 ([711, 664, 703, 703, 703, 721, 721, 721, 721,...
4 ([850, 810, 822, 822, 842, 857, 857, 857, 857,.
below is how single data looks
([179, 142, 176],
['Qtr- Oct-20','Qtr- Oct-20','Qtr- Oct-20',],
['High','Low','Close'],
[43.8, 26.05,33.1])
what i want is
0 1 2 3
0 179 Qtr- Oct-20 High 43.8
1 142 Qtr- Oct-20 Low 26.05
2 176 Qtr- Oct-20 High_Volume 1123132
3 177 Qtr- Oct-20 High_Delivery 42499
what i am getting
0
0 ([179, 142, 176, 177, 176, 180, 180, 180, 180,...
1 ([353, 314, 349, 349, 344, 359, 359, 359, 359,...
2 ([535, 504, 535, 535, 535, 540, 540, 540, 540,...

Let's do apply + pd.Series.explode:
pd.DataFrame(df['col'].tolist()).apply(pd.Series.explode).reset_index(drop=True)
0 1 2 3
0 179 Qtr- Oct-20 High 43.8
1 142 Qtr- Oct-20 Low 26.05
2 176 Qtr- Oct-20 Close 33.1
Note: df['col'] is the column in the dataframe which contains list of lists.

__init__() takes from 1 to 6 positional arguments but 11 were given

pls help noob to solve the problem.
i got 2 lists filled with str variables:
crops = ['Кук зер', 'Подсол', 'Пшен оз', 'Сах св', 'Соя', 'Ячм оз', 'Ячм яр']
clusters = ['Восток', 'Восток_2', 'Курск', 'Север', 'Центр', 'Юг',
'Юг_Краснодар', 'Юг_Ставрополь', 'Агросервис']
and then i wanna make a simple panda data set with them:
import pandas as pd
begrow = pd.DataFrame({'Crops': crops},
{clusters[0]: [2, 232, 503, 2442, 3858, '#Н/Д', 4706]},
{clusters[1]: [10, 259, 773, 2620, 3956, '#Н/Д', 4788]},
{clusters[2]: [13, 275, 900, 2754, 3961, '#Н/Д', 4843]},
{clusters[3]: [37, 313, 1446, 3085, 4171, '#Н/Д', 5039]},
{clusters[4]: [90, 322, 1647, 3207, 4285, '#Н/Д', 5090]},
{clusters[5]: [114, 360, 1810, 3293, 4351, '#Н/Д', 5155]},
{clusters[6]: [140, '#Н/Д', 2171, 3546, 4472, 4592, '#Н/Д']},
{clusters[7]: [187, 489, 2341, 3764, 4582, 4695, '#Н/Д']},
{clusters[8]: ['#Н/Д', 230, 490, 2421, 3811, '#Н/Д', 4704]})
print(begrow)
but Spyder returns the following mistake:
TypeError: __init__() takes from 1 to 6 positional arguments but 11 were given
how to fix it?

The error __init__() takes from 1 to 6 positional arguments but 11 were given means that the DataFrame constructor takes a max of 6 different arguments and you fed it 11.
Each set of {} creates a separate dict in Python, which is not what you want. If you remove all the {} except the initial { and the closing }, like so:
begrow = pd.DataFrame({'Crops': crops,
clusters[0]: [2, 232, 503, 2442, 3858, '#Н/Д', 4706],
clusters[1]: [10, 259, 773, 2620, 3956, '#Н/Д', 4788],
clusters[2]: [13, 275, 900, 2754, 3961, '#Н/Д', 4843],
clusters[3]: [37, 313, 1446, 3085, 4171, '#Н/Д', 5039],
clusters[4]: [90, 322, 1647, 3207, 4285, '#Н/Д', 5090],
clusters[5]: [114, 360, 1810, 3293, 4351, '#Н/Д', 5155],
clusters[6]: [140, '#Н/Д', 2171, 3546, 4472, 4592, '#Н/Д'],
clusters[7]: [187, 489, 2341, 3764, 4582, 4695, '#Н/Д'],
clusters[8]: ['#Н/Д', 230, 490, 2421, 3811, '#Н/Д', 4704]})
Then it combines all your data into a single dict and outputs what I think you are looking for:
print(begrow)
runfile('/home/master/.config/spyder-py3/temp.py', wdir='/home/master/.config/spyder-py3')
Crops Восток Восток_2 Курск Север Центр Юг Юг_Краснодар Юг_Ставрополь Агросервис
0 Кук зер 2 10 13 37 90 114 140 187 #Н/Д
1 Подсол 232 259 275 313 322 360 #Н/Д 489 230
2 Пшен оз 503 773 900 1446 1647 1810 2171 2341 490
3 Сах св 2442 2620 2754 3085 3207 3293 3546 3764 2421
4 Соя 3858 3956 3961 4171 4285 4351 4472 4582 3811
5 Ячм оз #Н/Д #Н/Д #Н/Д #Н/Д #Н/Д #Н/Д 4592 4695 #Н/Д
6 Ячм яр 4706 4788 4843 5039 5090 5155 #Н/Д #Н/Д 4704

How to interpolate a 2D curve in Python

I have a set of x & y coordinate which is a curve / shape, I want the smooth the curve / sharp and plot a graph.
I tried different interpolation to smooth the curve / shape, But it still cannot fit my expectation. Using point to draw a smooth curve / shape.
Like the following, using x, y point to get a smooth circle / curve
However, I get something like
circle.jpg
curve.jpg
square.jpg
I also get trouble on spline interpolation, and rbf interpolation.
for cubic_spline_interpolation, I got
ValueError: Error on input data
for univariate_spline_interpolated, I got
ValueError: x must be strictly increasing
for rbf, I got
numpy.linalg.linalg.LinAlgError: Matrix is singular.
I have on idea to fix them and get correct sharp and curve. Many thanks for help.
Edit
For those cannot download the source code and x, y coordinate file, I post the code and x, y coordinate in question.
The following is my code:
#!/usr/bin/env python3
from std_lib import *
import os
import numpy as np
import cv2
from scipy import interpolate
import matplotlib.pyplot as plt
CUR_DIR = os.getcwd()
CIRCLE_FILE = "circle.txt"
CURVE_FILE = "curve.txt"
SQUARE_FILE = "square.txt"
#test
CIRCLE_NAME = "circle"
CURVE_NAME = "curve"
SQUARE_NAME = "square"
SYS_TOKEN_CNT = 2 # x, y
total_pt_cnt = 0 # total no. of points
x_arr = np.array([]) # x position set
y_arr = np.array([]) # y position set
def convert_coord_to_array(file_path):
global total_pt_cnt
global x_arr
global y_arr
if file_path == "":
return FALSE
with open(file_path) as f:
content = f.readlines()
content = [x.strip() for x in content]
total_pt_cnt = len(content)
if (total_pt_cnt <= 0):
return FALSE
##
x_arr = np.empty((0, total_pt_cnt))
y_arr = np.empty((0, total_pt_cnt))
#compare the first and last x
# if ((content[0][0]) > (content[-1])):
# is_reverse = TRUE
for x in content:
token_cnt = get_token_cnt(x, ',')
if (token_cnt != SYS_TOKEN_CNT):
return FALSE
for idx in range(token_cnt):
token_string = get_token_string(x, ',', idx)
token_string = token_string.strip()
if (not token_string.isdigit()):
return FALSE
# save x, y set
if (idx == 0):
x_arr = np.append(x_arr, int(token_string))
else:
y_arr = np.append(y_arr, int(token_string))
return TRUE
def linear_interpolation(fig, axs):
xnew = np.linspace(x_arr.min(), x_arr.max(), len(x_arr))
f = interpolate.interp1d(xnew , y_arr)
axs.plot(xnew, f(xnew))
axs.set_title('linear')
def cubic_interpolation(fig, axs):
xnew = np.linspace(x_arr.min(), x_arr.max(), len(x_arr))
f = interpolate.interp1d(xnew , y_arr, kind='cubic')
axs.plot(xnew, f(xnew))
axs.set_title('cubic')
def cubic_spline_interpolation(fig, axs):
xnew = np.linspace(x_arr.min(), x_arr.max(), len(x_arr))
tck = interpolate.splrep(x_arr, y_arr, s=0) #always fail (ValueError: Error on input data)
ynew = interpolate.splev(xnew, tck, der=0)
axs.plot(xnew, ynew)
axs.set_title('cubic spline')
def parametric_spline_interpolation(fig, axs):
xnew = np.linspace(x_arr.min(), x_arr.max(), len(x_arr))
tck, u = interpolate.splprep([x_arr, y_arr], s=0)
out = interpolate.splev(xnew, tck)
axs.plot(out[0], out[1])
axs.set_title('parametric spline')
def univariate_spline_interpolated(fig, axs):
s = interpolate.InterpolatedUnivariateSpline(x_arr, y_arr)# ValueError: x must be strictly increasing
xnew = np.linspace(x_arr.min(), x_arr.max(), len(x_arr))
ynew = s(xnew)
axs.plot(xnew, ynew)
axs.set_title('univariate spline')
def rbf(fig, axs):
xnew = np.linspace(x_arr.min(), x_arr.max(), len(x_arr))
rbf = interpolate.Rbf(x_arr, y_arr) # numpy.linalg.linalg.LinAlgError: Matrix is singular.
fi = rbf(xnew)
axs.plot(xnew, fi)
axs.set_title('rbf')
def interpolation():
fig, axs = plt.subplots(nrows=4)
axs[0].plot(x_arr, y_arr, 'r-')
axs[0].set_title('org')
cubic_interpolation(fig, axs[1])
# cubic_spline_interpolation(fig, axs[2])
parametric_spline_interpolation(fig, axs[2])
# univariate_spline_interpolated(fig, axs[3])
# rbf(fig, axs[3])
linear_interpolation(fig, axs[3])
plt.show()
#------- main -------
if __name__ == "__main__":
# np.seterr(divide='ignore', invalid='ignore')
file_name = CUR_DIR + "/" + CIRCLE_FILE
convert_coord_to_array(file_name)
#file_name = CUR_DIR + "/" + CURVE_FILE
#convert_coord_to_array(file_name)
#file_name = CUR_DIR + "/" + SQUARE_FILE
#convert_coord_to_array(file_name)
#
interpolation()
circle x, y coordinate
307, 91
308, 90
339, 90
340, 91
348, 91
349, 92
351, 92
352, 93
357, 93
358, 94
361, 94
362, 95
364, 95
365, 96
369, 96
370, 97
374, 97
375, 98
376, 98
377, 99
379, 99
380, 100
382, 100
383, 101
386, 101
387, 102
389, 102
390, 103
392, 103
393, 104
394, 104
395, 105
398, 105
399, 106
400, 106
401, 107
402, 107
403, 108
405, 108
406, 109
407, 109
408, 110
410, 110
411, 111
413, 111
414, 112
415, 112
416, 113
417, 113
418, 114
419, 114
420, 115
421, 115
422, 116
423, 116
425, 118
426, 118
428, 120
429, 120
430, 121
430, 122
431, 122
433, 124
434, 124
435, 125
435, 126
437, 128
437, 129
441, 133
441, 134
442, 135
442, 137
443, 137
444, 138
444, 140
445, 141
445, 142
446, 143
446, 146
447, 147
447, 148
448, 149
448, 153
449, 154
449, 191
448, 192
448, 223
447, 224
447, 240
446, 241
446, 242
445, 243
445, 248
444, 249
444, 253
443, 254
443, 256
442, 257
442, 259
441, 260
441, 263
440, 264
440, 267
439, 268
439, 269
438, 270
438, 272
436, 274
436, 275
435, 276
435, 279
434, 280
434, 281
433, 282
433, 283
431, 285
431, 288
429, 290
429, 291
428, 292
428, 293
426, 295
426, 296
425, 297
425, 298
424, 299
424, 300
423, 301
423, 303
422, 304
422, 305
420, 307
420, 308
419, 309
419, 310
417, 312
417, 313
415, 315
415, 316
414, 317
414, 318
412, 320
411, 320
410, 321
410, 322
409, 323
409, 324
408, 325
407, 325
402, 330
401, 330
401, 331
399, 333
398, 333
395, 336
395, 337
394, 338
393, 338
390, 341
388, 341
387, 342
387, 343
386, 344
384, 344
383, 345
382, 345
380, 347
379, 347
377, 349
376, 349
374, 351
373, 351
373, 352
372, 353
370, 353
369, 354
368, 354
367, 355
366, 355
365, 356
364, 356
363, 357
362, 357
359, 360
358, 360
357, 361
356, 361
355, 362
353, 362
353, 363
352, 364
348, 364
347, 365
314, 365
313, 364
297, 364
296, 363
284, 363
283, 362
280, 362
279, 361
273, 361
272, 360
271, 360
270, 359
265, 359
264, 358
262, 358
261, 357
260, 357
258, 355
257, 355
256, 354
255, 354
252, 351
251, 351
246, 346
245, 346
237, 338
237, 337
235, 335
234, 335
231, 332
231, 331
230, 330
230, 329
222, 321
222, 320
217, 315
217, 314
213, 310
213, 309
210, 306
210, 305
204, 299
204, 298
203, 297
203, 296
199, 292
199, 291
198, 290
198, 289
197, 289
194, 286
194, 285
191, 282
191, 280
187, 276
187, 275
185, 273
185, 271
184, 270
184, 269
183, 268
183, 266
182, 265
182, 264
180, 262
180, 261
179, 260
179, 258
177, 256
177, 254
176, 253
176, 251
175, 250
175, 249
174, 248
174, 246
173, 245
173, 243
171, 241
171, 237
170, 236
170, 232
169, 231
169, 230
168, 229
168, 211
169, 210
169, 205
170, 204
170, 199
171, 198
171, 195
172, 194
172, 193
173, 192
173, 189
174, 188
174, 185
176, 183
176, 180
177, 179
177, 177
178, 176
178, 175
179, 174
179, 173
180, 172
180, 170
182, 168
182, 167
183, 166
183, 165
185, 163
185, 162
186, 161
186, 160
189, 157
189, 156
191, 154
191, 153
192, 152
192, 149
197, 144
197, 143
203, 137
204, 137
207, 134
208, 134
211, 131
213, 131
216, 128
217, 128
218, 127
219, 127
221, 125
222, 125
223, 124
224, 124
225, 123
226, 123
227, 122
228, 122
229, 121
231, 121
233, 119
234, 119
237, 116
239, 116
240, 115
241, 115
242, 114
244, 114
245, 113
246, 113
247, 112
250, 112
251, 111
252, 111
253, 110
256, 110
257, 109
258, 109
259, 108
262, 108
263, 107
266, 107
267, 106
269, 106
272, 103
274, 103
275, 102
276, 102
277, 101
278, 101
279, 100
281, 100
282, 99
283, 99
284, 98
286, 98
287, 97
288, 97
289, 96
290, 96
291, 95
293, 95
295, 93
298, 93
299, 92
302, 92
303, 91
Solved
def linear_interpolateion(self, x, y):
points = np.array([x, y]).T # a (nbre_points x nbre_dim) array
# Linear length along the line:
distance = np.cumsum( np.sqrt(np.sum( np.diff(points, axis=0)**2, axis=1 )) )
distance = np.insert(distance, 0, 0)
alpha = np.linspace(distance.min(), int(distance.max()), len(x))
interpolator = interpolate.interp1d(distance, points, kind='slinear', axis=0)
interpolated_points = interpolator(alpha)
out_x = interpolated_points.T[0]
out_y = interpolated_points.T[1]
return out_x, out_y

Because the interpolation is wanted for generic 2d curve i.e. (x, y)=f(s) where s is the coordinates along the curve, rather than y = f(x), the distance along the line s have to be computed first. Then, the interpolation for each coordinates is performed relatively to s. (for instance, in the circle case y = f(x) have two solutions)
s (or distance in the code here) is calculated as the cumulative sum of the length of each segments between the given points.
import numpy as np
from scipy.interpolate import interp1d
import matplotlib.pyplot as plt
# Define some points:
points = np.array([[0, 1, 8, 2, 2],
[1, 0, 6, 7, 2]]).T # a (nbre_points x nbre_dim) array
# Linear length along the line:
distance = np.cumsum( np.sqrt(np.sum( np.diff(points, axis=0)**2, axis=1 )) )
distance = np.insert(distance, 0, 0)/distance[-1]
# Interpolation for different methods:
interpolations_methods = ['slinear', 'quadratic', 'cubic']
alpha = np.linspace(0, 1, 75)
interpolated_points = {}
for method in interpolations_methods:
interpolator = interp1d(distance, points, kind=method, axis=0)
interpolated_points[method] = interpolator(alpha)
# Graph:
plt.figure(figsize=(7,7))
for method_name, curve in interpolated_points.items():
plt.plot(*curve.T, '-', label=method_name);
plt.plot(*points.T, 'ok', label='original points');
plt.axis('equal'); plt.legend(); plt.xlabel('x'); plt.ylabel('y');
which gives:
Regarding the graphs, it seems you are looking for a smoothing method rather than an interpolation of the points. Here, is a similar approach use to fit a spline separately on each coordinates of the given curve (see Scipy UnivariateSpline):
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import UnivariateSpline
# Define some points:
theta = np.linspace(-3, 2, 40)
points = np.vstack( (np.cos(theta), np.sin(theta)) ).T
# add some noise:
points = points + 0.05*np.random.randn(*points.shape)
# Linear length along the line:
distance = np.cumsum( np.sqrt(np.sum( np.diff(points, axis=0)**2, axis=1 )) )
distance = np.insert(distance, 0, 0)/distance[-1]
# Build a list of the spline function, one for each dimension:
splines = [UnivariateSpline(distance, coords, k=3, s=.2) for coords in points.T]
# Computed the spline for the asked distances:
alpha = np.linspace(0, 1, 75)
points_fitted = np.vstack( spl(alpha) for spl in splines ).T
# Graph:
plt.plot(*points.T, 'ok', label='original points');
plt.plot(*points_fitted.T, '-r', label='fitted spline k=3, s=.2');
plt.axis('equal'); plt.legend(); plt.xlabel('x'); plt.ylabel('y');
which gives:

Plot a data frame

I have a data frame like this:
ReviewDate_month,ProductId,Reviewer
01,185,185
02,155,155
03,130,130
04,111,111
05,110,110
06,98,98
07,101,92
08,71,71
09,73,73
10,76,76
11,105,105
12,189,189
I want to plot it, ReviewDate_Month in X, Product ID and Reviewer in Y ideally. But I will start with 1 line either Product ID or Reviewer.
so i tried:
df_no_monthlycount.plot.line
Got below error msg:
File "C:/Users/user/PycharmProjects/Assign2/Main.py", line 59, in <module>
01 185 185
02 155 155
03 130 130
04 111 111
05 110 110
06 98 98
07 101 92
08 71 71
09 73 73
10 76 76
df_no_monthlycount.plot.line
AttributeError: 'function' object has no attribute 'line'
11 105 105
12 189 189
Process finished with exit code 1
I also tried this:
df_no_monthlycount.plot(x=df_helful_monthlymean['ReviewDate_month'],y=df_helful_monthlymean['ProductId'],style='o')
Error msg like this:
Traceback (most recent call last):
File "C:/Users/user/PycharmProjects/Assign2/Main.py", line 52, in <module>
df_no_monthlycount.plot(x=df_helful_monthlymean['ReviewDate_month'],y=df_helful_monthlymean['ProductId'],style='o')
File "C:\Python34\lib\site-packages\pandas\core\frame.py", line 1797, in __getitem__
return self._getitem_column(key)
File "C:\Python34\lib\site-packages\pandas\core\frame.py", line 1804, in _getitem_column
return self._get_item_cache(key)
File "C:\Python34\lib\site-packages\pandas\core\generic.py", line 1084, in _get_item_cache
values = self._data.get(item)
File "C:\Python34\lib\site-packages\pandas\core\internals.py", line 2851, in get
loc = self.items.get_loc(item)
File "C:\Python34\lib\site-packages\pandas\core\index.py", line 1572, in get_loc
return self._engine.get_loc(_values_from_object(key))
File "pandas\index.pyx", line 134, in pandas.index.IndexEngine.get_loc (pandas\index.c:3838)
File "pandas\index.pyx", line 154, in pandas.index.IndexEngine.get_loc (pandas\index.c:3718)
File "pandas\hashtable.pyx", line 686, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12294)
File "pandas\hashtable.pyx", line 694, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12245)
KeyError: 'ReviewDate_month'

Call the plot as shown below:
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv('data.csv')
print(df)
df.plot(x ='ReviewDate_month',y=['ProductId', 'Reviewer'] ,kind='line')
plt.show()
Will give you:

If you want to plot ReviewDate_Month in X, Product ID and Reviewer in Y, you can do it this way:
df_no_monthlycount.plot(x='ReviewDate_Month', y=['Product ID', 'Reviewer'])