I am trying to use XGBoost library using .train function and DMatrix but I am a little stuck because of an error :
Traceback (most recent call last):
File "", line 1, in
runfile('E:/CrossValidation.py', wdir='E:/')
File
"C:\Users\users\Anaconda3\envs\Lightgbm\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 705, in runfile
execfile(filename, namespace)
File
"C:\Users\users\Anaconda3\envs\Lightgbm\lib\site-packages\spyder\utils\site\sitecustomize.py",
line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "E:/CrossValidation.py", line 218, in
mainXGB()
File "E:/CrossValidation.py", line 214, in mainXGB
crossval_preds, val_preds = cv.train(X_data=X_train.values, y_data=y_train.values, X_test=X_val.values, params=xgb_params)
File "E:/CrossValidation.py", line 136, in train
early_stopping_rounds=early_stopping_rounds)
File
"C:\Users\users\Anaconda3\envs\Lightgbm\lib\site-packages\xgboost\training.py",
line 204, in train
xgb_model=xgb_model, callbacks=callbacks)
File
"C:\Users\users\Anaconda3\envs\Lightgbm\lib\site-packages\xgboost\training.py",
line 32, in _train_internal
bst = Booster(params, [dtrain] + [d[0] for d in evals])
File
"C:\Users\users\Anaconda3\envs\Lightgbm\lib\site-packages\xgboost\training.py",
line 32, in
bst = Booster(params, [dtrain] + [d[0] for d in evals])
TypeError: 'DMatrix' object does not support indexing
Here my piece of code :
dtrain = xgb.DMatrix(X_data[train_idx], label=np.log1p(y_data[train_idx])) # datas.slice(train_idx)
dtest = xgb.DMatrix(X_data[val_idx], label=np.log1p(y_data[val_idx]))
print('data created for xgboost')
model = self.model_base.train(params=params, dtrain=dtrain, num_boost_round=number_iteration, evals=[dtest], early_stopping_rounds=early_stopping_rounds)
Does anyone know how to solve the problem ?
The problem is with the evals argument. A list of tuples is expected, so change evals=[dtest] to evals=[(dtest, "Test")].
Related
Error message
Traceback (most recent call last):
File "tools/train.py", line 244, in <module>
main()
File "tools/train.py", line 233, in main
train_detector(
File "/home/christ/dev/repos/railsight/mmdetection-2.25.3/mmdet/apis/train.py", line 244, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 130, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 47, in train
for i, data_batch in enumerate(self.data_loader):
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 368, in __iter__
return self._get_iterator()
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 314, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 965, in __init__
self._reset(loader, first_iter=True)
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 996, in _reset
self._try_put_index()
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1230, in _try_put_index
index = self._next_index()
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in _next_index
return next(self._sampler_iter) # may raise StopIteration
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 226, in __iter__
for idx in self.sampler:
File "/home/christ/dev/repos/railsight/mmdetection-2.25.3/mmdet/datasets/samplers/group_sampler.py", line 36, in __iter__
indices = np.concatenate(indices)
File "<__array_function__ internals>", line 180, in concatenate
ValueError: need at least one array to concatenate
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
pid, sts = os.waitpid(self.pid, flag)
File "/home/christ/miniconda3/envs/mmdetection/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
_error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 35413) is killed by signal: Terminated.
This is the error message I am faced with when trying to train using SSD with MMDetection. I have checked through my dataset and it works with faster_rcnn so I am not understanding why I am having such an issue with the SSD training. Any advice would be great!
_base_ = '../ssd/ssd300_coco.py'
dataset_type = 'CocoDataset'
classes = ('pantograph',)
data = dict(
train=dict(
img_prefix='configs/pantograph/train/',
classes=classes,
ann_file='configs/pantograph/train/SSDTrain.json',
dataset=dict(
ann_file='configs/pantograph/train/SSDTrain.json',
img_prefix='configs/pantograph/train/')),
val=dict(
img_prefix='configs/pantograph/val/',
classes=classes,
ann_file='configs/pantograph/val/SSDVal.json'),
test=dict(
img_prefix='configs/pantograph/test/',
classes=classes,
ann_file='configs/pantograph/test/SSDTest.json'))
Above is my custom SSD config that I tried to run. I have double checked the file locations and made sure they are all correct.
mmdet has a very good community, you can try to look at this link to solve your problem. Shortly, you should change your classes = ('pantograph',) as the following codes
cfg.metainfo = {
'classes': ('balloon', ),
'palette': [
(220, 20, 60),
]
}
I am trying to train a YOLO model on a custom dataset using YOLOv8 Ultralytics, using M1 MacBook Pro MPS acceleration which is supported by PyTorch but I am getting the following error while executing the train command.
Command:
!yolo task=detect mode=train model=yolov8s.pt data={HOME}/datasets/data.yaml epochs=25 device=mps
Error Message:
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/cli.py", line 56, in cli
func(cfg)
File "/opt/homebrew/lib/python3.10/site-packages/hydra/main.py", line 79, in decorated_main
return task_function(cfg_passthrough)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 207, in train
model.train(**cfg)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/model.py", line 193, in train
self.trainer.train()
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
self._do_train(int(os.getenv("RANK", -1)), world_size)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 301, in _do_train
self.loss, self.loss_items = self.criterion(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 77, in criterion
return self.compute_loss(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 168, in __call__
targets = self.preprocess(targets.to(self.device), batch_size, scale_tensor=imgsz[[1, 0, 1, 0]])
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Is there any way to get around this issue and use MPS for training?
I have tried looking for this issue on PyTorch's GitHub conversation, but got no simple solution.
Edit:
I have set HYDRA_FULL_ERROR to 1, and this is the updated error traceback:
Error executing job with overrides: ['task=detect', 'mode=train', 'model=yolov8s.pt', 'data=/Users/aditya_nemiwal/Documents/C++/Python/WiDS_Week3_3/datasets/data.yaml', 'epochs=25', 'device=mps']
Traceback (most recent call last):
File "/opt/homebrew/bin/yolo", line 8, in <module>
sys.exit(cli())
File "/opt/homebrew/lib/python3.10/site-packages/hydra/main.py", line 90, in decorated_main
_run_hydra(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
raise ex
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
return func()
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
File "/opt/homebrew/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/opt/homebrew/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/opt/homebrew/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/cli.py", line 56, in cli
func(cfg)
File "/opt/homebrew/lib/python3.10/site-packages/hydra/main.py", line 79, in decorated_main
return task_function(cfg_passthrough)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 207, in train
model.train(**cfg)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/model.py", line 193, in train
self.trainer.train()
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
self._do_train(int(os.getenv("RANK", -1)), world_size)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/engine/trainer.py", line 301, in _do_train
self.loss, self.loss_items = self.criterion(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 77, in criterion
return self.compute_loss(preds, batch)
File "/opt/homebrew/lib/python3.10/site-packages/ultralytics/yolo/v8/detect/train.py", line 168, in __call__
targets = self.preprocess(targets.to(self.device), batch_size, scale_tensor=imgsz[[1, 0, 1, 0]])
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
I am getting the above error.
I Already installed opencv-contrib module but still not working out
def dictionary(descriptors, N):
em = cv2.EM(N)
em.train(descriptors)
Traceback (most recent call last):
File "", line 1, in
runfile('C:/Users/Ankuran Das/Desktop/Pyhton/Case/fishvec.py', wdir='C:/Users/Ankuran Das/Desktop/Pyhton/Case')
File "D:\Anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "D:\Anaconda\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/Ankuran Das/Desktop/Pyhton/Case/fishvec.py", line 141, in
gmm = generate_gmm(working_folder,args.number)
File "C:/Users/Ankuran Das/Desktop/Pyhton/Case/fishvec.py", line 87, in generate_gmm
means, covs, weights = dictionary(words, N)
File "C:/Users/Ankuran Das/Desktop/Pyhton/Case/fishvec.py", line 16, in dictionary
em = cv2.EM.train(N)
AttributeError: module 'cv2.cv2' has no attribute 'EM'
Try the following lines of codes:
em = cv2.ml.EM_create()
em.setClustersNumber(N)
em.trainEM(descriptors)
What I want to do is actually pretty simple. I want to run a OLS regression by using data from dask. It is fine, if I just install data into dask dataframe or I use pandas (but I get memory error when I use Pandas).
# import modules
import statsmodels.formula.api as smf
import dask.dataframe as dd
df = dd.read_csv(input_file)
mod = smf.ols(formula='lnq ~ lnp + lnp:C(product_code)+ C(product_code)', data=df)
results = mod.fit(cov_type = 'cluster', cov_kwds={'groups': df['product_code']})
print(results.summary())
However, I get the following error:
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
execfile(filename, namespace)
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "H:/yoox_3.py", line 11, in <module>
mod = smf.ols(formula='lnq ~ lnp + lnp:C(product_code)+ C(product_code)', data=df)
File "C:\ProgramData\Anaconda3\lib\site-packages\statsmodels\base\model.py", line 155, in from_formula
missing=missing)
File "C:\ProgramData\Anaconda3\lib\site-packages\statsmodels\formula\formulatools.py", line 68, in handle_formula_data
NA_action=na_action)
File "C:\ProgramData\Anaconda3\lib\site-packages\patsy\highlevel.py", line 310, in dmatrices
NA_action, return_type)
File "C:\ProgramData\Anaconda3\lib\site-packages\patsy\highlevel.py", line 165, in _do_highlevel_design
NA_action)
File "C:\ProgramData\Anaconda3\lib\site-packages\patsy\highlevel.py", line 70, in _try_incr_builders
NA_action)
File "C:\ProgramData\Anaconda3\lib\site-packages\patsy\build.py", line 696, in design_matrix_builders
NA_action)
File "C:\ProgramData\Anaconda3\lib\site-packages\patsy\build.py", line 448, in _examine_factor_types
done = cat_sniffers[factor].sniff(value)
File "C:\ProgramData\Anaconda3\lib\site-packages\patsy\categorical.py", line 204, in sniff
for value in data:
File "C:\ProgramData\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 1703, in __getitem__
raise NotImplementedError()
NotImplementedError
Any ideas?
I don't believe that statsmodels offers support for dask dataframes
I am trying to run some simle example with Pyomo + glpk Solver (Anaconda2 64bit Spyder):
from pyomo.environ import *
model = ConcreteModel()
model.x_1 = Var(within=NonNegativeReals)
model.x_2 = Var(within=NonNegativeReals)
model.obj = Objective(expr=model.x_1 + 2*model.x_2)
model.con1 = Constraint(expr=3*model.x_1 + 4*model.x_2 >= 1)
model.con2 = Constraint(expr=2*model.x_1 + 5*model.x_2 >= 2)
opt = SolverFactory("glpk")
instance = model.create()
#results = opt.solve(instance)
#results.write()
But i get the following error message:
invalid literal for int() with base 10: 'c'
Traceback (most recent call last):
File "<ipython-input-5-e074641da66d>", line 1, in <module>
runfile('D:/..../Exampe.py', wdir='D:.../exercises/pyomo')
File "C:\...\Continuum\Anaconda21\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile
execfile(filename, namespace)
File "C:\....\Continuum\Anaconda21\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)
File "D:/...pyomo/Exampe.py", line 34, in <module>
results = opt.solve(instance)
File "C:\....\Continuum\Anaconda21\lib\site-packages\pyomo\opt\base\solvers.py", line 580, in solve
result = self._postsolve()
File "C:\...Continuum\Anaconda21\lib\site-packages\pyomo\opt\solver\shellcmd.py", line 267, in _postsolve
results = self.process_output(self._rc)
File "C:\...\Continuum\Anaconda21\lib\site-packages\pyomo\opt\solver\shellcmd.py", line 329, in process_output
self.process_soln_file(results)
File "C:\....\Continuum\Anaconda21\lib\site-packages\pyomo\solvers\plugins\solvers\GLPK.py", line 454, in process_soln_file
raise ValueError(msg)
ValueError: Error parsing solution data file, line 1
I downloaded glpk from http://winglpk.sourceforge.net/ --> unziped + added parth to the environmental variable "path".
Hope someone can help me - thank you!
This is a known problem with GLPK 4.60 (glpsol changed the format of their output which broke Pyomo 4.3's parser). You can either download an older release of GLPK, or upgrade Pyomo to 4.4.1 (which contains an updated parser).