0%| | 0/4370 [00:00<?, ?it/s]03 17:05:42[mgb] WRN indexing_multi_axis_vec(concat[4117])[4131]{IndexingMultiAxisVec}: no vector indexer; consider using Subtensor family for better performance; you can set MGB_THROW_ON_SCALAR_IDX to throw an exception to help tracking the related operator
03 17:05:42[mgb] WRN indexing_multi_axis_vec(concat[4117])[4150]{IndexingMultiAxisVec}: no vector indexer; consider using Subtensor family for better performance; you can set MGB_THROW_ON_SCALAR_IDX to throw an exception to help tracking the related operator
03 17:05:42[mgb] WRN indexing_multi_axis_vec(concat[4117])[4169]{IndexingMultiAxisVec}: no vector indexer; consider using Subtensor family for better performance; you can set MGB_THROW_ON_SCALAR_IDX to throw an exception to help tracking the related operator
0%| | 1/4370 [26:25<1924:12:08, 1585.52s/it]03 17:05:43[mgb] WRN indexing_multi_axis_vec(concat[24370])[24520]{IndexingMultiAxisVec}: no vector indexer; consider using Subtensor family for better performance; you can set MGB_THROW_ON_SCALAR_IDX to throw an exception to help tracking the related operator
03 17:05:43[mgb] WRN indexing_multi_axis_vec(concat[24370])[24513]{IndexingMultiAxisVec}: no vector indexer; consider using Subtensor family for better performance; you can set MGB_THROW_ON_SCALAR_IDX to throw an exception to help tracking the related operator
03 17:05:43[mgb] WRN indexing_multi_axis_vec(concat[24370])[24506]{IndexingMultiAxisVec}: no vector indexer; consider using Subtensor family for better performance; you can set MGB_THROW_ON_SCALAR_IDX to throw an exception to help tracking the related operator
03 17:05:43[mgb] WRN indexing_multi_axis_vec(concat[24370])[24499]{IndexingMultiAxisVec}: no vector indexer; consider using Subtensor family for better performance; you can set MGB_THROW_ON_SCALAR_IDX to throw an exception to help tracking the related operator
03 17:05:43[mgb] ERR error occurred in computing sequence; synchronizing all comp nodes and releasing vars now …
Process Process-1:
Traceback (most recent call last):
File “/home/a230/.conda/envs/megengine/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/home/a230/.conda/envs/megengine/lib/python3.6/multiprocessing/process.py”, line 93, in run
self._target(*self._args, **self._kwargs)
File “test.py”, line 69, in inference
pred_boxes = val_func().numpy()
File “/home/a230/.conda/envs/megengine/lib/python3.6/site-packages/megengine/jit/init.py”, line 386, in call
self._compiled_func()
File “/home/a230/.conda/envs/megengine/lib/python3.6/site-packages/megengine/_internal/mgb.py”, line 1109, in call
self._execute()
File “/home/a230/.conda/envs/megengine/lib/python3.6/site-packages/megengine/_internal/mgb.py”, line 993, in _execute
return _mgb.AsyncExec__execute(self)
megengine._internal.exc.MegBrainError: MegBrain core throws exception: mgb::cg::GraphError
var {id:24530, shape:{0,256,7,7}, Float32, owner:indexing_multi_axis_vec(concat[24527])[24529]{IndexingMultiAxisVec}, name:indexing_multi_axis_vec(concat[24527])[24529], slot:0, gpu0:0, d, 8, 1} has empty memplan, but allowed=1 receiver=VarReceiverInfo(nr_direct_comp_req=0 dev_value=1, host_value=0, shape=0, allow_empty_value=0)
- bt:/home/a230/.conda/envs/megengine/lib/python3.6/site-packages/megengine/_internal/_mgb.cpython-36m-x86_64-linux-gnu.so{1c6249d,1c65bc2,1c1c191,1c1b777}
| Associated operator: id=24529 name=indexing_multi_axis_vec(concat[24527])[24529] type=mgb::opr::IndexingMultiAxisVec
| input variables:
| 0: {id:24528, shape:{4,256,7,7}, Float32, owner:concat(roi_align[24502]:o0,roi_align[24509]:o0,roi_align[24516]:o0,…)[24527]{Concat}, name:concat(roi_align[24502]:o0,roi_align[24509]:o0,roi_align[24516]:o0,…)[24527], slot:0, gpu0:0, d, 8, 8}
| 1: {id:24464, shape:{0}, Int32, owner:cond_take(indexing_multi_axis_vec[24459],indexing_multi_axis_vec[24459])[24462]{CondTake}, name:cond_take(indexing_multi_axis_vec[24459],indexing_multi_axis_vec[24459])[24462]:o1, slot:1, gpu0:0, d, 1, 1}
| output variables:
| 0: {id:24530, shape:{0,256,7,7}, Float32, owner:indexing_multi_axis_vec(concat[24527])[24529]{IndexingMultiAxisVec}, name:indexing_multi_axis_vec(concat[24527])[24529], slot:0, gpu0:0, d, 8, 1}
| 1: {id:24531, shape:{0}, Byte, owner:indexing_multi_axis_vec(concat[24527])[24529]{IndexingMultiAxisVec}, name:indexing_multi_axis_vec(concat[24527])[24529]:workspace, slot:1, gpu0:0, d, 8, 1}
|
| Unoptimized equivalent of associated operator: id=4208 name=indexing_multi_axis_vec(concat[4197])[4208] type=mgb::opr::IndexingMultiAxisVec
| input variables:
| 0: {id:4198, shape:{1004,256,7,7}, Float32, owner:concat(roi_align[4134]:o0,roi_align[4153]:o0,roi_align[4172]:o0,…)[4197]{Concat}, name:concat(roi_align[4134]:o0,roi_align[4153]:o0,roi_align[4172]:o0,…)[4197], slot:0, gpu0:0, d, 8, 8}
| 1: {id:4207, shape:{1000}, Int32, owner:set_grad(cond_take[4202]:o1)[4206]{SetGrad}, name:set_grad(cond_take[4202]:o1)[4206], slot:0, gpu0:0, d, 8, 8}
| output variables:
| 0: {id:4209, shape:{1000,256,7,7}, Float32, owner:indexing_multi_axis_vec(concat[4197])[4208]{IndexingMultiAxisVec}, name:indexing_multi_axis_vec(concat[4197])[4208], slot:0, gpu0:0, d, 8, 1}
| 1: {id:4210, shape:{4000}, Byte, owner:indexing_multi_axis_vec(concat[4197])[4208]{IndexingMultiAxisVec}, name:indexing_multi_axis_vec(concat[4197])[4208]:workspace, slot:1, gpu0:0, d, 8, 1}
