RuntimeError :The cuda env is set before the forked thread starts报错该怎么解决啊是megengine的问题还是代码问题啊

为使您的问题得到快速解决,建议选择对应标签。
megengine版本:1.10.0
Ubuntu 18.04.5 LTS
gpu: NVIDIA GeForce RTX 2080 Ti
python :3.8

报错代码:
Process Process-5:
Traceback (most recent call last):
File “/opt/miniconda3/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/opt/miniconda3/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/chenrufeng/.local/lib/python3.8/site-packages/megengine/distributed/launcher.py”, line 36, in _run_wrapped
_check_device_initialized(device_type, dev)
File “/home/chenrufeng/.local/lib/python3.8/site-packages/megengine/distributed/helper.py”, line 162, in _check_device_initialized
raise RuntimeError(errmsg)
RuntimeError: The cuda env is set before the forked thread starts. Please do not use any cuda function or variable before forking.
Process Process-4:
Traceback (most recent call last):
File “/opt/miniconda3/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/opt/miniconda3/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/home/chenrufeng/.local/lib/python3.8/site-packages/megengine/distributed/launcher.py”, line 36, in _run_wrapped
_check_device_initialized(device_type, dev)
File “/home/chenrufeng/.local/lib/python3.8/site-packages/megengine/distributed/helper.py”, line 162, in _check_device_initialized
raise RuntimeError(errmsg)
RuntimeError: The cuda env is set before the forked thread starts. Please do not use any cuda function or variable before forking.
Traceback (most recent call last):
File “train_mge.py”, line 260, in
main()
File “train_mge.py”, line 105, in main
train_proc(args)
File “/home/chenrufeng/.local/lib/python3.8/site-packages/megengine/distributed/launcher.py”, line 148, in call
assert (
AssertionError: subprocess 2 exit with code 1

看起来是因为在启动多机函数前造了Tensor,在多机训练时Tensor操作只允许在worker内做。 分布式训练(Distributed Training) 常见问题 — MegEngine 1.10 文档

1赞