ShufflenetV2花卉识别模型量化过程中，将模型转换为量化模型，并执行 Dump 时报错

#1

开发环境：

MegEngine 1.0.0
Python 3.7
Jupyter Notebook
按照https://www.bilibili.com/video/BV1Zf4y1Q7aZ?p=2中第六讲进行部署进阶：推理端优化时，执行到QAT量化第四步：将模型转换为量化模型，并执行 Dump 用于后续模型部署时报错。执行代码如下：

!python3 inference.py -a shufflenet_v2_x0_5 --checkpoint /home/megstudio/workspace/megengine_quantize_test/Models/official/quantization/result/model/shufflenet_v2_x0_5.qat/checkpoint-final.pkl --mode quantized --dump

报错信息如下：

Traceback (most recent call last):
  File "inference.py", line 111, in <module>
    main()
  File "inference.py", line 87, in main
    probs = infer_func(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/jit/tracing.py", line 507, in __call__
    outputs = self.__wrapped__(*args, **kwargs)
  File "inference.py", line 79, in infer_func
    logits = model(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 103, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/megengine_quantize_test/Models/official/quantization/models/shufflenet_v2.py", line 192, in forward
    x = self.features(x)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 103, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/sequential.py", line 97, in forward
    inp = layer(inp)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 103, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/megengine_quantize_test/Models/official/quantization/models/shufflenet_v2.py", line 108, in forward
    return self.concat((self.branch_proj(x_proj), self.branch_main(x)), 1)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/module.py", line 23, in __call__
    return super().__call__(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 103, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/concat.py", line 27, in forward
    return F.concat(new_inps, axis)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/functional/tensor.py", line 260, in concat
    if len(inps) == 1:
TypeError: object of type 'generator' has no len()
#2

根据log不确定是megengine的问题还是infer脚本的问题。
我试了下最新的megengine==1.2.0版本和最新的model hub https://github.com/MegEngine/Models/tree/master/official/quantization
可以正常dump qat的shfflenet模型。

git clone https://github.com/MegEngine/Models.git
cd Models/official/quantization/
wget https://data.megengine.org.cn/models/weights/shufflenet_v1_x1_5_g3_qat_70806.pkl
python3 inference.py -a shufflenet_v1_x1_5_g3 --checkpoint shufflenet_v1_x1_5_g3_qat_70806.pkl --mode quantized --dump
#3

好的，谢谢老师，我再尝试一下新的版本

#4

老师，您好！
我刚刚在运行这部分的时候也出现了这个问题：

Traceback (most recent call last):
  File "inference.py", line 221, in <module>
    main()
  File "inference.py", line 197, in main
    probs = infer_func(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/jit/tracing.py", line 644, in __call__
    outputs = self.__wrapped__(*args, **kwargs)
  File "inference.py", line 189, in infer_func
    logits = model(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/Models/official/quantization/models/shufflenet_v2.py", line 204, in forward
    x = self.features(x)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/sequential.py", line 96, in forward
    inp = layer(inp)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/Models/official/quantization/models/shufflenet_v2.py", line 115, in forward
    return self.concat((self.branch_proj(x_proj), self.branch_main(x)), 1)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/module.py", line 23, in __call__
    return super().__call__(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/concat.py", line 27, in forward
    return F.concat(new_inps, axis)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/functional/tensor.py", line 257, in concat
    if len(inps) == 1:
TypeError: object of type 'generator' has no len()

开发环境：
MegEngine 1.2.0
Python 3.7
Jupyter Notebook
而且已经用了最新版本代码
老师您看看什么情况

#5

确认下model代码是最新的，https://github.com/MegEngine/Models/blob/master/official/quantization/inference.py 没有221行和

  File "inference.py", line 221, in <module>
    main()
  File "inference.py", line 197, in main
    probs = infer_func(processed_img)

对不上

#6

老师以下是运行代码(用的是新代码)：

# -*- coding: utf-8 -*-
# MegEngine is Licensed under the Apache License, Version 2.0 (the "License")
#
# Copyright (c) 2014-2021 Megvii Inc. All rights reserved.
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT ARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
"""Inference a pretrained model with given inputs"""
import argparse
import json
import os

# pylint: disable=import-error
import models

import cv2
import numpy as np

import megengine as mge
import megengine.data.transform as T
import megengine.functional as F
import megengine.quantization as Q
from megengine.jit import trace
from megengine.quantization.quantize import quantize, quantize_qat

logger = mge.get_logger(__name__)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("-a", "--arch", default="resnet18", type=str)
    parser.add_argument("-c", "--checkpoint", default=None, type=str)
    parser.add_argument("-i", "--image", default=None, type=str)

    parser.add_argument(
        "-m",
        "--mode",
        default="quantized",
        type=str,
        choices=["normal", "qat", "quantized"],
        help="Quantization Mode\n"
        "normal: no quantization, using float32\n"
        "qat: quantization aware training, simulate int8\n"
        "quantized: convert mode to int8 quantized, inference only",
    )
    parser.add_argument("--dump", action="store_true", help="Dump quantized model")
    args = parser.parse_args()

    model = models.__dict__[args.arch]()

    if args.mode != "normal":
        quantize_qat(model, qconfig=Q.ema_fakequant_qconfig)

    if args.mode == "quantized":
        quantize(model)

    if args.checkpoint:
        logger.info("Load pretrained weights from %s", args.checkpoint)
        ckpt = mge.load(args.checkpoint)
        ckpt = ckpt["state_dict"] if "state_dict" in ckpt else ckpt
        model.load_state_dict(ckpt, strict=False)

    rpath = os.path.realpath(__file__ + "/../../")
    if args.image is None:
        path = rpath + "/assets/cat.jpg"
    else:
        path = args.image
    image = cv2.imread(path, cv2.IMREAD_COLOR)

    transform = T.Compose(
        [T.Resize(160), T.CenterCrop(128), T.Normalize(mean=128), T.ToMode("CHW")]
    )

    @trace(symbolic=True, capture_as_const=True)
    def infer_func(processed_img):
        model.eval()
        logits = model(processed_img)
        probs = F.softmax(logits)
        return probs

    processed_img = transform.apply(image)[np.newaxis, :]

    processed_img = mge.tensor(processed_img, dtype="float32")

    probs = infer_func(processed_img)

    top_probs, classes = F.topk(probs, k=5, descending=True)

    if args.dump:
        output_file = ".".join([args.arch, args.mode, "megengine"])
        logger.info("Dump to {}".format(output_file))
        infer_func.dump(output_file, arg_names=["data"])
        mge.save(model.state_dict(), output_file.replace("megengine", "pkl"))

    with open(rpath + "/assets/imagenet_class_info.json") as fp:
        imagenet_class_index = json.load(fp)

    for rank, (prob, classid) in enumerate(
        zip(top_probs.numpy().reshape(-1), classes.numpy().reshape(-1))
    ):
        print(
            "{}: class = {:20s} with probability = {:4.1f} %".format(
                rank, imagenet_class_index[str(classid)][1], 100 * prob
            )
        )


if __name__ == "__main__":
    main()

运行结果：

Traceback (most recent call last):
  File "inference.py", line 110, in <module>
    main()
  File "inference.py", line 86, in main
    probs = infer_func(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/jit/tracing.py", line 644, in __call__
    outputs = self.__wrapped__(*args, **kwargs)
  File "inference.py", line 78, in infer_func
    logits = model(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/Models/official/quantization/models/shufflenet_v2.py", line 204, in forward
    x = self.features(x)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/sequential.py", line 96, in forward
    inp = layer(inp)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/Models/official/quantization/models/shufflenet_v2.py", line 115, in forward
    return self.concat((self.branch_proj(x_proj), self.branch_main(x)), 1)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/module.py", line 23, in __call__
    return super().__call__(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/concat.py", line 27, in forward
    return F.concat(new_inps, axis)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/functional/tensor.py", line 257, in concat
    if len(inps) == 1:
TypeError: object of type 'generator' has no len()
#7

整个model目录都是最新的吗(git clone一个新的或者git pull一下)，不能只更新inference.py
再确认下megengine.__version__是1.2.0

#8

以下是我的项目开发环境截图（一直都是1.2.0）：

image
image565×576 14.1 KB

也对代码进行了更新，还是一样的错误：

Traceback (most recent call last):
  File "inference.py", line 110, in <module>
    main()
  File "inference.py", line 86, in main
    probs = infer_func(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/jit/tracing.py", line 644, in __call__
    outputs = self.__wrapped__(*args, **kwargs)
  File "inference.py", line 78, in infer_func
    logits = model(processed_img)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/Models/Models-master/official/quantization/models/shufflenet_v2.py", line 416, in forward
    x = self.features(x)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/sequential.py", line 96, in forward
    inp = layer(inp)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/workspace/Models/Models-master/official/quantization/models/shufflenet_v2.py", line 327, in forward
    return self.concat((self.branch_proj(x_proj), self.branch_main(x)), 1)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/module.py", line 23, in __call__
    return super().__call__(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/module.py", line 113, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/module/quantized/concat.py", line 27, in forward
    return F.concat(new_inps, axis)
  File "/home/megstudio/.miniconda/envs/xuan/lib/python3.7/site-packages/megengine/functional/tensor.py", line 257, in concat
    if len(inps) == 1:
TypeError: object of type 'generator' has no len()
​
#9


这是我的复现，你可以对一下
image

#10

好的，谢谢老师！

#11

老师好，我执行dump操作时也遇到了一模一样的报错，按照您的方法dump了提供的qat好的模型，我们自己训练好的模型还是没办法dump，可能是哪里出的问题呀？