add rknn-toolkit2-v0.7.0

Change-Id: I77282cfa9113063f946c2a0b40225180b069f6ee

doc/RKNNToolKit2_OP_Support-v0.7.0.md

@@ -41,8 +41,9 @@ Based on this protocol, the list of Caffe OPs supported by RKNN Toolkit2 Version
 | BatchNorm          |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
 | bn (BatchNorm + Scale) |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br /> according to https://github.com/TimoSaemann/caffe-segnet-cudnn5|
 | BNLL                 ||
-| Concat            |axis: channel (only support channel direction)|
+| Concat               |axis: 1,2,3|
-| Convolution          |**Conv**: <br />channel:<br /> [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />group: 1, channel / N <br /><br />**Convolution depthwise**:  <br />channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]|
+| Convolution          |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />group: 1, channel / N <br /><br />|
+| ConvolutionDepthwise |channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]|
 | Crop                 ||
 | Deconvolution        |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: 2, 4, 8<br />kernels: [1, 8192]<br />pad left/right/top/bottom: [0, 15]|
 | Dropout             ||
@@ -57,6 +58,8 @@ Based on this protocol, the list of Caffe OPs supported by RKNN Toolkit2 Version
 | Proposal           |batch: 1|
 | Reduction          |output dims <= 4|
 | Relu                |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
+| Relu6               |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
+| Reorg               ||
 | Reshape             ||
 | Resize              |bilinear; nearest|
 | Reverse             ||
@@ -67,6 +70,7 @@ Based on this protocol, the list of Caffe OPs supported by RKNN Toolkit2 Version
 | Softmax             ||
 | Split               ||
 | TanH                |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
+| Transpose           ||
 | Upsample            |according to https://github.com/SeanQ88/caffe_upsample and https://github.com/TimoSaemann/caffe-segnet-cudnn5|
 
 ## ONNX OPs supported by RKNN Toolkit2
@@ -80,7 +84,7 @@ The list of ONNX OPs supported by RKNN Toolkit2 Version 0.6.0 is as follows:
 | AveragePool           |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]|
 | BatchNormalization            |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
 | Clip                          |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
-| Concat                        |axis: 1 (only support channel direction)|
+| Concat                        |axis: 1,2,3|
 | Conv                          |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N|
 | ConvTranspose                 |channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: 2, 4, 8<br />kernels: [1, 8192]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N|
 | DepthToSpace                  ||
@@ -128,7 +132,7 @@ The list of Pytorch OPs supported by RKNN Toolkit2 Version 0.6.0 is as follows:
 | aten::add                 |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element |
 | aten::avg_pool2d          |channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]|
 | aten::batch_norm          |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
-| aten::cat                 |axis: 1 (only support channel direction)|
+| aten::cat                 |axis: 1,2,3|
 | aten::dropout             ||
 | aten::flatten             ||
 | aten::leaky_relu          |channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
@@ -158,7 +162,7 @@ The list of TensorFlow OPs supported by RKNN Toolkit2 is as follows:
 | **Operators**                      | **Remarks** |
 | ---------------------------------- | ----------- |
 | AvgPool                |channel: [1, 8192]<br>kernel height/width: [1, 7]<br>stride height/width: [1, 8]<br>pad left/right/top/bottom: [0, 7]|
-| Concat                 |axis: 1 (only support channel direction)|
+| Concat                 |axis: 1,2,3|
 | Conv2D                 |channel: [1, 8192]<br>kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N|
 | DepthToSpace           ||
 | DepthwiseConv2d        |channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]|
@@ -198,7 +202,7 @@ The list of TensorFlow Lite OPs supported by RKNN Toolkit2 is as follows:
 |---| ----------- |
 |ADD|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element |
 |AVERAGE_POOL_2D|channel: [1, 8192]<br />kernel height/width: [1, 7]<br />stride height/width: [1, 8]<br />pad left/right/top/bottom: [0, 7]|
-|CONCATENATION|axis: 1 (only support channel direction)|
+|CONCATENATION|axis: 1,2,3|
 |CONV_2D_TRANSPOSE|channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: 2, 4, 8<br />kernels: [1, 8192]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N|
 |CONV_2D|channel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N|
 |DEPTH_TO_SPACE||
@@ -233,7 +237,7 @@ The list of Darknet OPs supported by RKNN Toolkit2 Version 0.6.0 is as follows:
 |---| ----------- |
 |add|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]<br />support broadcast rule: per-layer/channel/element |
 |batchnormalize|channel: [1, 8192]<br />height: [1, 8192]<br />width: [1, 8176]|
-|concat|axis: 1 (only support channel direction)|
+|concat|axis: 1,2,3|
 |convolutional|hannel: [1, 8192]<br />kernel height/width: [1, 31]<br />stride height/width: [1, 7]<br />kernels: [1, 8184]<br />pad left/right/top/bottom: [0, 15]<br />dilation: [1, 31]<br />group: 1, channel / N|
 |depthwise_convolutional|channel:[1, 8192]<br />kernel height/width: [1, 8]<br />stride height/width: [1, 7]<br />kernels: 1<br />pad left/right/top/bottom: [0, 15]|
 |fullconnect|channel: [1, 8192]|

doc/changelog-v0.7.0.txt

@@ -1,3 +1,14 @@
+2021-4-2
+版本：v0.7.0
+更新内容:
+1. 新功能: 新的量化算法支持(mmse), 添加支持tensorflow的预量化模型导入
+2. 添加了Caffe新OP支持：relu6/ConvolutionDepthwise/Transpose/reorg
+3. 修复一些已知的bug:
+    1) 增加concat的非channel维度，非4维输入的支持
+    2) 修复了第一层是scale的预处理bug
+    3）更新了onnxruntime==1.7.0的版本
+4. 更新了文档，更新了OP支持列表
+
 2021-3-1
 版本：v0.6.0
 更新内容:

doc/requirements-v0.7.0.txt

@@ -1,7 +1,7 @@
 numpy==1.16.6
 onnx==1.7.0
 onnxoptimizer==0.1.0
-onnxruntime==1.5.2
+onnxruntime==1.7.0
 tensorflow==1.14.0
 tensorboard==1.14.0
 protobuf==3.12.0
@@ -14,4 +14,5 @@ scipy==1.2.1
 tqdm==4.27.0
 requests==2.21.0
 tflite==2.3.0
-opencv-python==4.4.0.46
+opencv-python==4.4.0.46
+PuLP==2.4

examples/common_function_demos/load_quantized_model/README.md

@@ -0,0 +1,3 @@
+This demo shows how to load a quantized model.
+Download address of inception_v3_quant_frozen.pb:
+https://storage.googleapis.com/download.tensorflow.org/models/tflite_11_05_08/inception_v3_quant.tgz

examples/common_function_demos/load_quantized_model/test.py

@@ -0,0 +1,164 @@
+import numpy as np
+import cv2
+import os
+import urllib
+import tarfile
+import shutil
+import traceback
+import time
+import sys
+from rknn.api import RKNN
+
+PB_FILE = './inception_v3_quant_frozen.pb'
+RKNN_MODEL_PATH = './inception_v3_quant_frozen.rknn'
+INPUTS = ['input']
+OUTPUTS = ['InceptionV3/Logits/SpatialSqueeze']
+IMG_PATH = './goldfish_299x299.jpg'
+INPUT_SIZE = 299
+
+def show_outputs(outputs):
+    output = outputs[0][0]
+    output_sorted = sorted(output, reverse=True)
+    top5_str = 'inception_v3\n-----TOP 5-----\n'
+    for i in range(5):
+        value = output_sorted[i]
+        index = np.where(output == value)
+        for j in range(len(index)):
+            if (i + j) >= 5:
+                break
+            if value > 0:
+                topi = '{}: {}\n'.format(index[j], value)
+            else:
+                topi = '-1: 0.0\n'
+            top5_str += topi
+    print(top5_str)
+
+
+def readable_speed(speed):
+    speed_bytes = float(speed)
+    speed_kbytes = speed_bytes / 1024
+    if speed_kbytes > 1024:
+        speed_mbytes = speed_kbytes / 1024
+        if speed_mbytes > 1024:
+            speed_gbytes = speed_mbytes / 1024
+            return "{:.2f} GB/s".format(speed_gbytes)
+        else:
+            return "{:.2f} MB/s".format(speed_mbytes)
+    else:
+        return "{:.2f} KB/s".format(speed_kbytes)
+
+
+def show_progress(blocknum, blocksize, totalsize):
+    speed = (blocknum * blocksize) / (time.time() - start_time)
+    speed_str = " Speed: {}".format(readable_speed(speed))
+    recv_size = blocknum * blocksize
+
+    f = sys.stdout
+    progress = (recv_size / totalsize)
+    progress_str = "{:.2f}%".format(progress * 100)
+    n = round(progress * 50)
+    s = ('#' * n).ljust(50, '-')
+    f.write(progress_str.ljust(8, ' ') + '[' + s + ']' + speed_str)
+    f.flush()
+    f.write('\r\n')
+
+if __name__ == '__main__':
+
+    # Create RKNN object
+    rknn = RKNN()
+
+    # If inception_v3_quant_frozen.pb does not exist, download it.
+    # Download address:
+    # https://storage.googleapis.com/download.tensorflow.org/models/tflite_11_05_08/inception_v3_quant.tgz
+    if not os.path.exists(PB_FILE):
+        print('--> Download {}'.format(PB_FILE))
+        url = 'https://storage.googleapis.com/download.tensorflow.org/models/tflite_11_05_08/inception_v3_quant.tgz'
+        download_file = 'inception_v3_quant.tgz'
+        try:
+            start_time = time.time()
+            urllib.request.urlretrieve(url, download_file, show_progress)
+        except:
+            print('Download {} failed.'.format(download_file))
+            print(traceback.format_exc())
+            exit(-1)
+        try:
+            tar = tarfile.open(download_file)
+            target_dir = os.path.splitext(download_file)[0]
+            if os.path.isdir(target_dir):
+                pass
+            else:
+                os.mkdir(target_dir)
+            tar.extractall(target_dir)
+            tar.close()
+        except:
+            print('Extract {} failed.'.format(download_file))
+            exit(-1)
+        pb_file = os.path.join(target_dir, PB_FILE)
+        if os.path.exists(pb_file):
+            shutil.copyfile(pb_file, './inception_v3_quant_frozen.pb')
+            shutil.rmtree(target_dir)
+            os.remove(download_file)
+        print('done')
+    # pre-process config
+    print('--> Config model')
+    rknn.config(reorder_channel=False)
+    print('done')
+
+    # Load tensorflow model
+    print('--> Loading model')
+    ret = rknn.load_tensorflow(tf_pb=PB_FILE,
+                               inputs=INPUTS,
+                               outputs=OUTPUTS,
+                               input_size_list=[[1, INPUT_SIZE, INPUT_SIZE, 3]],
+                               predef_file=None,
+                               mean_values=[[128]],
+                               std_values=[[128]])
+    if ret != 0:
+        print('Load inception_v3_quant_frozen failed!')
+        exit(ret)
+    print('done')
+
+    # Build model
+    print('--> Building model')
+    ret = rknn.build(do_quantization=False)
+    if ret != 0:
+        print('Build inception_v3_quant_frozen.rknn failed!')
+        exit(ret)
+    print('done')
+
+    # Export rknn model
+    print('--> Export RKNN model')
+    ret = rknn.export_rknn(RKNN_MODEL_PATH)
+    if ret != 0:
+        print('Export inception_v3_quant_frozen.rknn failed!')
+        exit(ret)
+    print('done')
+
+    # Set inputs
+    img = cv2.imread(IMG_PATH)
+    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+
+    # init runtime environment
+    print('--> Init runtime environment')
+    ret = rknn.init_runtime()
+    if ret != 0:
+        print('Init runtime environment failed')
+        exit(ret)
+    print('done')
+
+    # Inference
+    print('--> Running model')
+    outputs = rknn.inference(inputs=[img])
+    x = outputs[0]
+    output = np.exp(x)/np.sum(np.exp(x))
+    outputs = [output]
+    show_outputs(outputs)
+    print('done')
+
+    # perf
+    print('--> Begin evaluate model performance')
+    perf_results = rknn.eval_perf(inputs=[img])
+    print('done')
+
+    rknn.release()
+