如何在 Android 裝置上執行深度網路

目錄

• 簡介
• 要求
• 建立一個空的Android Studio專案並新增OpenCV依賴
• 製作一個應用

上一個教程： 使用OpenCV進行Android開發
下一個教程： 在基於Android相機預覽的CV應用中使用OpenCL

另請參見

深度神經網路 (dnn 模組)

原始作者	Dmitry Kurtaev
相容性	OpenCV >= 4.9

簡介

在本教程中，你將瞭解如何使用OpenCV深度學習模組在Android裝置上執行深度學習網路。本教程是為Android Studio 2022.2.1編寫的。

要求

• 從https://developer.android.com/studio下載並安裝Android Studio。
• 從https://github.com/opencv/opencv/releases獲取最新預構建的Android版OpenCV版本並解壓（例如，opencv-4.X.Y-android-sdk.zip，需要最低版本4.9）。
• 從https://github.com/chuanqi305/MobileNet-SSD下載MobileNet物件檢測模型。需要配置檔案MobileNetSSD_deploy.prototxt和模型權重MobileNetSSD_deploy.caffemodel。

建立一個空的Android Studio專案並新增OpenCV依賴

使用使用OpenCV進行Android開發教程來初始化你的專案並新增OpenCV。

製作一個應用

我們的示例將從攝像頭獲取影像，將其轉發到深度網路，並接收一組矩形、類別識別符號和範圍為[0, 1]的置信度值。

• 首先，我們需要新增一個必要的部件來顯示處理後的幀。修改app/src/main/res/layout/activity_main.xml

  <?xml version="1.0" encoding="utf-8"?>
  <FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
      xmlns:app="http://schemas.android.com/apk/res-auto"
      xmlns:tools="http://schemas.android.com/tools"
      android:layout_width="match_parent"
      android:layout_height="match_parent"
      tools:context="org.opencv.samples.opencv_mobilenet.MainActivity">
   
  <org.opencv.android.JavaCameraView
          android:id="@+id/CameraView"
          android:layout_width="match_parent"
          android:layout_height="match_parent"
          android:visibility="visible" />
  </FrameLayout>

• 修改/app/src/main/AndroidManifest.xml以啟用全屏模式，設定正確的螢幕方向並允許使用攝像頭。

  <?xml version="1.0" encoding="utf-8"?>
  <manifest xmlns:android="http://schemas.android.com/apk/res/android">
   
  <application
          android:label="@string/app_name">

  <activity
                    android:exported="true"
                    android:name=".MainActivity"
                    android:screenOrientation="landscape"> <!--螢幕方向-->
  <intent-filter>
  <action android:name="android.intent.action.MAIN" />
  <category android:name="android.intent.category.LAUNCHER" />
  </intent-filter>
  </activity>
  </application>
   
      <!--允許使用攝像頭-->
  <uses-permission android:name="android.permission.CAMERA"/>
  <uses-feature android:name="android.hardware.camera" android:required="false"/>
  <uses-feature android:name="android.hardware.camera.autofocus" android:required="false"/>
  <uses-feature android:name="android.hardware.camera.front" android:required="false"/>
  <uses-feature android:name="android.hardware.camera.front.autofocus" android:required="false"/>
   
  </manifest>

• 替換app/src/main/java/com/example/myapplication/MainActivity.java的內容，並在必要時設定自定義包名

package com.example.myapplication;

import android.content.Context;
import android.content.res.AssetManager;
import android.os.Bundle;
import android.util.Log;
import android.widget.Toast;
 
import org.opencv.android.CameraActivity;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;
import org.opencv.android.OpenCVLoader;
import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.core.MatOfByte;
import org.opencv.core.Point;
import org.opencv.core.Scalar;
import org.opencv.core.Size;
import org.opencv.dnn.Net;
import org.opencv.dnn.Dnn;
import org.opencv.imgproc.Imgproc;
 
import java.io.InputStream;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
 
public class MainActivity extends CameraActivity implements CvCameraViewListener2 {
 
@Override
    public void onResume() {
super.onResume();
        if (mOpenCvCameraView != null)
mOpenCvCameraView.enableView();
    }
 
@Override
    protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
 
        if (OpenCVLoader.initLocal()) {
Log.i(TAG, "OpenCV loaded successfully");
} else {
Log.e(TAG, "OpenCV initialization failed!");
(Toast.makeText(this, "OpenCV initialization failed!", Toast.LENGTH_LONG)).show();
            return;
        }
 
mModelBuffer = loadFileFromResource(R.raw.mobilenet_iter_73000);
mConfigBuffer = loadFileFromResource(R.raw.deploy);
        if (mModelBuffer == null || mConfigBuffer == null) {
Log.e(TAG, "Failed to load model from resources");
} else
Log.i(TAG, "Model files loaded successfully");
 
net = Dnn.readNet("caffe", mModelBuffer, mConfigBuffer);
Log.i(TAG, "Network loaded successfully");
 
setContentView(R.layout.activity_main);
 
        // 設定攝像頭監聽器。
mOpenCvCameraView = (CameraBridgeViewBase)findViewById(R.id.CameraView);
mOpenCvCameraView.setVisibility(CameraBridgeViewBase.VISIBLE);
mOpenCvCameraView.setCvCameraViewListener(this);
    }
 
@Override
    public void onPause()
    {
super.onPause();
        if (mOpenCvCameraView != null)
mOpenCvCameraView.disableView();
    }
 
@Override
    protected List<? extends CameraBridgeViewBase> getCameraViewList() {
        return Collections.singletonList(mOpenCvCameraView);
    }
 
    public void onDestroy() {
super.onDestroy();
        if (mOpenCvCameraView != null)
mOpenCvCameraView.disableView();
 
mModelBuffer.release();
mConfigBuffer.release();
    }
 
    // 載入網路。
    public void onCameraViewStarted(int width, int height) {
    }
 
    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
        final int IN_WIDTH = 300;
        final int IN_HEIGHT = 300;
        final float WH_RATIO = (float)IN_WIDTH / IN_HEIGHT;
        final double IN_SCALE_FACTOR = 0.007843;
        final double MEAN_VAL = 127.5;
        final double THRESHOLD = 0.2;
 
        // 獲取新幀
Log.d(TAG, "handle new frame!");
Mat frame = inputFrame.rgba();
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);
 
        // 將影像轉發到網路。
Mat blob = Dnn.blobFromImage(frame, IN_SCALE_FACTOR,
                new Size(IN_WIDTH, IN_HEIGHT),
                new Scalar(MEAN_VAL, MEAN_VAL, MEAN_VAL), /*swapRB*/false, /*crop*/false);
net.setInput(blob);
Mat detections = net.forward();
 
        int cols = frame.cols();
        int rows = frame.rows();
 
detections = detections.reshape(1, (int)detections.total() / 7);
 
        for (int i = 0; i < detections.rows(); ++i) {
            double confidence = detections.get(i, 2)[0];
            if (confidence > THRESHOLD) {
                int classId = (int)detections.get(i, 1)[0];
 
                int left = (int)(detections.get(i, 3)[0] * cols);
                int top = (int)(detections.get(i, 4)[0] * rows);
                int right = (int)(detections.get(i, 5)[0] * cols);
                int bottom = (int)(detections.get(i, 6)[0] * rows);
 
                // 在檢測到的物件周圍繪製矩形。
Imgproc.rectangle(frame, new Point(left, top), new Point(right, bottom),
                                  new Scalar(0, 255, 0));
                String label = classNames[classId] + ": " + confidence;
                int[] baseLine = new int[1];
                Size labelSize = Imgproc.getTextSize(label, Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, 1, baseLine);
 
                // 繪製標籤背景。
Imgproc.rectangle(frame, new Point(left, top - labelSize.height),
                                  new Point(left + labelSize.width, top + baseLine[0]),
                                  new Scalar(255, 255, 255), Imgproc.FILLED);
                // 寫入類別名稱和置信度。
Imgproc.putText(frame, label, new Point(left, top),
Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, new Scalar(0, 0, 0));
            }
        }
 
        return frame;
    }
 
    public void onCameraViewStopped() {}
 
    private MatOfByte loadFileFromResource(int id) {
       byte[] buffer;
        try {
            // 從應用程式資源載入級聯檔案
InputStream is = getResources().openRawResource(id);
 
            int size = is.available();
buffer = new byte[size];
            int bytesRead = is.read(buffer);
is.close();
} catch (IOException e) {
e.printStackTrace();
Log.e(TAG, "Failed to ONNX model from resources! Exception thrown: " + e);
(Toast.makeText(this, "Failed to ONNX model from resources!", Toast.LENGTH_LONG)).show();
            return null;
        }
 
        return new MatOfByte(buffer);
    }
 
    private static final String TAG = "OpenCV-MobileNet";
    private static final String[] classNames = {"background",
            "aeroplane", "bicycle", "bird", "boat",
            "bottle", "bus", "car", "cat", "chair",
            "cow", "diningtable", "dog", "horse",
            "motorbike", "person", "pottedplant",
            "sheep", "sofa", "train", "tvmonitor"};
 
    private MatOfByte mConfigBuffer;
    private MatOfByte mModelBuffer;
    private Net net;
    private CameraBridgeViewBase mOpenCvCameraView;
}

• 將下載的deploy.prototxt和mobilenet_iter_73000.caffemodel放入app/src/main/res/raw資料夾。OpenCV DNN模型主要設計用於從檔案載入ML和DNN模型。現代Android在沒有額外許可權的情況下不允許這樣做，但提供了Java API來從資源載入位元組。該示例使用了替代的DNN API，該API從記憶體緩衝區而不是檔案初始化模型。以下函式從資源中讀取模型檔案並將其轉換為適用於OpenCV Java API的MatOfBytes物件（C++世界中std::vector<char>的對應物）

    private MatOfByte loadFileFromResource(int id) {
       byte[] buffer;
        try {
            // 從應用程式資源載入級聯檔案
InputStream is = getResources().openRawResource(id);
 
            int size = is.available();
buffer = new byte[size];
            int bytesRead = is.read(buffer);
is.close();
} catch (IOException e) {
e.printStackTrace();
Log.e(TAG, "Failed to ONNX model from resources! Exception thrown: " + e);
(Toast.makeText(this, "Failed to ONNX model from resources!", Toast.LENGTH_LONG)).show();
            return null;
        }
 
        return new MatOfByte(buffer);
    }

然後透過以下幾行程式碼完成網路初始化

mModelBuffer = loadFileFromResource(R.raw.mobilenet_iter_73000);
mConfigBuffer = loadFileFromResource(R.raw.deploy);
        if (mModelBuffer == null || mConfigBuffer == null) {
Log.e(TAG, "Failed to load model from resources");
} else
Log.i(TAG, "Model files loaded successfully");
 
net = Dnn.readNet("caffe", mModelBuffer, mConfigBuffer);
Log.i(TAG, "Network loaded successfully");

另請參閱Android資源文件

• 看看DNN模型輸入是如何準備的以及推理結果是如何解釋的

Mat blob = Dnn.blobFromImage(frame, IN_SCALE_FACTOR,
                new Size(IN_WIDTH, IN_HEIGHT),
                new Scalar(MEAN_VAL, MEAN_VAL, MEAN_VAL), /*swapRB*/false, /*crop*/false);
net.setInput(blob);
Mat detections = net.forward();
 
        int cols = frame.cols();
        int rows = frame.rows();
 
detections = detections.reshape(1, (int)detections.total() / 7);
 
        for (int i = 0; i < detections.rows(); ++i) {
            double confidence = detections.get(i, 2)[0];
            if (confidence > THRESHOLD) {
                int classId = (int)detections.get(i, 1)[0];
 
                int left = (int)(detections.get(i, 3)[0] * cols);
                int top = (int)(detections.get(i, 4)[0] * rows);
                int right = (int)(detections.get(i, 5)[0] * cols);
                int bottom = (int)(detections.get(i, 6)[0] * rows);
 
                // 在檢測到的物件周圍繪製矩形。
Imgproc.rectangle(frame, new Point(left, top), new Point(right, bottom),
                                  new Scalar(0, 255, 0));
                String label = classNames[classId] + ": " + confidence;
                int[] baseLine = new int[1];
                Size labelSize = Imgproc.getTextSize(label, Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, 1, baseLine);
 
                // 繪製標籤背景。
Imgproc.rectangle(frame, new Point(left, top - labelSize.height),
                                  new Point(left + labelSize.width, top + baseLine[0]),
                                  new Scalar(255, 255, 255), Imgproc.FILLED);
                // 寫入類別名稱和置信度。
Imgproc.putText(frame, label, new Point(left, top),
Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, new Scalar(0, 0, 0));
            }
        }

Dnn.blobFromImage將相機幀轉換為神經網路輸入張量。應用了大小調整和統計歸一化。網路輸出張量的每一行都包含一個檢測到的物件的資訊，順序如下：置信度範圍[0, 1]、類別ID、左、上、右、下框座標。所有座標都在[0, 1]範圍內，並在渲染前應按影像大小進行縮放。

• 啟動應用程式，盡情玩樂吧！