From 0aa7404922328b729242521276e5fa2592dea879 Mon Sep 17 00:00:00 2001 From: ZwX1616 Date: Mon, 8 Nov 2021 15:15:55 -0800 Subject: [PATCH] Add DM model readme (#22801) * add dm model * link to parsing old-commit-hash: eaaccaa3d82f2cff0a4c78014f798edc476886a5 --- models/README.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/models/README.md b/models/README.md index ed79b1dd6c..9f1bfdba02 100644 --- a/models/README.md +++ b/models/README.md @@ -76,3 +76,34 @@ To view the architecture of the ONNX networks, you can use [netron](https://netr [^1]: All probabilities are in logits, so you need to apply sigmoid or softmax functions to get actual probabilities [^2]: These outputs come directly from the vision blocks, they do not have access to temporal state or the desire input + +## Driver Monitoring Model +* .onnx model can be run with onnx runtimes +* .dlc file is a pre-quantized model and only runs on qualcomm DSPs + +### input format +* single image (640 * 320 * 3 in RGB): + * full input size is 6 * 640/2 * 320/2 = 307200 + * represented in YUV420 with 6 channels: + * Channels 0,1,2,3 represent the full-res Y channel and are represented in numpy as Y[::2, ::2], Y[::2, 1::2], Y[1::2, ::2], and Y[1::2, 1::2] + * Channel 4 represents the half-res U channel + * Channel 5 represents the half-res V channel + * normalized, ranging from -1.0 to 1.0 + +### output format +* 39 x float32 outputs ([parsing example](https://github.com/commaai/openpilot/blob/master/selfdrive/modeld/models/dmonitoring.cc#L165)) + * face pose: 12 = 6 + 6 + * face orientation [pitch, yaw, roll] in camera frame: 3 + * face position [dx, dy] relative to image center: 2 + * normalized face size: 1 + * standard deviations for above outputs: 6 + * face visible probability: 1 + * eyes: 20 = (8 + 1) + (8 + 1) + 1 + 1 + * eye position and size, and their standard deviations: 8 + * eye visible probability: 1 + * eye closed probability: 1 + * wearing sunglasses probability: 1 + * poor camera vision probability: 1 + * face partially out-of-frame probability: 1 + * (deprecated) distracted probabilities: 2 + * face covered probability: 1