About this skill

Problem

When deploying PyTorch/ONNX models to the Horizon RDK S100/S600 BPU, the hard part is not always quantization itself, but precision loss and unstable board-side inference. The Paraformer-large case shows that stride memory alignment can drop Cosine from 0.959 to 0.157; Predictor overfitting can create trailing garbage characters; and skipping warmup can skew first-inference latency.

How It Works

The skill packages a BPU quantization workflow: ONNX export, calibration-data generation, quantization execution, accuracy verification, and post-processing cleanup. Key steps include:
- using the default calibration method with 50-100 representative audio samples;
- using stride-aware copies on board instead of direct memcpy;
- applying correction_factor to Predictor output to remove invalid trailing tokens;
- validating deployment with Cosine, text accuracy, and end-to-end latency.

Boundaries

This is most useful as a reference for RDK speech recognition or similar BPU INT8 quantization work, not as a generic model deployment tool. The material centers on Paraformer-large, RDK S100P/S600, and BPU constraints. Models with many custom operators, unsupported structures, or different frontend feature parameters still need separate checks for operator support, CMVN/LFR alignment, and board-side compatibility.

Screenshots

Use Cases

Deploy Paraformer speech recognition on an RDK S100P board, then complete INT8 quantization and check Cosine plus text accuracy.
Export an existing PyTorch speech model to ONNX, generate BPU calibration data, run quantization, and verify precision.
Debug BPU inference with trailing garbage tokens or low Cosine, then apply correction_factor and stride-aware copy fixes.
Evaluate calibration methods on an RDK S600 board, prefer the default method, and record Cosine and latency differences.

Best For

Algorithm engineers delivering RDK speech recognition who need to quantize Paraformer-large from FP32 to INT8 while preserving text accuracy.
System engineers optimizing Horizon board inference who need to handle stride alignment, warmup, and Predictor post-processing.
Tool engineers maintaining ONNX-to-BPU pipelines who need reusable export, calibration, verification, and cleanup scripts.
Technical leads assessing RDK S100P/S600 feasibility who need to compare default-calibration accuracy and speedup benefits.