OE-LLM Quantize & Deploy on RDK S-Series

<!--
SPDX-License-Identifier: Apache-2.0
Copyright (c) 2026 shockley6668
-->

Purpose

Turn a HuggingFace LLM / VLM checkpoint into an OE-LLM .hbm (Hobot BPU model) that runs on an RDK S-series board. The skill owns the host-side PTQ pipeline: environment sanity check, calibration data preparation, Vision / Text compile, cosine verification, and packaging. It stops at the tarball; the on-board runtime (hbdk4_runtime) is out of scope.

When To Use

Activate when the user's task involves any of:

• "quantize / compile / PTQ" a model for RDK S100 / S100P / S600
• "run oellm_build.py" or "leap_llm" against a HuggingFace model
• prepare calibration data (COCO images for Vision, prompt JSON for Text)
• verify cosine similarity between float and quantized outputs
• pack .hbm + tok_embeddings.bin + tokenizer into a board bundle
• upload the artefacts to HuggingFace Hub

Do NOT use for:

• on-board C++ / Python inference wiring (that's a board-side skill)
• ONNX-based tool-chains — OE-LLM ingests PyTorch nn.Module via leap_llm, not ONNX
• training or LoRA fine-tuning

Prerequisites

• Host machine. Ubuntu 22.04, x86_64, ≥ 64 GB RAM (128 GB recommended — Text decode compile peaks at ~100 GB RSS), ≥ 300 GB free disk, optional CUDA GPU for faster Vision calibration.
• OE-LLM SDK 1.0.0. Obtained from D-Robotics — a tarball containing oellm_build/requirements.txt, hbdk4_compiler-*.whl, hbdk4_runtime_aarch64-*.whl, leap_llm-*.whl.
• HuggingFace checkpoint. config.json + model.safetensors + tokenizer files, downloaded ahead of time (some models are gated — request access first).
• leap_llm model adapter. SDK-shipped leap_llm covers common backbones; families the SDK does not know (e.g. Gemma4) need a community adapter dropped into leap_llm/models/<family>/ and registered in leap_llm/apis/model/model_factory.py. See references/model-adapter-checklist.md.
• Board march string. Must match the target — do not guess. nash-e = S100, nash-m = S100P, nash-p = S600. Getting this wrong produces an HBM the board rejects at load.

Instructions

1. Confirm the target board and pick the right march

Ask the user which board they are targeting and select march from references/board-march-map.md. Never proceed past this step until confirmed — this is the single most common misconfiguration and every downstream artefact bakes it in.

2. Set up the environment

# Create the OE-LLM conda env once per host.
conda create -n oellm python=3.10 -y
conda activate oellm

cd /path/to/D-Robotics_LLM_S<version>_SDK/oellm_build
pip install -r requirements.txt
pip install hbdk4_compiler-*.whl
pip install leap_llm-*.whl
# hbdk4_runtime_aarch64-*.whl is board-side, install on the board, not the host.

Install order matters: requirements.txt pins the PyTorch line that hbdk4_compiler and leap_llm are built against.

If the model family is not in stock leap_llm, drop the adapter into $(python -c "import leap_llm, os; print(os.path.dirname(leap_llm.__file__))") — see references/model-adapter-checklist.md.

3. Prepare calibration data

Calibration decides how well PTQ scales fit the deployment distribution. Get this wrong and you will chase a "board outputs garbage" bug that has nothing to do with the compile.

Vision (VLM only): 50 real, diverse images. Do not use synthetic solid-colour blocks — we hit exactly this failure and the fix was 50 COCO val2017 photos.

python3 scripts/download_coco_calib_images.py \
  --output-dir ./calibration_data/images --count 50

Text: 100–200 varied prompts (multi-language, multi-length, multi-topic) in the OE-LLM JSON format.

python3 scripts/gen_text_calib_json.py \
  --output ./calibration_data/text/calibration.json --count 150

Full layout: references/calibration-data-layout.md.

4. Compile Vision (skip for pure LLM)

bash scripts/run_vision_compile.sh \
  --march <nash-e|nash-m|nash-p> \
  --model-name <e.g. gemma4-e2b-vision> \
  --input-model ./<hf_checkpoint_dir> \
  --output-dir ./output/<name>_vision \
  --calib-image-dir ./calibration_data/images \
  --device cuda:0 --vit-core-num 1

The wrapper sets ulimit -s unlimited (mandatory — compile_hbo can segfault otherwise), disables stdout buffering, and tees to a log. Expect ~110 min for a 16-layer ViT on a single core; compile_hbo is single-threaded CPU work, HBDK_JOBS does not speed it up.

Artefacts (output/<name>_vision/):

5. Compile Text

bash scripts/run_text_compile.sh \
  --march <nash-e|nash-m|nash-p> \
  --model-name <e.g. gemma4-e2b-text> \
  --input-model ./<hf_checkpoint_dir> \
  --output-dir ./output/<name>_text \
  --calib-text ./calibration_data/text \
  --chunk-size 256 --cache-len 4096 --core-num 1 \
  [--hbdk-jobs 25] [--enable-swap] [--memlimit-gb 110]

Text decode compile peaks at ~100 GB RSS. On a 128 GB machine, the wrapper accepts --enable-swap (creates 64 GB swap via scripts/setup_swap.sh) and --memlimit-gb (uses prlimit --as) to keep OOM from taking down the host. HBDK_JOBS here is host-side compile parallelism, unrelated to on-board BPU core count.

Artefacts (output/<name>_text/):

tok_embeddings.bin is exported automatically by Gemma4TextApi.__init__ (and equivalent apis for other families) as weight * embed_scale. It stays outside the HBM because a 262k×1536×4B ≈ 1.5 GB embedding table is inefficient to bake into a BPU program — runtime does the lookup and √hidden scaling on CPU.

6. Verify cosine similarity

Run verifier_cli.py against the exported BC to compare quantized vs float output.

# Vision BC vs float
python3 scripts/verify_vision.py \
  --model-name <e.g. gemma4-e2b-vision> \
  --model-dir ./<hf_checkpoint_dir> \
  --quant-bc ./output/<name>_vision/*_vit_ptq.bc \
  --image ./calibration_data/images/<one_image>

# Text BC vs float
python3 scripts/verify_text.py \
  --model-name <e.g. gemma4-e2b-text> \
  --model-dir ./<hf_checkpoint_dir> \
  --quant-bc ./output/<name>_text/*_ptq.prefill.bc \
  --out ./output/<name>_text/verify_text.json

A healthy PTQ run lands in the 0.95–0.99 cosine range per branch. Anything under 0.90 means the calibration set is not representative — go back to step 3 before touching compile flags.

Reference measurement on Gemma4-E2B / S100P (nash-m): Vision 0.9888, Text 0.9540. See references/verification-guide.md for the interpretation table.

7. Pack the board bundle

bash scripts/pack_deploy_bundle.py \
  --vision-hbm ./output/<name>_vision/*_vit_ptq.hbm \
  --text-hbm ./output/<name>_text/*_ptq.hbm \
  --tok-embeddings ./output/<name>_text/tok_embeddings.bin \
  --tokenizer-dir ./<hf_checkpoint_dir> \
  --out ./deploy_bundle

Bundle layout:

deploy_bundle/
├── model/
│   ├── <name>_vit_ptq.hbm
│   ├── <name>_lm_chunk_256_cache_4096_ptq.hbm
│   └── tok_embeddings.bin
└── tokenizer/
    ├── tokenizer.json
    ├── tokenizer_config.json
    ├── chat_template.jinja        # if the model has one
    └── config.json

Then sha256sum deploy_bundle/model/*.hbm deploy_bundle/model/*.bin > deploy_bundle/SHA256SUMS so the board side can verify what it downloaded.

8. (Optional) Publish to HuggingFace Hub

hf auth login                              # once per host
hf upload <user>/<repo> ./deploy_bundle . --repo-type model

If your host is behind a slow link to huggingface.co, exporting HTTPS_PROXY before hf upload gives a 5–10× speedup on typical corporate networks.

Output Contract

After running, produce:

• one .hbm per compiled branch (Vision, Text) with correct --march
• tok_embeddings.bin alongside the Text HBM
• a deploy_bundle/ matching the layout above, with SHA256SUMS
• a short report listing: model name, board target + march, chunk/cache sizes, per-branch cosine, calibration image count and prompt count, total bundle size

Available Scripts

Examples

Case A — LLM on S600 (nash-p), no Vision.

bash scripts/run_text_compile.sh \
  --march nash-p --model-name qwen2-7b-text \
  --input-model ./qwen2-7b --output-dir ./output/qwen2_7b_text \
  --calib-text ./calibration_data/text --chunk-size 512 --cache-len 8192
bash scripts/pack_deploy_bundle.py \
  --text-hbm ./output/qwen2_7b_text/*_ptq.hbm \
  --tok-embeddings ./output/qwen2_7b_text/tok_embeddings.bin \
  --tokenizer-dir ./qwen2-7b --out ./deploy_bundle

Case B — VLM on S100P (nash-m), Vision + Text (the Gemma4-E2B reference).

bash scripts/download_coco_calib_images.py --output-dir ./calibration_data/images --count 50
bash scripts/run_vision_compile.sh --march nash-m --model-name gemma4-e2b-vision \
  --input-model ./gemma4-e2b --output-dir ./output/gemma4_e2b_vision \
  --calib-image-dir ./calibration_data/images --device cuda:0
bash scripts/run_text_compile.sh --march nash-m --model-name gemma4-e2b-text \
  --input-model ./gemma4-e2b --output-dir ./output/gemma4_e2b_text \
  --calib-text ./calibration_data/text --chunk-size 256 --cache-len 4096
bash scripts/pack_deploy_bundle.py \
  --vision-hbm ./output/gemma4_e2b_vision/*_vit_ptq.hbm \
  --text-hbm ./output/gemma4_e2b_text/*_ptq.hbm \
  --tok-embeddings ./output/gemma4_e2b_text/tok_embeddings.bin \
  --tokenizer-dir ./gemma4-e2b --out ./deploy_bundle

Limitations

• Compile-only. Does not run inference on the board or validate end-to-end token output.
• march is user-supplied. There is no runtime check that the string matches the connected board.
• Vision compile assumes a ViT-shaped encoder. Audio branches (Conformer, e.g. Gemma4 audio) are not covered — treat as out of scope until a matching leap_llm adapter is written.
• --enable-swap writes a 64 GB file to /swapfile and calls sudo swapon. Only run on hosts you own.
• Cosine ≥ 0.99 does not guarantee correct end-to-end text; end-to-end verification requires running the bundle on a real board, which lives in the board-side skill.

Troubleshooting

See references/troubleshooting.md for the full pit-log. The top five, distilled:

References

• references/board-march-map.md — S100 / S100P / S600 ↔ nash-e / nash-m / nash-p (verified against SDK docs)
• references/calibration-data-layout.md — directory shape and format for Vision/Text calibration data
• references/model-adapter-checklist.md — what to add under leap_llm/models/<family>/ when the SDK does not ship your architecture
• references/gemma4-adapter-walkthrough.md — file-by-file dissection of the reference Gemma4 adapter (2.7k LOC); use this as the concrete template when porting a new family
• references/vlm-fusion-notes.md — soft-token id substitution, external tok_embeddings.bin, PLE-style per-layer inputs, image-token embedding replacement — for debugging on-board VLM output
• references/verification-guide.md — how to read cosine numbers and what to change if they are too low
• references/troubleshooting.md — full failure catalog from the Gemma4-E2B / S100P bring-up