TensorRT Model Connect: Build a Qwen Bundle, Then Keep Inference Native

NVIDIA’s TensorRT Model Connect is a collection of inspectable reference implementations that turns a supported open checkpoint into TensorRT engines and a bundle, then exposes task-oriented native C++ APIs.[1][2] The public quick start uses Qwen/Qwen3-0.6B: build a bundle with trtmc, then run the fixed France prompt.[3]

That is a useful boundary, not a production guarantee. The project calls itself experimental; its APIs, scope, and direction may change.[2] The official support table applies to an exact model ID, profile, precision, configuration, and evidence record. It explicitly says that same-family fine-tunes are best-effort, not verified support.[5]

What was and was not verified here

The source commit checked for this guide is f23a07e54521725d33d31c8d1f51d3a030d46a66 (current main HEAD on 2026-09-01). Its LICENSE is Apache-2.0.[2][6] The selected public checkpoint is Qwen/Qwen3-0.6B, which appears in NVIDIA’s official support matrix with an FP16 profile; NVIDIA’s quick start specifies a BF16 first build profile for this path.[3][5]

The local preflight did not reach installation: nvidia-smi was not installed, trtmc was absent, and Docker’s daemon socket returned permission denied. That is a hardware/access block, not a failed TensorRT Model Connect result. No model was downloaded, no container was built, and no bundle or C++ inference output exists. The complete command receipt is in benchmarks/tensorrt-model-connect/command-log.md; the machine-readable record is environment.json.

Do not replace that missing evidence with a same-family checkpoint, a cloud GPU purchase, or a guessed source revision. Before a measured run, resolve the checkpoint’s immutable Hugging Face commit and record its license and file hashes. NVIDIA’s current matrix does not publish an immutable revision for this Qwen row, so this guide does not pretend the floating model name is a pin.[5]

Keep the layers separate

• Model: Qwen/Qwen3-0.6B is the Hugging Face checkpoint. It has its own license, revision, files, behavior, and supply-chain risk.[7]
• Inference infrastructure: TensorRT Model Connect maps a supported checkpoint to TensorRT engines and packages model-specific assets in a .bundle. TensorRT remains the execution substrate.[1]
• Harnesses and skills: a shell harness or agent skill can fetch, build, inspect, and log work. It is orchestration, not a model implementation, TensorRT runtime, or a security boundary. Review it before it runs with credentials or broad filesystem access.
• Application: your C++ process loads the bundle and calls the task API. NVIDIA says the deployed application can run without a Python interpreter or PyTorch in its production runtime; that must be verified in the exact runtime container you ship.[1]

NVIDIA’s own repository includes an Agent Skills plugin and tells agents to read the repository instructions, inspect the exact model descriptors/manifests, and separate static evidence from GPU execution evidence.[8] That is sensible process guidance, not a reason to execute an unreviewed plugin in a privileged profile.

Prerequisites: use the source path on x86_64

The current release-wheel route is for Linux aarch64 with Python 3.10 or 3.12, glibc 2.39 or newer, and official TensorRT 11.1.0.106. Linux x86_64 users should use the source-build route instead.[4] For the source route, NVIDIA requires Linux x86_64 or aarch64, Docker, NVIDIA Container Toolkit, enough disk for the image and bundle, and a compatible GPU visible to nvidia-smi.[4]

On an approved GPU host, start with:

uname -m
nvidia-smi
docker --version

Record the GPU model, driver, reported CUDA version, compute capability, TensorRT version inside the container, container image digest, and free disk space. Do not mix TensorRT libraries, DSOs, or bundles from different cohorts.[4]

The pinned source checkout must precede any build:

git clone https://github.com/NVIDIA/TensorRT-Model-Connect.git
cd TensorRT-Model-Connect
git checkout f23a07e54521725d33d31c8d1f51d3a030d46a66

Then resolve Qwen/Qwen3-0.6B to a specific Hugging Face revision, inspect the model card and files, and write that revision plus file hashes into the environment manifest. Stop if the license or access conditions are not acceptable.

Build the documented Qwen bundle

NVIDIA’s source-build guide selects a GPU once, derives its SM, builds the matching development image, and mounts the source checkout into it.[4] From the pinned checkout:

GPU=0
SM="$(nvidia-smi -i "$GPU" --query-gpu=compute_cap --format=csv,noheader,nounits | tr -d '.[:space:]')"
IMAGE=trtmc-quickstart
case "$(uname -m)" in
  x86_64) DOCKERFILE=Dockerfile.dev.x86 ;;
  aarch64) DOCKERFILE=Dockerfile.dev.aarch64 ;;
  *) echo "Unsupported host architecture: $(uname -m)" >&2; exit 1 ;;
esac
docker build -f "$DOCKERFILE" -t "$IMAGE" requirements
SOURCE_DIR="$(git rev-parse --show-toplevel)"
docker run --rm -it --gpus "device=${GPU}" --ipc=host \
  --mount "type=bind,source=${SOURCE_DIR},target=/src" --workdir /src \
  --env TRTMC_SM="$SM" "$IMAGE" bash

Inside that disposable container, build only the CLI, TensorRT backend, and Qwen model DSO that the quick start needs:

python -m pip install --no-deps -e . -C py-only=true
TRTMC_BUILD_DIR="build-sm${TRTMC_SM}"
cmake -S . -B "$TRTMC_BUILD_DIR" -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES="${TRTMC_SM}-real" \
  -DTRTMC_BUILD_BACKEND_TRT=ON -DTRTMC_BUILD_BACKEND_RTX=OFF \
  -DTRTMC_BUILD_TESTS=OFF -DTRTMC_BUILD_BENCHMARKS=OFF \
  -DTRTMC_ENABLE_LIBTORCH_MULTINOMIAL=OFF
cmake --build "$TRTMC_BUILD_DIR" --parallel "$(nproc)" --target trtmc trtmc_backend_trt trtmc_model_qwen
export TRTMC_MODEL_PLUGIN_DIR="$TRTMC_BUILD_DIR/models"
export PATH="$PWD/$TRTMC_BUILD_DIR:$PATH"
trtmc version

Now build the bundle using NVIDIA’s bounded cache profile:

trtmc build Qwen/Qwen3-0.6B \
  --precision bf16 \
  --max-cache-length 16384 \
  --output qwen3-0.6b.bundle
trtmc inspect ./qwen3-0.6b.bundle
trtmc inspect ./qwen3-0.6b.bundle --list-engines

Keep the first failing command and full error rather than changing several flags. The documented inspection signals for this route are the Qwen family, qwen_decoder_kv_cache, BF16, the configured cache length, and two engine plans.[3]

Run twice, then prove the native boundary

The fixed CLI smoke prompt uses NVIDIA’s quick-start decoding values. It is deterministic in the harness because the prompt and seed are fixed; it is not a benchmark:

trtmc run ./qwen3-0.6b.bundle \
  --prompt "What is the capital of France? Answer in one word." \
  --chat-template --no-thinking --max-new-tokens 64 \
  --temperature 0.7 --top-k 20 --top-p 0.8 --seed 42

The expected signal is Paris and a clean exit, but record the complete non-empty output twice rather than filtering to one word.[3] benchmarks/tensorrt-model-connect/run-smoke.sh runs the build, both inspections, and two smoke calls with guards for trtmc, nvidia-smi, and TRTMC_MODEL_PLUGIN_DIR.

The second command in the “two-command” story is the application load/run, not another Python build step. The minimal task-level shape NVIDIA documents is:

#include <trtmc/pipeline.h>
auto pipeline = trtmc::load("qwen3-0.6b.bundle");
auto result = pipeline->generate(
    "What is the capital of France? Answer in one word.",
    {.max_new_tokens = 64});
std::cout << result.text << std::endl;

benchmarks/tensorrt-model-connect/native-smoke.cpp turns that into a complete program that rejects empty output. Resolve the compile/link command from the pinned checkout’s CMake/package integration rather than guessing flags in a tutorial. Preserve that command, its compiler output, and a runtime-container inventory that demonstrates neither Python nor PyTorch is required for the deployed process.

A measured pass requires all five conditions: a bundle built from the recorded source/checkpoint pair, successful bundle inspection, a successful native C++ load/generate, two non-empty fixed-prompt outputs, and cleanup evidence. The expected contract is stored in output-fixture.json; it is not captured output.

Troubleshooting and cleanup

Treat the first boundary as the problem. NVIDIA’s first-run guide maps missing nvidia-smi to driver/GPU access, an absent CLI to the active source build, CMake errors to the development container, missing model DSOs to the plugin directory, and ABI errors to mixed TensorRT/DSO cohorts.[9]

For a support request, retain the source commit, install path, model ID and resolved revision, build command, run command, bundle checksum, GPU/driver/CUDA/TensorRT receipt, first failing command, and complete error.[9] Do not post access tokens, gated-model URLs, private cache paths, or full agent transcripts.

Cleanup is part of the check. Remove only the recorded bundle path, stop/remove the disposable development container if it was not started with --rm, and delete the disposable checkout and cache paths you created. Do not delete a shared Hugging Face cache, a shared Docker image, or a model you did not build. Re-run docker ps -a, inspect the recorded bundle path, and record the cleanup result.

This is a narrow, source-grounded deployment recipe. A successful run says this source, checkpoint, stack, and prompt worked on one recorded host. It does not prove a performance advantage, stable API, production reliability, or compatibility for another GPU or fine-tune.

Sources

[1] https://developer.nvidia.com/blog/deploy-an-open-model-from-checkpoint-to-inference-in-two-commands-with-nvidia-tensorrt-model-connect/ — NVIDIA Technical Blog, “Deploy an Open Model from Checkpoint to Inference in Two Commands with NVIDIA TensorRT Model Connect,” August 28, 2026.

[2] https://github.com/NVIDIA/TensorRT-Model-Connect — project README, project status, source revision, and repository guidance, checked 2026-09-01.

[3] https://nvidia.github.io/TensorRT-Model-Connect/getting-started/quick-start — first Qwen bundle, inspection, fixed smoke prompt, and expected signal.

[4] https://nvidia.github.io/TensorRT-Model-Connect/getting-started/environment-and-repro and https://nvidia.github.io/TensorRT-Model-Connect/getting-started/source-build — host requirements, source-container commands, and source-build targets.

[5] https://nvidia.github.io/TensorRT-Model-Connect/models-recipes/overview — support-matrix scope and Qwen/Qwen3-0.6B entries.

[6] https://github.com/NVIDIA/TensorRT-Model-Connect/blob/main/LICENSE — Apache License 2.0 text.

[7] https://huggingface.co/Qwen/Qwen3-0.6B — Qwen model card, model identity, and model-level deployment information, checked 2026-09-01.

[8] https://nvidia.github.io/TensorRT-Model-Connect/agent-guide — agent workflow and evidence boundaries.

[9] https://nvidia.github.io/TensorRT-Model-Connect/getting-started/troubleshooting — first-run failure boundaries and support receipt.

Put this to work

Separate a source checkpoint, TensorRT deployment infrastructure, automation harnesses, and the native application that loads a bundle.

Try

Use a disposable supported-GPU host, pin the source and Hugging Face revisions, and follow the documented source-container path without local patches.

Prove it worked

Keep the host receipt, source commit, resolved checkpoint revision, build and native C++ commands, two fixed outputs, bundle checksum, and cleanup record.

Where it can pay

A passing smoke test establishes a narrow compatibility receipt, not a throughput claim, a production certification, or coverage for same-family fine-tunes.

Keep in view

• The official first NLP path builds Qwen/Qwen3-0.6B with trtmc, then loads the resulting bundle through a C++ task API.
• On Linux x86_64, the published path is a source build in NVIDIA's development container; release wheels are currently aarch64-only.
• This checkout is explicitly unmeasured: nvidia-smi is absent and Docker daemon access is denied, so no bundle or inference result was fabricated.
• The repository is Apache-2.0 licensed but explicitly experimental; match the exact checkpoint, source revision, GPU, CUDA and TensorRT cohort before treating it as support.

Learn the workflow: build a scoped project with evidence →