In Part 1, I tested whether a YOLO detector could help vision-language models (VLMs) understand road video. In Part 2, I compared the combined pipeline with both of its components and selected a stronger detector.
This time I moved the experiment to an 8 GB NVIDIA Jetson Orin Nano Super. I wanted to know what actually fits, whether the PC results survive the move, and whether this small board can eventually run the detector and VLM together.
The most useful discovery was not an out-of-memory error. Ollama kept one model loaded for five minutes, then loaded the next package mostly on the CPU. The benchmark completed and returned perfectly plausible answers, but it was about four times slower than a clean run. Fixing that one default changed the fit, latency, and energy conclusions.
Quick summary
- Both tested MiniCPM-V 4.6 packages fit on JetPack 7.2.1 with a 32768-token context and
100% GPUplacement. They peaked at about 5.2–5.4 GB of the 7546 MB unified-memory pool, with swap essentially flat. - Both packages are nominally 1B, despite an earlier version of this article calling the namespaced package 8B. Their different tags and manifests still make them separate experiment rows, but this is not a model-size comparison.
- The 4B text-only
qwen3.5:4bpackage completed at a 7300 MB peak, but swap grew by about 391 MB. By my fit rule, it ran but did not fit. The 4B vision modelqwen3-vl:4b-instructran out of memory. - With clean, fully GPU-resident loads, the Jetson took 6.9–8.4 times as long as the RTX 5090. Its measured module-input energy was 39–46% lower than the PC’s measured GPU-board energy for these requests, although those sensor scopes are not identical.
- The current counting runs did not resolve an accuracy difference between the PC and Jetson deployments. The gaps stayed within ±0.03, below the 0.037 drift I had already seen on a fixed configuration rerun.
The current test system
I tested the 8 GB Jetson Orin Nano Developer Kit in its Super configuration. The Super boost comes from the flashed configuration and higher power and clock modes, not a different system on module. A production installation would still need a production carrier, power supply, thermal design, and enclosure.
The board is now on the software stack NVIDIA publishes as JetPack 7.2.1:
| Setting | Measured value |
|---|---|
| Hardware | Jetson Orin Nano Developer Kit, 8 GB, Super configuration |
| Memory pool | 7546 MB reported by tegrastats, shared by CPU and GPU |
| Swap | 2 GB swap file on NVMe; no zram |
| Compute | 6 CPU cores, 4 GPU TPCs |
| Software | Ubuntu 24.04.5, L4T R39.2.1, Python 3.12.3 |
| GPU stack | CUDA 13.2, cuDNN 9.20, TensorRT 10.16.2 |
| Ollama | 0.33.3 |
| Idle input power | About 6.45 W stock; 6.96 W with clocks pinned |
CPU and GPU share one physical pool. On the PC, the NVML metric I logged was whole-device GPU memory use. On the Jetson, tegrastats reports system-wide unified memory, including the operating system. Each metric is useful within its machine, but their absolute values are not comparable across machines.
JetPack 7.2.1 also changed the swap story. The previous JetPack 6 setup used 3.7 GiB of zram, compressed into the same RAM pool being measured. The current stack uses an NVMe-backed swap file, so swap growth now means the workload is hitting storage. That is easier to interpret, but it still means the model has not met my deployment requirement.
I count a run as fitting only when all four checks pass:
- Ollama loads the requested context window.
ollama psreports100% GPUprocessor placement.- Swap does not materially grow during the workload.
- Peak physical RAM stays below the pool with measurable headroom.
On this unified-memory system, 100% GPU describes Ollama’s processor placement. It does not mean the board has a physically separate VRAM pool.
What each configuration means
The counting experiment reuses the four configurations from the earlier articles:
| Config | Pipeline |
|---|---|
| C0 | Return YOLO’s counts directly |
| C1 | Pass YOLO’s counts to a text LLM, without an image |
| C2 | Give the image to a VLM, without detector hints |
| C3 | Give the VLM both the image and detector hints |
The graded counting score ranges from 0 to 1, with higher being better. It gives partial credit from relative count error for each road-user class present in a frame and penalizes hallucinated classes. The same PC-hosted judge scored both the PC and Jetson answers.
Each configuration ran once over 26 adjacent frames from one fixed-camera clip. This is one clip-level pilot, not 26 independent scene replications. C3 also uses dedicated instructions for applying detector hints, so C3−C2 compares two complete pipelines rather than isolating detections alone.
What fits on JetPack 7.2.1
The reliable answer came from loading each package on an otherwise clean Jetson, setting num_ctx = 32768, checking processor placement, and watching both physical memory and swap.
| Ollama tag | Public package | Peak RAM | Swap during run | Placement | Verdict |
|---|---|---|---|---|---|
minicpm-v4.6 |
1B, 1.6 GB | 5394 MB | 93–96 MB | 100% GPU |
Fits |
openbmb/minicpm-v4.6 |
1B, 1.6 GB | 5350 MB | 91–93 MB | 100% GPU |
Fits |
qwen3.5:4b |
4B, 3.4 GB | 7300 MB | 84→475 MB | 100% GPU |
Ran, did not fit |
qwen3-vl:4b-instruct |
4B, 3.3 GB | — | — | — | OOM |
gemma4:e2b |
7.2 GB | — | — | — | Excluded; package exceeds the usable budget |
gemma4:e4b |
9.6 GB | — | — | — | Excluded |
qwen3.6:35b |
About 23 GB | — | — | — | PC-only judge by design |
The swap ranges for the two MiniCPM runs moved by only 2–3 MB across thousands of samples. I treat that as effectively flat. qwen3.5:4b is different: it finished and stayed under the 7546 MB RAM number, but it pushed hundreds of megabytes into NVMe swap and left only 246 MB of physical headroom. That is not a safe deployment choice.
The two MiniCPM tags need a separate warning. Ollama’s public registry identifies openbmb/minicpm-v4.6 as a 1B model built around a Qwen3.5-0.8B language model and a SigLIP2-400M vision encoder. The package recorded about 1.10 GB resident through Ollama’s API. The non-namespaced package recorded about 1.13 GB.
They resolved to different manifests in these runs—36c9b04e8c47 and e95583acac77—so I report them as separate package-level results. I do not interpret their differences as a 1B-versus-8B effect. That incorrect label was responsible for a much bigger conclusion than it deserved.
Two Ollama defaults that changed the experiment
The context window
My first Jetson smoke test worked, but ollama ps showed a context length of 4096. The same model on the 32 GB PC received 32768.
I had never set num_ctx. Ollama documents VRAM-based default context tiers: systems below 24 GiB receive 4K, while systems from 24 to 48 GiB receive 32K. The observed difference followed those tiers; it was not a model choice.
The longest prompts made the risk concrete:
| Experiment | Maximum prompt tokens | Fits 4096? |
|---|---|---|
| Counting, one frame, C2 and C3 | 2237 | Yes |
| Situation reasoning, eight-frame strip, MiniCPM C3 | 5252 | No |
| Situation reasoning, eight-frame strip, Qwen3-VL C3 | 16799 | No |
The counting prompts fit either window, but the situation prompts did not. Depending on the exact endpoint, model, and Ollama version, an overlength request may be truncated or rejected. Either outcome makes a PC-versus-Jetson comparison invalid.
I now set num_ctx = 32768 explicitly and record the effective context. Available memory at model load time affects processor placement; it does not change the context value I requested.
The five-minute keep-alive
The second default was harder to spot. Ollama keeps models loaded for five minutes by default. My benchmark processed minicpm-v4.6 first and openbmb/minicpm-v4.6 second. When the second package loaded, the first was still occupying the shared pool.
On Ollama 0.33.3, ollama ps showed the result:
NAME SIZE PROCESSOR CONTEXT
openbmb/minicpm-v4.6:latest 1.1 GB 89% CPU/11% GPU 32768
minicpm-v4.6:latest 1.1 GB 100% GPU 32768
The benchmark did not crash. It produced normal-looking answers and scores. Only the very low prefill rate hinted that most of the work was no longer running where I thought it was.
I fixed the benchmark at the model boundary: before starting the next model, it explicitly stops every other resident model. C2 and C3 for the same package still share one load, avoiding a roughly 129-second cold load before every frame. Each result row now records Ollama’s total loaded size, GPU-resident size, and derived GPU fraction. A row is rejected if the placement is not the intended 100% GPU.
This check is more important than inferring trouble from throughput. Tokens per second can reveal an anomaly, but ollama ps or /api/ps tells me directly where Ollama loaded the model.
How the corrected load changed the result
The historical bad run used JetPack 6/L4T R36.4.7 and a split processor placement. The corrected run uses JetPack 7.2.1/L4T R39.2.1, Ollama 0.33.3, and 100% GPU placement:
| Namespaced MiniCPM package | Old split run | Current clean run | Difference |
|---|---|---|---|
| C2 stage latency | 35322 ms | 9001 ms | 3.9× faster |
| C3 stage latency | 30175 ms | 6452 ms | 4.7× faster |
| C2 measured energy | 481 J | 163 J | 2.9× lower |
| C3 measured energy | 411 J | 118 J | 3.5× lower |
| Prefill throughput | 28 tok/s | 270 tok/s | 9.6× higher |
| Peak RAM | 7461 MB | 5239 MB | 2222 MB lower |
This is an operational before-and-after correction, not a clean experiment that isolates the layer split. The operating system, JetPack stack, Ollama version, swap implementation, and GPU software also changed. The official-library MiniCPM package became 8–19% slower after the upgrade, which is another reason not to assign the entire difference to one mechanism.
What I can say is narrower and still useful: the old 7461 MB, 30-second result did not describe a clean, fully GPU-resident load. It cannot support the earlier claims that this package used the whole memory budget or that a larger model reversed the Jetson’s energy advantage.
Counting accuracy on the current stack
The post-upgrade comparison uses L4T R39.2.1, corrected detector caches, ground-truth version 2, the same model manifests, 1920-pixel VLM inputs, prompt version, 32K context, and PC-hosted judge. Ollama 0.33.3 is matched for C0, C1, and C3. The older PC C2 rows use Ollama 0.30.11, because C2 does not consume the detector cache and was not repeated with the cache-focused batch.
| Config | Model package | PC | Jetson | Jetson − PC |
|---|---|---|---|---|
| C0 | yolo26s@2560 |
0.6478 | 0.6478 | 0.0000 |
| C1 | qwen3.5:4b |
— | 0.6478 | — |
| C2 | minicpm-v4.6 |
0.6031 | 0.6084 | +0.0053 |
| C3 | minicpm-v4.6 |
0.6948 | 0.6766 | −0.0182 |
| C2 | openbmb/minicpm-v4.6 |
0.5632 | 0.5661 | +0.0029 |
| C3 | openbmb/minicpm-v4.6 |
0.6478 | 0.6747 | +0.0269 |
C0 is identical because it reads the same saved detections and invokes no VLM. It checks that the cache and scoring path crossed machines intact, but it is not evidence about Jetson inference.
The VLM differences have no consistent direction and remain within ±0.03. The two C3 rows are the cleanest current comparison because their Ollama application version is matched; their gaps are −0.018 and +0.027. Both are below the 0.037 drift observed when rerunning one fixed configuration. I therefore cannot resolve an accuracy cost or benefit between these two deployment stacks.
Within the Jetson runs, C3 scored above C2 by 0.068 for minicpm-v4.6 and 0.109 for the namespaced package. C3 also stayed near the detector-only C0 score: +0.029 and +0.027. With one run per configuration, that repeats the qualitative “fusion helps the VLM” result without showing that the combined pipeline beats a good detector at counting.
Latency and measured energy
The timing and energy columns have different boundaries. “Stage ms” sums the stages reported by Ollama. Energy integrates sampled power across the full client-observed request, including overhead outside those stages. That is why multiplying the displayed stage time by the displayed average power does not reproduce the joules column.
The sensor scopes differ as well:
- PC watts and joules come from NVML’s RTX 5090 GPU-board power reading.
- Jetson watts and joules come from the module’s
VDD_INinput rail throughtegrastats. - Neither value is baseline-subtracted, and neither is a whole-PC wall-power measurement.
These are operational readings of the tested components, not laboratory-equivalent total-system energy measurements.
As in the accuracy table, the PC C2 rows retain Ollama 0.30.11; the C3 application versions are matched at 0.33.3. I therefore treat the C2 slowdown as a comparison of complete stacks, not a hardware-only ratio.
Latency:
| Config | Model package | PC stage ms | Jetson stage ms | Slowdown |
|---|---|---|---|---|
| C2 | minicpm-v4.6 |
1302 | 9775 | 7.5× |
| C3 | minicpm-v4.6 |
1134 | 9498 | 8.4× |
| C2 | openbmb/minicpm-v4.6 |
1304 | 9001 | 6.9× |
| C3 | openbmb/minicpm-v4.6 |
886 | 6452 | 7.3× |
Measured energy and average power:
| Config | Model package | PC J | Jetson J | PC W | Jetson W |
|---|---|---|---|---|---|
| C2 | minicpm-v4.6 |
320 | 177 | 147 | 17.7 |
| C3 | minicpm-v4.6 |
285 | 173 | 153 | 17.8 |
| C2 | openbmb/minicpm-v4.6 |
302 | 163 | 150 | 17.6 |
| C3 | openbmb/minicpm-v4.6 |
217 | 118 | 137 | 17.6 |
For these four requests, the Jetson took 6.9–8.4 times as long. Its module-input energy readings were 39–46% lower than the PC’s GPU-board readings. The broader Jetson sensor still reported less energy, but the unequal scopes mean this is directional rather than a complete whole-system comparison.
The two packages are both nominally 1B, so the table says nothing about a model-size crossover. It says that, once both packages are fully GPU-resident, they show the same broad trade: much longer latency on the Jetson with lower measured energy for these requests.
The detector does not show the same ratio. TensorRT yolo26s measured 3.43 J per frame on the Jetson versus 3.69 J from the PC GPU, only about 7% lower, while taking 7.3 times as long. That is a separate component result with the same sensor-scope caveat.
Can the current box do the whole job alone?
The controlled measurements above read a detection cache built on the PC. That is deliberate: it keeps detector input fixed while comparing VLM deployments. A real system needs camera input, YOLO, and a VLM on the same Jetson.
On the previous JetPack 6 stack, I demonstrated that the official-library MiniCPM package and a TensorRT FP16 yolo26s detector could coexist at the full 2560-pixel detector input. The probe peaked at 5837 MB of 7620 MB, leaving 1783 MB, with swap flat at 121 MB across 3331 samples.
That is historical feasibility evidence, not a current JetPack 7.2.1 result. The memory pool and software stack changed, and a TensorRT 10.3 engine should not be treated as a TensorRT 10.16.2 artifact. I have not yet repeated the co-resident detector-plus-VLM measurement with a freshly built current engine and the corrected model-unload policy. I therefore do not claim a current autonomous headroom, latency, or energy result here.
I do not reuse the earlier autonomous timing numbers. The namespaced MiniCPM rows had the same magnitude as the known split-load runs, and processor placement was not recorded. Repeating those numbers would repeat the mistake this article is meant to document.
Situation reasoning: input variation dominated
The other half of the project asks clip-level questions such as where a road is, which university appears, and how many lane changes occur. Each request receives an eight-frame strip. Early single runs seemed to show the Jetson outperforming the PC on geolocation, but repeated strips showed how fragile that interpretation was.
I repeated C2 and C3 five times with the namespaced MiniCPM package on two dashcam clips. The PC and Jetson used identical five-strip frame selections for each matched comparison.
| Clip | Config | PC median | PC range width | Jetson median | Jetson range width | Median gap |
|---|---|---|---|---|---|---|
| A | C2 | 0.2000 | 0.0000 | 0.2000 | 0.0000 | 0.0000 |
| A | C3 | 0.2000 | 0.0000 | 0.2000 | 0.0000 | 0.0000 |
| B | C2 | 0.2000 | 0.7579 | 0.4437 | 0.7648 | +0.2437 |
| B | C3 | 0.4179 | 0.9298 | 0.5229 | 0.1058 | +0.1050 |
On clip B, the PC C3 scores ranged from 0.0045 to 0.9343 depending on which eight frames appeared. The original Jetson score of 0.4213 sits almost exactly on the PC median of 0.4179. Nine of the ten Jetson strips fell within the matching PC range, while clip A stayed flat at 0.2000 everywhere.
At this sample size, strip selection varied far more than the platform gap. I could not resolve a device effect on this task. The earlier explanation involving a sampled token was wrong too: at temperature 0, decoding is greedy. Changing the eight-frame strip changed the answer.
Practical lessons
Fit is a measured state, not a file-size estimate. Check context, processor placement, physical memory, and swap together. “The request completed” is not a fit criterion.
Pin the server’s knobs. Ollama’s context default gave the two machines different windows, and its keep-alive let one benchmark row inherit the memory state of the row before it.
Isolate models at arm boundaries. Unload other packages, keep the active package resident across its requests, and record /api/ps placement on every result row.
Use throughput as an alarm, not as proof. A 9× prefill-rate difference exposed this problem, but the processor split was the direct evidence.
Treat energy as a property of the measured pairing. Timing boundaries, power rails, serving behavior, output length, and package manifests all matter. Parameter labels alone do not explain the result.
Limitations and next step
The counting results are one run per configuration over one 26-frame clip. The current C2 PC rows also retain an Ollama-version gap. The operating systems, drivers, CUDA stacks, and hardware differ between PC and Jetson, so this compares two deployment platforms rather than silicon in isolation.
Peak memory is sampled from a shared system-wide pool. The 2–3 MB swap movement in the MiniCPM runs is small enough to treat as noise here, but longer soak tests are still needed. The 246 MB physical headroom in the text-only Qwen run is nowhere near enough for a camera pipeline.
Finally, nothing here is a continuous live-camera workload. The current controlled runs reuse a saved detector cache, and the previous co-residency test belongs to JetPack 6. The next useful test is a JetPack 7.2.1 camera → TensorRT detector → VLM run with a freshly built engine, 100% GPU placement recorded, swap flat, and the whole pipeline held under load long enough to expose memory creep.
The answer today is narrower but more trustworthy: both 1B MiniCPM packages fit as isolated VLMs on the 8 GB Jetson, the 4B text model does not meet my deployment rule, and the strongest 4B VLM still OOMs. The full autonomous pipeline is the next measurement, not a conclusion I can carry forward from the old stack.