I kept seeing examples of large language models (LLMs) and vision-language models (VLMs) using more specialized machine-learning models as tools. A general model could handle the question and final answer, while a model built for one narrow job—such as detecting objects or reading text—could provide more precise input. It sounded useful, but I wanted to see what would actually happen in my own experiments.
I started this project with two goals:
- Check whether pairing a general LLM or VLM with specialized models improves its answers.
- See how well this kind of pipeline can run on an edge device—in my case, an 8 GB Jetson Orin Nano.
I’m starting on my PC with an RTX 5090 because it lets me iterate quickly and focus on the first question without fighting the edge-device limits at the same time. This part covers two PC experiments: counting road users with help from YOLO, and answering broader questions with help from an OCR model. In Part 2, I’ll move the work to the Jetson and measure what really fits and how it performs there.
Quick summary
In these PC pilots:
- The YOLO-assisted pipeline got closer to the correct counts overall for all four VLMs I tested, reducing both average error and the larger misses.
- It did not improve every scoring method. The class-relative score rose for the weakest model and fell for the other three; in those three runs, car error fell while truck error rose.
- On one dashcam clip, adding text read by OCR moved one model’s location guess from about 504 km away to about 8 km away. On a second clip, OCR was neutral or harmful for the model that already read the important road sign by itself.
- My working hypothesis from these pilots is that a specialist is most useful when it supplies a cue the general model missed, while extra specialist output can become noise when the cue is already available. More clips and controlled prompts are needed to test how often that holds.
- Every full-resolution VLM run produced an observed device-wide PC peak above 8 GiB; the lowest was 13.2 GiB. These readings include all GPU allocations and do not reproduce the Jetson runtime, so they are screening signals rather than proof of what will fit there.
The counting results cover one fixed-camera clip, and the broader questions cover two dashcam clips. That is enough to expose some useful failure modes, but I’m still treating the results as pilots, not a verdict.
Two experiments, one question
I wanted to test the same general idea in two different ways:
- Give a VLM the object detections from YOLO and ask it to count road users.
- Give a VLM text extracted by PaddleOCR and ask broader questions about a dashcam video, including where it was recorded.
In both cases, the main comparison is the VLM with and without help from the specialist. The specialist changes with the task: YOLO is useful for locating objects, while OCR is useful for reading signs that may be too small or blurry for the VLM.
Experiment 1: Counting road users
Vision-language models (VLMs) can struggle to count small, repeated objects such as cars on a road or people in a crowd. Object detectors such as YOLO take a more direct route by locating individual objects, but their counts still depend on image resolution, thresholds, and recall.
That led me to the first experiment: if I give YOLO’s detections to a VLM along with the image, does the resulting pipeline count more accurately than the VLM alone?
I tested four configurations:
| Config | What it is |
|---|---|
| C0 | YOLO only — count straight from detections |
| C1 | YOLO → model via text only — detections serialized into a prompt, with no image |
| C2 | VLM only — the image, with no detections |
| C3 | VLM + YOLO — the image, detections, and instructions for using them |
The headline comparison is C3 versus C2. One limitation is worth stating up front: C3 differs from C2 in both the detector data and the instructions that explain how to use it. This pilot therefore compares two complete pipeline configurations. It does not isolate the causal effect of the detections alone; that would require another control prompt.
This experiment focuses on counting accuracy. I collected some PC-side memory telemetry to choose what to optimize next, but the end-to-end latency and energy cost on the 8 GB Jetson are still future work.
Counting setup
Data
I used a 3840 × 2160 fixed-camera clip of a two-lane road with about 12 cars per frame. I’ll refer to it simply as the counting clip; its internal test-data filename is not meaningful to the experiment. I sampled one frame per second, giving me 26 frames, and hand-labeled the six road-user classes in the question: person, car, truck, bus, motorcycle, and bicycle.
Only cars and trucks are present in the ground truth. That produces 50 present-class observations: 26 car counts and 24 truck counts. The fixed-camera clip also has little scene-level variation, so I use it only for counting. The broader questions use separate dashcam clips, which I’ll return to later.
Detector and cache
For the results below, I ran Ultralytics 8.4.81 with yolo26n.pt, imgsz=1920, conf=0.25, and the remaining prediction defaults. I also built a cache at the documented default imgsz=640 for comparison. See Ultralytics’ prediction settings.
C0, C1, and C3 reused the same imgsz=1920 detections; C2 used the image without detections. This removed detector-run variation from the comparisons that involved YOLO, but the measured runs still exclude live YOLO cost.
Resolution made a large difference. At imgsz=640, YOLO produced 91 car detections against 305 hand-counted cars, or about 30% detected-count coverage. At imgsz=1920, that rose to 192 of 305, or about 63%. I’m deliberately calling this coverage, not recall: I compared total detected and hand-labeled counts, not matched boxes at an intersection-over-union threshold. The detector still undercounted the cars substantially at imgsz=1920.
Models and run policy
I ran the experiment under WSL 2/Ubuntu on an RTX 5090 with 32 GB of VRAM. One local Ollama server handled every language and vision model, with one model used at a time.
The VLM tags were qwen3-vl:4b-instruct, openbmb/minicpm-v4.6, gemma4:e2b, and gemma4:e4b. For the text-only C1 path, I tested qwen3.5:4b, phi4-mini:3.8b, ministral-3:3b, and the PC-only qwen3.6:27b ceiling.
I discarded one warmup inference, then ran each configuration once across the 26 frames. I used temperature=0 and seed=0 to minimize sampling variation. That is not a guarantee of bit-for-bit reproducibility across hardware or software versions. The run metadata preserved the prompt and rubric versions, but not the Ollama server version or model digests; that is a reproducibility gap I’ll close in future runs.
Scoring
Each model produced a free-text answer. An offline qwen3.6:35b judge extracted the predicted integer count for each class, and ordinary code calculated the scores against ground truth. The judge did extraction, not arithmetic.
For each class present in a frame, the graded credit is max(0, 1 - abs(predicted - ground truth) / ground truth). Exact counts receive 1, and credit falls with relative error. A class absent from the ground truth but predicted more than once adds to the denominator as a hallucination penalty. I averaged the resulting per-frame scores over all 26 frames.
I also report mean absolute error (MAE), root mean squared error (RMSE), and median absolute error across the 50 present-class observations. These say how many objects the prediction was off by; RMSE gives larger misses more weight. I did not manually audit every judge extraction, so extraction mistakes remain a limitation.
Two methodology traps I walked into
The most interesting part of this project so far wasn’t the models. It was how close I came to drawing the wrong conclusion twice.
Trap 1: the fusion prompt made the VLM obey the detector
My first C3 prompt said to “use both the image and these detections.” On a clip where YOLO undercounted, the VLM read “9 cars, 0 trucks (confirmed by YOLO)” and deferred. It discarded its own better car count and even dropped a truck it had seen correctly. C3 collapsed to YOLO’s score.
I changed the prompt to say that detections were a fallible cross-check and that the image was primary. I also warned that the detector could miss small, distant, or occluded objects. After that, the VLM started overriding some YOLO undercounts upward.
There are two caveats. First, I called the detector counts a “lower bound” in the tested prompt, which is too strong unless false positives have been ruled out. Second, C2 received only the bare question, so this prompt change is a confound. The results below describe the tested C2 and revised-C3 pipelines; an empty, shuffled, or withheld-detection control is still needed to separate detector information from prompt-induced behavior.
Trap 2: the accuracy metric was blind
My first count metric was binary: a prediction was correct when it was within ±1 of the truth. On a 14-car frame, predicting 9 and predicting 12 were both simply “wrong.” The metric gave no credit for getting much closer and ranked the configurations nearly flat, around 0.46–0.50.
Switching to the graded score above, alongside MAE, RMSE, and median error, revealed structure the binary metric had hidden.
The lesson, twice over: on hard perception tasks, the prompt and metric can each manufacture a conclusion. Both need stress-testing before I trust the result.
Counting results
The graded score is the mean across 26 frames. MAE, RMSE, and median use the 50 present-class observations. Values are rounded to two decimals.
| Config | Model | Graded ↑ | MAE ↓ | RMSE ↓ | Median ↓ |
|---|---|---|---|---|---|
| C0 | YOLO only | 0.46 | 2.62 | 3.36 | 2.0 |
| C1 | Four tested C1 models | 0.46 | 2.62 | 3.36 | 2.0 |
| C2 | gemma4:e2b |
0.40 | 2.74 | 3.75 | 2.0 |
| C2 | gemma4:e4b |
0.56 | 2.62 | 3.75 | 1.0 |
| C2 | openbmb/minicpm-v4.6 |
0.65 | 2.80 | 4.05 | 2.0 |
| C2 | qwen3-vl:4b-instruct |
0.65 | 2.52 | 3.52 | 1.0 |
| C3 | gemma4:e2b |
0.51 | 2.18 | 2.89 | 2.0 |
| C3 | gemma4:e4b |
0.48 | 2.12 | 2.85 | 1.0 |
| C3 | openbmb/minicpm-v4.6 |
0.50 | 2.58 | 3.35 | 2.0 |
| C3 | qwen3-vl:4b-instruct |
0.57 | 2.40 | 3.21 | 2.0 |
1. Absolute count error fell for every tested VLM
On these 26 frames, every C3 pipeline had lower MAE and RMSE than its C2 counterpart, which is consistent with some of the larger errors becoming smaller. This is the cleanest pattern in this pilot, though the prompt confound means I cannot attribute it to detector data alone.
2. The class-relative score tells a different story
The graded score rose only for gemma4:e2b. It fell for the other three VLMs.
The same trade-off appeared in all three models whose graded score fell: car error went down while truck error went up. For qwen3-vl:4b-instruct, for example, the C2-to-C3 change reduced car MAE from 4.58 to 4.12 but increased truck MAE from 0.29 to 0.54. Missing a truck with a ground-truth count of 1 is a whole-class failure in the relative score, while it contributes only 1 to absolute error. The two metrics are measuring different things.
3. The weakest VLM benefited most on the graded score
Among these models, gemma4:e2b was the weakest counter by itself: its 0.40 graded score was below YOLO’s 0.46. The revised C3 pipeline lifted it to 0.51. The stronger VLMs did not show the same graded-score gain, although all of them reduced absolute error.
4. The tested C1 models did not change the count result
All four C1 models reproduced the same scores as YOLO on this clip. With this prompt and data, the model relayed the detector’s counts rather than improving them. The smallest tested model was therefore sufficient for this narrow C1 task; I would not generalize that result to other prompts or reasoning questions.
5. VLM choice produced the larger graded-score spread
For C2, the graded score ranged from 0.40 to 0.65 across the selected VLMs. That 0.25 spread was larger than any within-model C2-to-C3 change in this pilot. This statement applies to the graded score; the absolute-error metrics tell a more nuanced story.
Experiment 2: Reading clues from dashcam video
Counting was only one part of the experiment. I also wanted to ask questions that require a broader understanding of a video: Where was this drive recorded? Is a named university or landmark visible? How many turns and lane changes did the driver make?
YOLO is not the right specialist for those questions. The most useful external clues were words on street signs, highway signs, and buildings, so for this experiment I paired the VLMs with PaddleOCR instead.
Situation setup
I used two forward-facing dashcam clips: an urban drive in the New Orleans area and a motorway drive near Hoogeveen in the Netherlands. For each clip, I sampled an eight-frame strip that the VLM could inspect together. I tested the two models in the set that could accept multiple images: qwen3-vl:4b-instruct and openbmb/minicpm-v4.6.
The main comparison was simpler than the counting matrix:
| Config | What the VLM received |
|---|---|
| C2 | The same eight-frame strip, with no OCR text |
| C3 | The eight-frame strip plus text read by PaddleOCR, presented as a fallible hint |
I first ran OCR over the same eight frames seen by the VLM. Then I tried a denser OCR pass over 32 frames while still showing only eight frames to the VLM. OCR was cached, so the denser pass did not increase the number of images sent to the VLM.
One cross-clip difference matters here: the saved New Orleans runs used PaddleOCR’s auto-selected mobile models with frames capped at 2048 pixels, while the Netherlands runs used the PP-OCRv5 server models at the clip’s source resolution. The within-clip C2/C3 results still describe the pipelines I tested, but the reversal between clips is not a controlled replication because both the video and the OCR configuration changed.
The models answered in free text. An offline qwen3.6:35b judge extracted the most specific location each answer claimed and assigned it approximate coordinates. Ordinary code then calculated Haversine distance to my ground-truth map pin. I do not have a saved manual audit covering all 12 judge-selected locations and coordinates, so these kilometer values summarize the judged location claim, not every road or place assertion in an answer.
The university or landmark question used a name match and was scored only for the New Orleans-area clip; the Netherlands clip had no named institution in view. Turns and lane changes used absolute count error. I excluded the Netherlands turn result from my conclusions because its zero-turn ground-truth label is still provisional.
Geolocation results
The table below shows distance from the known filming location. Lower is better. These are single runs, and the two clips point in different directions.
| Clip | Model | VLM only | + OCR from 8 frames | + OCR from 32 frames |
|---|---|---|---|---|
| New Orleans-area drive | qwen3-vl:4b-instruct |
503.54 km | 503.54 km | 7.93 km |
| New Orleans-area drive | openbmb/minicpm-v4.6 |
955.55 km | 1,574.97 km | 506.15 km |
| Netherlands motorway drive | qwen3-vl:4b-instruct |
55.09 km | 57.40 km | 43.64 km |
| Netherlands motorway drive | openbmb/minicpm-v4.6 |
12.23 km | 32.42 km | 15.40 km |
The first clip showed why OCR coverage matters
On the New Orleans-area drive, qwen3-vl:4b-instruct guessed Houston from the images alone, about 504 km from the labeled location. The eight-frame OCR pass did not change that answer. The 32-frame pass found additional clues including CLAIBORNE and Ochsner; with those hints, the same model answered New Orleans and landed about 8 km away.
That was a dramatic improvement, but it was not a general OCR win. The denser hints moved openbmb/minicpm-v4.6 closer too, from about 956 km away to 506 km away, but its answer was still the wrong city. Neither model identified Chamberlain University correctly in the saved runs. A later native-resolution OCR test produced a closer reading of the sign, but I have not rerun the VLM with that improved OCR output, so I am not counting it as a success.
The second clip did not repeat the result
On the Netherlands motorway clip, openbmb/minicpm-v4.6 read the important place name directly from the images. Its VLM-only answer was already about 12 km from the known location near Hoogeveen. Adding OCR made that result worse: about 32 km away with the eight-frame OCR pass and 15 km away with the 32-frame pass.
qwen3-vl:4b-instruct improved modestly with the denser OCR, from about 55 km to 44 km, but it did not match MiniCPM’s image-only result. The model that performed best therefore changed between the two clips. With only two clips, I do not think it makes sense to call either model the stronger geolocation model.
The motion questions exposed a different limitation
The turn and lane-change answers were weak and inconsistent. That is not especially surprising: eight separated frames are enough to reveal signs, but they are a poor substitute for continuous motion. OCR also has no direct information about how the camera moved.
I am keeping those questions in the experiment because failed questions are useful too, but I would not use these runs to claim that the pipeline understands vehicle motion. I also left the Netherlands turn result out of this interpretation because its ground-truth label still needs confirmation. A future test needs a motion-oriented specialist such as optical flow, lane detection, visual odometry, or a video-native model.
What I take from both experiments
The combined result is more interesting than “specialists help.” In the counting experiment, YOLO reduced absolute error for every tested VLM, but it did not improve every metric. In the geolocation experiment, OCR transformed one answer when the important text was otherwise missed, then became unnecessary or harmful when another model could already read the decisive sign.
My working hypothesis is that a specialist is most valuable when it fills a specific gap in the general model’s perception. More input is not automatically better input. The specialist, its coverage, and the way its output is introduced all matter. These runs suggest that pattern; they do not yet isolate it from the OCR and prompt differences well enough to prove it.
What the PC memory readings do and do not say
These are whole-GPU NVML peaks from the RTX 5090, sampled every 100 ms during inference. They are not baseline-subtracted or process-specific, and I did not record or enforce Ollama’s resident-model state between runs. Each run used one warmup followed by the 26 measured frames, and the configurations ran sequentially. The table shows the counting runs, where all four VLMs were tested under the same task.
| Model | C2 peak GPU used | C3 peak GPU used |
|---|---|---|
gemma4:e2b |
19.0 GiB | 18.9 GiB |
gemma4:e4b |
20.7 GiB | 15.5 GiB |
openbmb/minicpm-v4.6 |
29.8 GiB | 29.8 GiB |
qwen3-vl:4b-instruct |
13.2 GiB | 13.2 GiB |
Every full-resolution VLM run produced an observed device-wide PC peak above 8 GiB. These PC readings tell me that optimization is necessary before the Jetson test; they do not establish what will fit on the device. qwen3-vl:4b-instruct is still my best candidate for an image-resolution optimization pass because it combined one of the best graded scores with the lowest observed PC peak.
The dashcam runs showed the same broad split between the two multi-image models: qwen3-vl:4b-instruct stayed around 12–16 GiB of observed whole-GPU use, while openbmb/minicpm-v4.6 reached roughly 30–32 GiB. I treat those numbers as screening evidence, not a prediction of Jetson memory use.
The gemma4:e4b C2-to-C3 drop from 20.7 to 15.5 GiB is an unexplained telemetry anomaly, so I would not interpret small memory differences between configurations. Likewise, openbmb/minicpm-v4.6 was a roughly 1.6 GB download but reached 29.8 GiB of whole-GPU use in this setup. The measurement shows that download size did not predict runtime GPU use; it does not isolate image activations as the cause.
Finally, this cached-detector protocol does not measure the deployed pipeline’s full latency, memory, or energy cost. I still need to run live YOLO and the VLM together on the Jetson and measure the complete path there.
Caveats
The counting experiment is one run over 26 temporally adjacent frames from one clip, not 26 independent scenes. Its C3 prompt differs from C2, the judge extractions were not all manually audited, and the detector was read from a cache rather than run as part of a live pipeline.
The situation experiment has only two clips and roughly one observation per question, model, and configuration. The two clips were useful precisely because the OCR result reversed, but they are nowhere near enough to establish how often that pattern will hold. Sparse frame strips also make the turn and lane-change questions especially fragile. The OCR preset and input width changed between clips, and C3 introduced both OCR text and instructions for using it; an empty or shuffled-OCR control is still needed. The kilometer values also depend on unaudited location extraction and approximate geocoding by the offline judge.
Across both experiments, the PC memory readings are rough screening measurements. The results support observations about these clips, pipelines, models, prompts, and metrics—not a general claim that model fusion always improves video understanding.
The next useful steps are prompt-only controls, more counting and situation clips, image and OCR resolution tuning, a better motion input, and a complete Jetson measurement.
The tables in this article were regenerated from the persisted run logs with the project’s report command. Hope this first look is useful—and saves someone else from trusting the first prompt, metric, or spectacular single result that appears to work.