<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Edge AI Notes]]></title><description><![CDATA[Edge AI Notes]]></description><link>https://hemkesh.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Wed, 18 Nov 2026 13:05:02 GMT</lastBuildDate><atom:link href="https://hemkesh.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Your slowest model shouldn't make your fastest decisions: lessons from an edge AI camera on a Jetson]]></title><description><![CDATA[How we keep slow models from acting on stale frames in Sentinel, an open-source edge AI camera on the Jetson Orin Nano. Four lessons and a live demo.]]></description><link>https://hemkesh.hashnode.dev/slowest-model-fastest-decisions-edge-ai-jetson</link><guid isPermaLink="true">https://hemkesh.hashnode.dev/slowest-model-fastest-decisions-edge-ai-jetson</guid><category><![CDATA[Computer Vision]]></category><category><![CDATA[Open Source]]></category><category><![CDATA[Jetson Orin Nano]]></category><category><![CDATA[NVIDIA]]></category><category><![CDATA[Python]]></category><category><![CDATA[AI]]></category><category><![CDATA[YoloV8]]></category><category><![CDATA[facenet]]></category><dc:creator><![CDATA[Hemkesh]]></dc:creator><pubDate>Tue, 17 Nov 2026 12:52:19 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a32aaa40e85c407103124f7/ec481d10-843b-4fb7-b1b9-a8127ad360af.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>One camera. One NVIDIA Jetson Orin Nano Super. Every model runs on the board.</p>
<p>That was the brief for <strong>Sentinel</strong>, our final-year project at Dayananda Sagar University: spot people, follow them across the frame, and open an incident when someone steps into a zone you care about. No cloud AI anywhere in the loop.</p>
<p>We expected the hard part to be the models. It wasn't.</p>
<p><strong>See it first:</strong> the <a href="https://hemkesh2021-dotcom.github.io/sentinel-site/">project website</a> has a hardware tour and a demo dashboard you can click around in. It is the real dashboard code replaying a scripted scene, so no Jetson is needed.</p>
<img src="https://raw.githubusercontent.com/hemkesh2021-dotcom/Sentinel_Surveillance/master/docs/media/dashboard-tour.gif" alt="Sentinel dashboard tour, recorded with synthetic demonstration data" style="display:block;margin:0 auto" />

<p><em>The dashboard with synthetic demo data. No camera or model was running.</em></p>
<h2>The pipeline</h2>
<table>
<thead>
<tr>
<th>Step</th>
<th>What does it</th>
</tr>
</thead>
<tbody><tr>
<td>See</td>
<td>YOLOv8n on TensorRT detects people</td>
</tr>
<tr>
<td>Follow</td>
<td>ByteTrack keeps each person's track</td>
</tr>
<tr>
<td>Decide</td>
<td>Zone rules open incidents in a local SQLite database</td>
</tr>
<tr>
<td>Tell</td>
<td>A restart-safe outbox delivers alerts (Telegram in our tests)</td>
</tr>
<tr>
<td>Show</td>
<td>A dashboard that listens on 127.0.0.1 only</td>
</tr>
</tbody></table>
<p>Two optional lanes sit beside it: face recognition (YuNet + Facenet512) and a local vision-language model (LFM3-VL-2B through llama.cpp). Both are still under validation, so they are off by default. More on that below.</p>
<h2>Lesson 1: your slowest model shouldn't make your fastest decisions</h2>
<p>On a small board, models finish at very different speeds. The detector keeps up with the stream. Face matching is slower. A vision-language model is slower still.</p>
<p>Now imagine the camera drops and reconnects. A slow model is still chewing on a frame from <em>before</em> the reconnect. When its answer finally lands, the naive pipeline treats it as news about <em>now</em>.</p>
<p>That is how an old frame fires a new alert.</p>
<p>Our rule: <strong>every observation is evidence about one specific frame, never about "now".</strong> So every observation carries three stamps:</p>
<ul>
<li>the frame's <strong>source time</strong></li>
<li>the <strong>boot ID</strong> of the run that produced it</li>
<li>a <strong>stream epoch</strong> that changes on every reconnect</li>
</ul>
<p>Before any result can drive a decision, it has to pass a check like this (a simplified sketch, not the project's exact code):</p>
<pre><code class="language-plaintext">def can_drive_live_decision(obs, now, current):
    if obs.boot_id != current.boot_id:
        return False          # from a previous run
    if obs.stream_epoch != current.stream_epoch:
        return False          # from before a reconnect
    if now - obs.frame_time &gt; MAX_AGE:
        return False          # simply too old
    return True
</code></pre>
<p>A result that fails is not thrown away. It can still annotate history. It just never gets to act on the present.</p>
<p>One more detail that mattered: age is measured with an injected monotonic clock, not wall time. That makes the rule testable. A fake clock lets a test replay "the model answered late" without waiting for a real model to be slow.</p>
<h2>Lesson 2: "I don't know" is a valid answer</h2>
<p>A monitoring system that guesses is worse than one that stays quiet. Three rules came out of that:</p>
<ul>
<li>A face is tied to <strong>at most one</strong> tracked person. If ownership is ambiguous, it stays unresolved.</li>
<li>An unknown face is <strong>never</strong> labelled a stranger. Not recognising someone is not evidence about who they are.</li>
<li>Fire or smoke reported by the scene model alone is <strong>never</strong> a critical alarm.</li>
</ul>
<p>The dashboard follows the same idea. Empty, fresh, stale and offline are four different states, and it shows each one explicitly instead of leaving an old picture on screen.</p>
<h2>Lesson 3: an alert that dies with the process was never an alert</h2>
<p>An in-memory queue loses everything on a restart. So incidents and their pending alerts are written to SQLite, and a worker delivers from that outbox with retries. If Sentinel restarts between "incident opened" and "alert sent", the alert still goes out afterwards.</p>
<h2>Lesson 4: test without the hardware</h2>
<p>Most of Sentinel can be tested on a laptop. The portable suite needs no camera, GPU or model, and CI runs it on every push under Python 3.10 and 3.12. On the commit the beta was built from, 4,224 tests passed.</p>
<p>The trick is keeping hardware adapters at the edges, so importing the core never requires CUDA or TensorRT.</p>
<h2>What isn't done</h2>
<p>Sentinel is an experimental beta, and we would rather say so than oversell it:</p>
<ul>
<li>Face recognition has not passed its formal validation.</li>
<li>Chat with the local vision-language model is not admitted for normal use.</li>
<li>The install package has run only on our own Jetson.</li>
</ul>
<p>The README keeps a feature table that says what is admitted and what is not, with the evidence behind each claim.</p>
<h2>Try it</h2>
<ul>
<li><strong>In a browser:</strong> the <a href="https://hemkesh2021-dotcom.github.io/sentinel-site/">project website</a> and its demo dashboard</li>
<li><strong>On any computer with Python 3.10+:</strong> clone the repo and run the test suite</li>
<li><strong>On a Jetson Orin Nano:</strong> the experimental install package in the repo</li>
</ul>
<p>Code (MIT): <a href="https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance">https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance</a></p>
<p>If you run a slow model next to a fast one, how do you keep the slow one's answers honest? I'd like to hear how others handle it.</p>
]]></content:encoded></item><item><title><![CDATA[Why cudaMalloc fails on NVIDIA Jetson Orin Nano Super — and the one flag that fixes it]]></title><description><![CDATA[Stock llama.cpp fails to load GGUF models on Jetson Orin Nano Super due to unified memory. Here's the one CMake flag that fixes it.]]></description><link>https://hemkesh.hashnode.dev/why-cudamalloc-fails-on-nvidia-jetson-orin-nano-super-and-the-one-flag-that-fixes-it</link><guid isPermaLink="true">https://hemkesh.hashnode.dev/why-cudamalloc-fails-on-nvidia-jetson-orin-nano-super-and-the-one-flag-that-fixes-it</guid><category><![CDATA[Jetson Orin Nano]]></category><category><![CDATA[NVIDIA]]></category><category><![CDATA[Cpp Tutorial ]]></category><category><![CDATA[ Edge AI]]></category><category><![CDATA[llama.cpp]]></category><category><![CDATA[embedded systems]]></category><category><![CDATA[Deep Learning]]></category><category><![CDATA[GPU Computing]]></category><dc:creator><![CDATA[Hemkesh]]></dc:creator><pubDate>Wed, 17 Jun 2026 14:22:33 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a32aaa40e85c407103124f7/2cba00cc-c30b-4c5b-a041-f1c133f27975.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>If you've tried running a GGUF model via llama.cpp on a Jetson Orin Nano Super and hit this error:</p>
<pre><code class="language-plaintext">NvMapMemAllocInternalTagged: error 12
cudaMalloc failed: out of memory
</code></pre>
<p>...while tegrastats shows the GPU idle and several GB of RAM free — this post is for you.</p>
<h2>The hardware context</h2>
<p>The Jetson Orin Nano Super (8 GB) uses unified memory. There is no separate VRAM pool — the CPU and GPU share one physical 8 GB block. This is how all Jetson Orin-series SoCs work, and it's what makes them cost-effective for edge AI.</p>
<h2>Why stock llama.cpp fails</h2>
<p>When llama.cpp allocates GPU tensor buffers, it calls cudaMalloc. On a discrete GPU, cudaMalloc carves out memory from a dedicated VRAM pool. On the Orin Nano Super, that dedicated pool doesn't exist — the allocator hits the NvMap interface, which rejects the request with error 12 (ENOMEM) even though the unified pool has plenty of space.</p>
<p>The result: the model fails to load with a misleading out-of-memory error despite free RAM.</p>
<h2>The fix</h2>
<p>Set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 as an environment variable when you launch the server. This is a runtime flag, not a build option — in llama.cpp's CUDA backend it switches GPU allocations from cudaMalloc to cudaMallocManaged, which works with the unified pool.</p>
<p>First, build llama.cpp with CUDA for the Orin's compute capability (sm_87):</p>
<pre><code class="language-shell">git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=87
cmake --build build --config Release -j$(nproc)
</code></pre>
<p>Then enable unified memory at launch by setting the environment variable:</p>
<pre><code class="language-shell">GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 ./build/bin/llama-server \
    -m LFM2-VL-1.6B-Q4_0.gguf \
    --mmproj mmproj-LFM2-VL-1.6B-Q8_0.gguf \
    --n-gpu-layers 999
</code></pre>
<p>That's it. The model loads and runs fully GPU-accelerated.</p>
<h2>What I was building</h2>
<p>I ran into this while building SENTINEL — a self-hosted AI surveillance system for my B.Tech minor project. It runs YOLOv8n + TensorRT for person detection, DeepFace for face recognition, and LFM2-VL 1.6B for scene understanding — all on the Jetson, zero cloud dependency.</p>
<p>Full project and complete BUILD.md:<br /><a href="https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance">https://github.com/hemkesh2021-dotcom/Sentinel_Surveillance</a></p>
<p>Hope this saves someone a few hours of debugging.</p>
]]></content:encoded></item></channel></rss>