Baidu Releases Unlimited OCR, a 3B Model That Keeps the KV Cache Flat for Long-Document Parsing

Most end-to-end OCR models slow down as output grows. Each generated token adds to the KV cache. Memory rises and generation drags. Parsing dozens of pages becomes impractical. Baidu’s Unlimited OCR addresses this directly. It swaps the decoder’s attention for a design that keeps memory constant. TL;DR Unlimited OCR is a 3B-parameter Mixture-of-Experts model, with only 500M parameters active. It replaces decoder attention with Reference Sliding Window Attention (R-SWA), keeping the KV cache constant. The model parses dozens of pages in one forward pass under a 32K maximum length. It scores 93.23 on OmniDocBench v1.5, beating the DeepSeek OCR baseline by 6.22 points. It builds on DeepSeek OCR via continue-training, not a from-scratch run. What is Unlimited OCR? Unlimited OCR takes DeepSeek OCR as its baseline. It keeps the DeepEncoder and the Mixture-of-Experts decoder. The MoE design holds 3B total parameters but activates only 500M at inference. The DeepEncoder is the compression engine. It cascades a SAM-ViT under window attention with a CLIP-ViT under global attention. At the bridge, it applies 16× token compression. A 1024×1024 PDF image becomes just 256 visual tokens. Fewer input tokens mean a smaller prefill. DeepEncoder natively supports five resolution modes, and Unlimited OCR keeps two. ‘Base’ mode runs at 1024×1024 for multi-page work. ‘Gundam’ mode uses dynamic resolution for single pages. https://arxiv.org/pdf/2606.23050 How R-SWA Keeps the Cache Constant The contribution is Reference Sliding Window Attention. Standard Multi-Head Attention stores a key and value for every token. As output length T grows, the cache grows with it. The size is CMHA(T) = Lm + T. Memory and latency climb without bound. R-SWA breaks that link. Each generated token attends to all reference tokens, meaning the visual tokens and the prompt. It also attends to the preceding n output tokens, where n defaults to 128. Everything older is evicted. The cache becomes a fixed queue of size m + n. The size is CR-SWA(T) = Lm + min(n, T) ≤ Lm + n. It is bounded by a constant. As T grows far beyond n, the cache ratio trends toward zero. So memory stays flat and per-step latency stays flat. The research team compare this to soft forgetting. A person copying a book glances at the source and the last few words. They do not re-read everything transcribed so far. Visual tokens never undergo state updates. That avoids the progressive blurring seen in linear attention. The interactive simulator below lets you vary T and watch both caches respond. Animate decoding</button> <button id="reset">Reset</button> </div> <div class="cards"> <div class="card mha"> <div class="k">MHA KV cache</div> <div class="v" id="mhaVal">10,048</div> <div class="u">key/value entries</div> </div> <div class="card swa"> <div class="k">R-SWA KV cache</div> <div class="v" id="swaVal">2,176</div> <div class="u">key/value entries</div> </div> <div class="card"> <div class="k">Cache ratio ρ</div> <div class="v" id="ratioVal">0.217</div> <div class="u">R-SWA ÷ MHA</div> </div> <div class="card"> <div class="k">Memory saved</div> <div class="v" id="saveVal">78%</div> <div class="u">vs standard MHA</div> </div> </div> <div class="bars"> <div class="bar-label"><span>Standard MHA — grows with every token</span><b id="mhaBarTxt">10,048</b></div> <div class="bar-track"><div class="bar-fill mha" id="mhaBar" style="width:100%"></div></div> <div class="bar-label"><span>R-SWA — bounded by L<sub>m</sub> + n</span><b id="swaBarTxt">2,176</b></div> <div class="bar-track"><div class="bar-fill swa" id="swaBar" style="width:21%"></div></div> <div class="formula" id="fMha">C<sub>MHA</sub>(T) = L<sub>m</sub> + T = <b id="fMhaN">2,048 + 8,000 = 10,048</b></div> <div class="formula" id="fSwa">C<sub>R-SWA</sub>(T) = L<sub>m</sub> + min(n, T) = <b id="fSwaN">2,048 + 128 = 2,176</b></div> </div> <div class="stream-wrap"> <div class="stream-title">Attention view: each new token sees <b>all reference tokens</b> plus only the <b>last n output tokens</b>. Earlier output is evicted from the cache.</div> <div class="stream" id="stream"></div> <div class="legend"> <span><i class="dot" style="background:#b0b0b0"></i> Reference tokens (visual + prompt, always visible)</span> <span><i class="dot" style="background:#ffffff"></i> Active window (last n tokens)</span> <span><i class="dot" style="background:#1c1c1c"></i> Evicted output (soft-forgotten)</span> </div> </div> <p class="note">Grounding: Unlimited OCR keeps the full reference cache of size L<sub>m</sub> but holds only the most recent n output tokens (n defaults to 128). As output length T grows far beyond n, the cache ratio ρ(T) trends toward zero, so MHA's linear growth is replaced by a constant footprint. The page-to-token estimate uses the DeepEncoder figure of 256 tokens per 1024×1024 page. Numbers illustrate the cache formulas in the report, not a benchmark run.</p> <div class="foot"> <span>R-SWA cache formulas from the Unlimited OCR technical report (arXiv:2606.23050)</span> <span>Built by <b>Marktechpost</b></span> </div> </div> <script> (function(){ var pages=document.getElementById('pages'), tokens=document.getElementById('tokens'), win=document.getElementById('window'); var TOK_PER_PAGE=256; var anim=null; function fmt(n){return Math.round(n).toLocaleString('en-US');} function render(){ var P=+pages.value, T=+tokens.value, n=+win.value; var Lm=P*TOK_PER_PAGE; var mha=Lm+T; var swa=Lm+Math.min(n,T); var ratio=swa/mha; var saved=Math.round((1-ratio)*100); document.getElementById('pagesVal').textContent=P+(P===1?' page':' pages'); document.getElementById('tVal').textContent=fmt(T); document.getElementById('nVal').textContent=n; document.getElementById('mhaVal').textContent=fmt(mha); document.getElementById('swaVal').textContent=fmt(swa); document.getElementById('ratioVal').textContent=ratio.toFixed(3); document.getElementById('saveVal').textContent=saved+'%'; var maxC=mha; document.getElementById('mhaBar').style.width='100%'; document.getElementById('swaBar').style.width=Math.max(2,(swa/maxC)*100)+'%'; document.getElementById('mhaBarTxt').textContent=fmt(mha); document.getElementById('swaBarTxt').textContent=fmt(swa); document.getElementById('fMhaN').textContent=fmt(Lm)+' + '+fmt(T)+' = '+fmt(mha); document.getElementById('fSwaN').textContent=fmt(Lm)+' + '+fmt(Math.min(n,T))+' = '+fmt(swa); drawStream(T,n); postHeight(); } function drawStream(T,n){ var s=document.getElementById('stream'); s.innerHTML=''; var REF=10, OUT=34; for(var i=0;i<REF;i++){ var r=document.createElement('div'); r.className='tok ref'; s.appendChild(r); } var prog=Math.min(1, T/120000); var generated=Math.round(prog*OUT); var winCells=Math.max(1,Math.round((n/512)*8)); for(var j=0;j<OUT;j++){