Timmy_Foundation/turboquant
15
0
Fork 0
Code Issues 38 Pull Requests 31 Actions Packages Projects Releases Wiki Activity
Files
27ebfa3525bfd02eb1915f374b837cf63dfa8900
turboquant/benchmarks/__pycache__/run_test_matrix.cpython-312.pyc

186 lines
22 KiB
Plaintext
Raw Normal View History

Fix #11: Full test matrix — 10 prompts + quality + performance Test matrix runner (benchmarks/run_test_matrix.py) implementing all acceptance criteria from #11: Quality Tests: - 10 practical prompts with expected-pattern matching - Perplexity proxy (WikiText-2 chunks) - Needle-in-Haystack at 8K/16K/32K contexts - Multi-turn context retention (prompt #7) Performance Tests: - tok/s at 4K/8K/16K context - TTFT proxy measurement - Peak memory (macOS/Linux) - Context ceiling binary search Outputs: - JSON: reports/test-matrix-YYYY-MM-DD.json - Markdown: reports/test-matrix-YYYY-MM-DD.md - Go/No-Go assessment with issue list Smoke test: 10/10 quality, 3/3 needle-in-haystack on qwen2.5:7b. Refs: Timmy_Foundation/turboquant#11
2026-04-14 22:10:39 -04:00
<EFBFBD>
<00><><EFBFBD>iu@<00><00>\<00>dZddlZddlZddlZddlZddlZddlZddlZddlmZm Z ddl
m Z ddl m Z mZmZmZ ddededed ed
ed ed efd <0A>Zdededed efd<0F>Zdededed efd<11>Zdeded efd<12>Zdeded efd<13>Zdeded efd<14>Zdededededededed eeeffd<1B>Zd<1C>Zedk(re<1B>yy)u<>
TurboQuant Full Test Matrix — Issue #11
Runs the complete validation matrix:
- 10 practical prompts (quality comparison)
- Perplexity (PPL) on WikiText-2
- Needle-in-Haystack at 8K/16K/32K/64K/128K
- Performance benchmarks (tok/s, TTFT, peak memory)
- Context ceiling test
Outputs: reports/test-matrix-YYYY-MM-DD.json + .md
Usage:
python3 benchmarks/run_test_matrix.py --model qwen2.5:7b --base-url http://localhost:11434
python3 benchmarks/run_test_matrix.py --model qwen2.5:7b --base-url http://localhost:11434 --skip-quality
python3 benchmarks/run_test_matrix.py --model qwen2.5:7b --base-url http://localhost:11434 --skip-performance
<EFBFBD>N)<02>datetime<6D>timezone)<01>Path)<04>Dict<63>List<73>Optional<61>Tuple<6C>prompt<70>model<65>base_url<72> num_predict<63>num_ctx<74>timeout<75>returnc <00><><00>ddl}ddl}|jd<03><00>d<04>}tj||d||d<06>d<07><04>j <00>} |j j|| dd id
<EFBFBD> <0B>}
|j<00>} tj<00>} |j j|
|| <0B> <0C>} tj| j<00><00>}tj<00>| z
}|jd d<01>}|jdd<0F>}|dkDr||dz z nd}|jdd<12>t|d<0F>t|d<13>||jdd<01>|jdd<01>d<16>S)zNCall Ollama /api/generate. Returns {response, eval_count, eval_duration, ...}.rN<>/z /api/generateF<65>r r)r r
<00>stream<61>optionsz Content-Typezapplication/json<6F>POST)<03>data<74>headers<72>method)r<00>context<78>
eval_count<EFBFBD> eval_duration<6F>ge<><65>A<EFBFBD>response<73><00><00>prompt_eval_count<6E>total_duration)r<00>tok_s<5F> wall_timerr!<00>total_duration_ns)<0F>urllib.request<73>ssl<73>rstrip<69>json<6F>dumps<70>encode<64>request<73>Request<73>create_default_context<78>time<6D>urlopen<65>loads<64>read<61>get<65>round)r
r r r rr<00>urllibr'<00>url<72>payload<61>req<65>ctx<74>start<72>resp<73>resultr$r<00>eval_duration_nsr#s <20>:/private/tmp/turboquant-burn/benchmarks/run_test_matrix.py<70>ollama_generater?#s[<00><00><1F> <15>_<EFBFBD>_<EFBFBD>S<EFBFBD> !<21> "<22>-<2D>
0<EFBFBD>C<EFBFBD><12>j<EFBFBD>j<EFBFBD><16><18><17>&<26><1E>
<EFBFBD> <06><07><0E>v<EFBFBD>x<EFBFBD> <0C> <11>.<2E>.<2E>
<20>
<20><13>7<EFBFBD>*8<>:L<>)M<>(.<2E> !<21> 0<>C<EFBFBD> <0E>
$<24>
$<24>
&<26>C<EFBFBD> <10>I<EFBFBD>I<EFBFBD>K<EFBFBD>E<EFBFBD> <11>><3E>><3E> !<21> !<21>#<23>w<EFBFBD><03> !<21> D<>D<EFBFBD> <11>Z<EFBFBD>Z<EFBFBD><04> <09> <09> <0B> $<24>F<EFBFBD><14> <09> <09> <0B>e<EFBFBD>#<23>I<EFBFBD><17><1A><1A>L<EFBFBD>!<21>,<2C>J<EFBFBD><1D>z<EFBFBD>z<EFBFBD>/<2F>1<EFBFBD>5<><14>5E<35><01>5I<35>J<EFBFBD>*<2A>S<EFBFBD>0<> 1<>q<EFBFBD>E<EFBFBD><1A>J<EFBFBD>J<EFBFBD>z<EFBFBD>2<EFBFBD>.<2E><16>u<EFBFBD>a<EFBFBD><1F><1A>9<EFBFBD>a<EFBFBD>(<28> <20>#<23>Z<EFBFBD>Z<EFBFBD>(;<3B>Q<EFBFBD>?<3F>#<23>Z<EFBFBD>Z<EFBFBD>(8<>!<21><<3C>  <06><06><00> prompts_pathc <00><><00>t|<02>5}tj|<03>}ddd<01>g}D<00>]}td|d<00>d|d<00>d<06>dd<08> <09> t |d
||d <0B> <0C>}|d }|j dd<0F>} | r.t tj| |tj<00><00>nd}
d|vrQt d|d
<00>d|d<00><00>||d<13> <0C>} t tj|d| d <00><00>} |
xr| }
|d| d zz }|j|d|d|d
dd|
|dt|<08>d<17><06>|
rdnd} t| <0A>d|d<00>d<1B><04><00><01>td<1F>|D<00><00>}t|<05>||rt|t|<05>z d <20>|d"<22>Sd!|d"<22>S#1swY<00><01>jxYw#t$r>}|j|d|ddt|<0E>d<1D><04>td|<0E><00><02>Yd}~<0E><01><>d}~wwxYw)#z0Run 10 test prompts and check expected patterns.Nz [<5B>idz/10] <20>category<72>...<2E> T<><02>end<6E>flushr
<00><00>r r<00>expected_patternr<00> follow_upzPrevious context: User said 'z' and you responded.
User: <20>z
---FOLLOW-UP---
<EFBFBD>dr#)rCrDr
<00>pattern_matchedr#<00> response_len<65>PASS<53>FAIL<49> (<28> tok/s)F)rCrDrP<00>error<6F>ERROR: c3<00>FK<00>|]}|jdd<01>s<01>d<02><01><00>y<03>w)rPFrN<>r3<00><02>.0<EFBFBD>rs r><00> <genexpr>z&run_quality_prompts.<locals>.<genexpr>us<00><00><><00>G<>q<EFBFBD>q<EFBFBD>u<EFBFBD>u<EFBFBD>-><3E><05>'F<><11>G<><47><00>!<01>!r r)<04>total<61>passed<65> pass_rate<74>details)<10>openr)<00>load<61>printr?r3<00>bool<6F>re<72>search<63>DOTALL<4C>append<6E>len<65> Exception<6F>str<74>sumr4)r r rA<00>f<>prompts<74>results<74>pr\r<00>pattern<72>matched<65>follow<6F>follow_matched<65>status<75>er`s r><00>run_quality_promptsryJs7<00><00> <0A>l<EFBFBD> <1B><1F>q<EFBFBD><16>)<29>)<29>A<EFBFBD>,<2C><07><1F><11>G<EFBFBD> <14>#!<21><01> <0A><03>A<EFBFBD>d<EFBFBD>G<EFBFBD>9<EFBFBD>E<EFBFBD>!<21>J<EFBFBD>-<2D><1F><03>4<>#<23>T<EFBFBD>J<>! !<21><1F><01>(<28> <0B>U<EFBFBD>H<EFBFBD>#<23>N<>A<EFBFBD><18><1A>}<7D>H<EFBFBD><17>e<EFBFBD>e<EFBFBD>.<2E><02>3<>G<EFBFBD>GN<47>d<EFBFBD>2<EFBFBD>9<EFBFBD>9<EFBFBD>W<EFBFBD>h<EFBFBD><02> <09> <09>B<>C<>TX<54>G<EFBFBD><1B>a<EFBFBD><1F>(<28>3<>A<EFBFBD>h<EFBFBD>K<EFBFBD>=<3D>@^<5E>_`<60>al<61>_m<5F>^n<>o<><19>8<EFBFBD><13><12><06>"&<26>b<EFBFBD>i<EFBFBD>i<EFBFBD><01>2D<32>0E<30>v<EFBFBD>j<EFBFBD>GY<47>&Z<>![<5B><0E>!<21>4<>n<EFBFBD><07><18>1<>F<EFBFBD>:<3A>4F<34>F<>F<><08> <13>N<EFBFBD>N<EFBFBD><17><04>g<EFBFBD><1D>j<EFBFBD>M<EFBFBD><1B>H<EFBFBD>+<2B>d<EFBFBD>s<EFBFBD>+<2B>#*<2A><1A>7<EFBFBD><1A> #<23>H<EFBFBD> <0A> <0E> <0F> '<27>V<EFBFBD>F<EFBFBD>F<EFBFBD> <11>V<EFBFBD>H<EFBFBD>B<EFBFBD>q<EFBFBD><17>z<EFBFBD>l<EFBFBD>'<27>2<> 3<>7#!<21>J<11>G<>G<EFBFBD>G<> G<>F<EFBFBD><14>W<EFBFBD><1C><18>8?<3F>U<EFBFBD>6<EFBFBD>C<EFBFBD><07>L<EFBFBD>0<>!<21>4<><1A>  <06><06>FG<01><1A>  <06><06>U<1F><1F><>@<19> !<21> <13>N<EFBFBD>N<EFBFBD><17><04>g<EFBFBD><1D>j<EFBFBD>M<EFBFBD>#(<28><1C>Q<EFBFBD><16> <0E> <0F> <12>G<EFBFBD>A<EFBFBD>3<EFBFBD>-<2D> <20> <20><> !<21>s$<00>F
<03>C:F<02>
F<07> G<05> 3G<05>G<05> corpus_pathc<00><><00>tjj|<02>sd|<02><00>dd<03>St|<02>5}|j <00>dd}ddd<04>t dt <04>d<07>D<00>cgc]
}|||dz<00><02> }}|dd}d}d} |D](}
td |
dd
<00><00>||d <0B> <0C>} || d z }| | dz } <09>*| dkDr|| z nd} |t |<07>t| d<0F>ddd<12>S#1swY<00><>xYwcc}w#t$rY<00>nwxYw)z4Estimate perplexity by scoring the corpus in chunks.zCorpus not found: F)rVr`NiP<69>ri<><00>
zContinue this text naturally:
i<EFBFBD>rOrKrr$ru2Proxy metric — real PPL requires logprob supportT)<05>corpus<75> chunks_scored<65> avg_tok_s<5F>noter`)
<EFBFBD>os<6F>path<74>existsrcr2<00>rangerkr?rlr4) r r rzro<00>text<78>i<>chunks<6B> sample_chunks<6B> total_tokens<6E>
total_time<EFBFBD>chunkr\rs r><00>run_perplexityr<79><00>s<<00><00> <0A>7<EFBFBD>7<EFBFBD>><3E>><3E>+<2B> &<26>-<2D>k<EFBFBD>]<5D>;<3B>u<EFBFBD>M<>M<> <0A>k<EFBFBD> <1A> <20>a<EFBFBD><10>v<EFBFBD>v<EFBFBD>x<EFBFBD><06><15><1F><04> <20>',<2C>A<EFBFBD>s<EFBFBD>4<EFBFBD>y<EFBFBD>$<24>&?<3F> @<40><11>d<EFBFBD>1<EFBFBD>Q<EFBFBD>t<EFBFBD>V<EFBFBD>n<EFBFBD> @<40>F<EFBFBD> @<40><1A>3<EFBFBD>B<EFBFBD>K<EFBFBD>M<EFBFBD><14>L<EFBFBD><12>J<EFBFBD><1E><11><05> <11><1F>"C<>E<EFBFBD>$<24>3<EFBFBD>K<EFBFBD>=<3D> Q<>$<24>h<EFBFBD>C<EFBFBD>A<01>A<EFBFBD> <18>A<EFBFBD>l<EFBFBD>O<EFBFBD> +<2B>L<EFBFBD> <16>!<21>K<EFBFBD>.<2E> (<28>J<EFBFBD> <11>.8<EFBFBD>!<21>^<5E> <0C>z<EFBFBD>)<29><11>I<EFBFBD><1E><1C>]<5D>+<2B><1A>9<EFBFBD>a<EFBFBD>(<28>D<><16>  <06><06>- <20> <20><>A<01><><19> <11> <10> <11>s#<00>C<03>'C!<04>%C&<02>C<07>& C2<05>1C2c <00><00>d}dddd<05>}i}|j<00>D]<5D>\}}td|<05>d<07>dd <09>
<EFBFBD>d |d zz}|d |dz<00>d|<02>d|d |dz<00><00>} td|d |<00><00>||d|d<12><13>} d| dvxsd| dj<00>v}
|
| d| dd dd<19>||<t|
rdnd<1B><00><>t d<1F>|j<00>D<00><00>} t|<04>| |d <20>S#t$r)} dt | <0B>d<1D>||<td| <0B><00><02>Yd } ~ <0B><>d } ~ wwxYw)!z*Test retrieval at various context lengths.z#The secret code is TURBOQUANT-7742.<2E> <00>@<00><00>)<03>8K<38>16K<36>32Kz Needle-in-Haystack rErFTrGz-The quick brown fox jumps over the lazy dog. <20>2Nr <00>
z*Read this text and find the secret code:
<EFBFBD>@i,<00>r rrzTURBOQUANT-7742r<00>
turboquantr#rO)<03> retrievedr#<00>response_excerptrRrSF)r<>rVrWc3<00>FK<00>|]}|jdd<01>s<01>d<02><01><00>y<03>w)r<>FrNrYrZs r>r]z)run_needle_in_haystack.<locals>.<genexpr><3E>s<00><00><><00>J<>q<EFBFBD><01><05><05>k<EFBFBD>5<EFBFBD>0I<30><11>J<>r^)r_r`rb) <09>itemsrer?<00>lowerrlrmrn<00>valuesrk) r r <00>needle<6C>contextsrq<00>label<65>ctx_len<65>filler<65>haystackr\<00>foundrxr`s r><00>run_needle_in_haystackr<6B><00>sv<00><00> 2<>F<EFBFBD><12><14><14><06>H<EFBFBD> <11>G<EFBFBD>"<22>.<2E>.<2E>*<2A>!<21><0E><05>w<EFBFBD> <0A>%<25>e<EFBFBD>W<EFBFBD>C<EFBFBD>0<>c<EFBFBD><14>F<>@<40>G<EFBFBD>r<EFBFBD>M<EFBFBD>R<><06><1C>[<5B>g<EFBFBD>q<EFBFBD>j<EFBFBD>)<29>*<2A>"<22>V<EFBFBD>H<EFBFBD>B<EFBFBD>v<EFBFBD>k<EFBFBD>w<EFBFBD><01>z<EFBFBD>7J<37>6K<36>L<><08> !<21><1F>><3E>x<EFBFBD><08><17>?Q<>>R<>S<><15>x<EFBFBD><1E><1F><1B> <0E>A<EFBFBD>&<26><11>:<3A><1D>6<>_<>,<2C>!<21>J<EFBFBD>-<2D>J]<5D>J]<5D>J_<4A>:_<>E<EFBFBD>"<22><1A>7<EFBFBD><1A>$%<25>j<EFBFBD>M<EFBFBD>$<24>3<EFBFBD>$7<><0E>G<EFBFBD>E<EFBFBD>N<EFBFBD>
<12>E<EFBFBD>&<26>v<EFBFBD> .<2E>)!<21>2<11>J<>G<EFBFBD>N<EFBFBD>N<EFBFBD>,<2C>J<> J<>F<EFBFBD><14>W<EFBFBD><1C><18><1A> <06><06><> <19> !<21>+0<>3<EFBFBD>q<EFBFBD>6<EFBFBD>B<>G<EFBFBD>E<EFBFBD>N<EFBFBD> <11>G<EFBFBD>A<EFBFBD>3<EFBFBD>-<2D> <20> <20><> !<21>s<00>AC<02> D <05>"D<05>D c
<00><><00>d}i}dD]s\}}td|<04>d<04>dd<06><07> tj<00>}t|||d|<05> <09>}|d
}|d t|d <0C>|d |dd<0F>||<t|d <00>d|d<11>d<12><04><00>u t jdk(r_tjddddt tj<00><00>gdd<06><1B>}
t|
jj<00><00>dz } nd} |t| d<1E>d<1F>S#t$r)} dt | <09>i||<td| <09><00><02>Yd} ~ <09><01>'d} ~ wwxYw#t$rd} Y<00>QwxYw) zAMeasure tok/s, TTFT proxy, and memory at different context sizes.zaExplain the concept of KV cache quantization in large language models. Be technical and detailed.))<02>4K<34>)r<>r<>)r<>r<>z Performance rErFTrGrNrr$r#r r!r)r#<00>ttft_s<5F> prompt_tokens<6E>generated_tokensz tok/s, TTFT z.2f<EFBFBD>srVrWN<>darwin<69>psz-ozrss=z-p)<02>capture_outputr<74>irr)r<><00>peak_memory_mb)rer/r?r4rlrm<00>sys<79>platform<72>
subprocess<EFBFBD>runr<6E><00>getpid<69>int<6E>stdout<75>strip) r r <00> test_prompt<70>perf<72> ctx_label<65>ctx_sizer:r\<00>ttftrxr<<00>peak_mbs r><00>run_performancer<65><00>so<00><00>u<>K<EFBFBD> <0A>D<EFBFBD>K<>!<21><1B> <09>8<EFBFBD> <0A><0E>y<EFBFBD>k<EFBFBD><13>-<2D>3<EFBFBD>d<EFBFBD>C<> !<21><18>I<EFBFBD>I<EFBFBD>K<EFBFBD>E<EFBFBD><1F> <0B>U<EFBFBD>H<EFBFBD>+.<2E><08>B<01>A<EFBFBD><14>[<5B>><3E>D<EFBFBD><1B>7<EFBFBD><1A><1F><04>a<EFBFBD>.<2E>!"<22>#6<>!7<>$%<25>l<EFBFBD>O<EFBFBD> <0E>D<EFBFBD><19>O<EFBFBD> <12>Q<EFBFBD>w<EFBFBD>Z<EFBFBD>L<EFBFBD> <0A>d<EFBFBD>3<EFBFBD>Z<EFBFBD>q<EFBFBD>9<> :<3A>!<21>*<14> <0E><<3C><<3C>8<EFBFBD> #<23><1F>^<5E>^<5E>T<EFBFBD>4<EFBFBD><16><14>s<EFBFBD>2<EFBFBD>9<EFBFBD>9<EFBFBD>;<3B>?O<>$P<>26<32>T<EFBFBD>C<01>F<EFBFBD><19>&<26>-<2D>-<2D>-<2D>-<2D>/<2F>0<>4<EFBFBD>7<>G<EFBFBD><17>G<EFBFBD>
<19><1F><07><11>+<2B> <06><06><><19> !<21>&<26><03>A<EFBFBD><06>/<2F>D<EFBFBD><19>O<EFBFBD> <11>G<EFBFBD>A<EFBFBD>3<EFBFBD>-<2D> <20> <20><> !<21><> <15><14><13><07><14>s+<00>AD<02>>A4D6<00> D3<05>
D.<05>.D3<05>6 E<03>Ec <00><><00>d}gd<02>}d}|D]:}td|<05>d<05>dd<07><08> t|||d |d
<EFBFBD> <0B>}|}td |d <00>d<0E><03><00><|d|dk\|d<12>S#t$r}td|<07><00><02>Yd}~<00>'d}~wwxYw)z0Binary search for max context length before OOM.a<> Summarize: word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word )r<>r<>r<>r<>rz Context ceiling rErFTrG<00> <00>xr<>zOK (r#rUzFAIL: N<>)<04> max_context<78>minimum_requiredr`<00>tested)rer?rl)r r r<><00> test_contexts<74> max_workingr9r\rxs r><00>run_context_ceilingr<67>s<><00><00>/<2F>K<EFBFBD>.<2E>M<EFBFBD><13>K<EFBFBD><1C> <12><03> <0A>"<22>3<EFBFBD>%<25>s<EFBFBD>+<2B><13>D<EFBFBD>A<> <12><1F> <0B>U<EFBFBD>H<EFBFBD>+-<2D>s<EFBFBD>C<EFBFBD>I<01>A<EFBFBD><1D>K<EFBFBD> <11>D<EFBFBD><11>7<EFBFBD><1A> <0C>G<EFBFBD>,<2C> -<2D> <12>#<23>!<21><1D><15>&<26><1F>  <06><06><> <19> <12> <11>F<EFBFBD>1<EFBFBD>#<23>,<2C> <1F> <11><> <12>s<00>%A<02> A4<05>A/<05>/A4<05>quality<74>
perplexityr<EFBFBD><00> performancer<00> timestampc<00><><00>|||||||d<01>}d}g} |jdd<04>dkr'd}| jd|jdd<04><00>d <09><03>|jdd<04>|jd
d<04>k(s7| jd |jdd<04><00>d |jd
d<04><00><00><04>|jd d<04>dkr%| jd|jd d<04><00>d<10><03>|r| sdnd|d<| |d<d|<06>d|<05>d|d<00>d| r!td<19>jd<1A>| D<00><00>nd<1B>d<1C> }
|jdg<00>D]Y} | jd<1E>rdnd } |
d!| jd"d#<23><00>d$| jd%d#<23><00>d$| <0C>d$| jd&d#<23><00>d'<27> z }
<EFBFBD>[|
d(|jdd<04><00>d |jd
d<04><00>d)|jdd<04>d*zd+<2B>d,<2C>z }
|
d-|jd.d/<2F><00>d0|jd1d/<2F><00>d2|jd3d#<23><00>d4<64>z }
|jdi<00>j <00>D]7\} }|
d!| <0A>d$|jd5<64>rdnd <20>d$|jd&d#<23><00>d'<27>z }
<EFBFBD>9|
d6|jdd<04><00>d |jd
d<04><00>d7<64>z }
|
d8z }
|jd9i<00>j <00>D]Z\} }|
d!| <0A>d$|jd&d#<23><00>d$|jd:d#<23><00>d$|jd;d#<23><00>d$|jd<d#<23><00>d'<27> z }
<EFBFBD>\|
d=|jd>d/<2F><00>d?<3F>z }
|
d@|jd d/<2F><00>dA|jd<08>rdBndC<64>dD<64>z }
||
fS)Ez Generate JSON + Markdown report.)r<>r r<>r<><00>needle_in_haystackr<6B><00>context_ceilingTrarg<><67><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?F<> Quality: r`z/10 passed (need >=9)r_zNeedle-in-Haystack: rr<>r<><00>Context ceiling: z < 64K required<65>GOzNO-GO<47>go_no_go<67>issuesz0# TurboQuant Test Matrix Report
**Generated:** z
**Model:** z
## Go/No-Go: z
r|c3<00>&K<00>|] }d|z<00><01><00> y<01>w)z- N<>)r[r<>s r>r]z"generate_report.<locals>.<genexpr>Ds<00><00><><00> '<27>1<EFBFBD>d<EFBFBD>Q<EFBFBD>h<EFBFBD> '<27>s<00>zAll criteria met.zv
## Quality (10 Practical Prompts)
| # | Category | Pattern Match | tok/s |
|---|----------|--------------|-------|
rbrPrRrSz| rCrz | rDr#z |
z
**Pass rate:** rTrOz.0fz%)
z!
## Perplexity
- Chunks scored: r~zN/Az
- Avg tok/s: rz
- Note: r<>zY
## Needle-in-Haystack
| Context | Retrieved | tok/s |
|---------|-----------|-------|
r<EFBFBD>z
**Retrieved:** r<>z<>
## Performance
| Context | tok/s | TTFT (s) | Prompt Tokens | Generated |
|---------|-------|----------|---------------|-----------|
r<EFBFBD>r<>r<>r<>z
Peak memory: r<>z MB
z,
## Context Ceiling
- Max working context: z%
- Minimum required: 65536
- Passed: <20>YES<45>NOz3
---
*Generated by run_test_matrix.py. Ref: #11.*
)r3rj<00>chr<68>joinr<6E>)r<>r<>r<>r<>rr r<><00>report<72>gor<6F><00>mdr\rwr<><00>detailr<6C>s r><00>generate_reportr<74>sL<00><00> <1F><16><1A> <20>$<24>"<22>"<22><06>F<EFBFBD>
<0E>B<EFBFBD> <0F>F<EFBFBD><0E>{<7B>{<7B>;<3B><01>"<22>S<EFBFBD>(<28> <12><02><0E> <0A> <0A> <09>'<27>+<2B>+<2B>h<EFBFBD><01>":<3A>!;<3B>;P<>Q<>R<> <11>:<3A>:<3A>h<EFBFBD><01> "<22>f<EFBFBD>j<EFBFBD>j<EFBFBD><17>!<21>&<<3C> <<3C><0E> <0A> <0A>,<2C>V<EFBFBD>Z<EFBFBD>Z<EFBFBD><08>!<21>-D<>,E<>Q<EFBFBD>v<EFBFBD>z<EFBFBD>z<EFBFBD>RY<52>[\<5C>G]<5D>F^<5E>_<>`<60><0E>{<7B>{<7B>=<3D>!<21>$<24>u<EFBFBD>,<2C><0E> <0A> <0A>)<29>'<27>+<2B>+<2B>m<EFBFBD>Q<EFBFBD>*G<>)H<><0F>X<>Y<>!#<23>F<EFBFBD><14><07>F<EFBFBD>:<3A><16><1D>F<EFBFBD>8<EFBFBD><14><10><19>{<7B> <0C> <11>7<EFBFBD><0E><14>Z<EFBFBD> <20> !<21>"<01>+1<><13>R<EFBFBD><17><1C><1C> '<27><06> '<27>'<27>7J<37>K<>L<01>
<04>B<EFBFBD><15>[<5B>[<5B><19>B<EFBFBD> '<27>b<01><01><1C>5<EFBFBD>5<EFBFBD>!2<>3<><16><16><06>
<EFBFBD><02>1<EFBFBD>5<EFBFBD>5<EFBFBD><14>b<EFBFBD>><3E>"<22>#<23>a<EFBFBD>e<EFBFBD>e<EFBFBD>J<EFBFBD>r<EFBFBD>&:<3A>%;<3B>3<EFBFBD>v<EFBFBD>h<EFBFBD>c<EFBFBD>!<21>%<25>%<25>PW<50>XZ<58>J[<5B>I\<5C>\`<60>a<>a<><02>b<01><07> <1D>g<EFBFBD>k<EFBFBD>k<EFBFBD>(<28>1<EFBFBD>5<>6<>a<EFBFBD><07> <0B> <0B>G<EFBFBD>A<EFBFBD>8N<38>7O<37>r<EFBFBD>RY<52>R]<5D>R]<5D>^i<>jk<6A>Rl<52>mp<6D>Rp<52>qt<71>Qu<51>uy<75>
z<EFBFBD>z<>B<EFBFBD><06><12><1D>.<2E>.<2E><1F>%<25>8<>9<>:<0E><18>n<EFBFBD>n<EFBFBD>[<5B>%<25>0<> 1<>2 <09> <13><1E><1E><06><02> #<23>$<24>%<01> <04> <04>B<EFBFBD> <20><1A><1A>I<EFBFBD>r<EFBFBD>2<>8<>8<>:<3A>l<01> <0A><05>v<EFBFBD>
<EFBFBD><02>5<EFBFBD>'<27><13>v<EFBFBD>z<EFBFBD>z<EFBFBD>+<2B>'><3E>V<EFBFBD>F<EFBFBD>K<>3<EFBFBD>v<EFBFBD>z<EFBFBD>z<EFBFBD>Za<5A>bd<62>Oe<4F>Nf<4E>fj<66>k<>k<><02>l<01><07> <1D>f<EFBFBD>j<EFBFBD>j<EFBFBD><18>!<21>4<>5<>Q<EFBFBD>v<EFBFBD>z<EFBFBD>z<EFBFBD>'<27>!<21>7L<37>6M<36>R<EFBFBD>
P<EFBFBD>P<>B<EFBFBD><06><01><04>B<EFBFBD> #<23><EFBFBD><EFBFBD>z<EFBFBD>2<EFBFBD>6<><<3C><<3C>><3E>W<02> <0B><05>t<EFBFBD>
<EFBFBD><02>5<EFBFBD>'<27><13>T<EFBFBD>X<EFBFBD>X<EFBFBD>g<EFBFBD>b<EFBFBD>1<>2<>#<23>d<EFBFBD>h<EFBFBD>h<EFBFBD>x<EFBFBD><02>6K<36>5L<35>C<EFBFBD>PT<50>PX<50>PX<50>Yh<59>ik<69>Pl<50>Om<4F>mp<6D>qu<71>qy<71>qy<71>{M<02>NP<02>rQ<02>qR<02>RV<02>W<02> W<02><02>W<02><07>O<EFBFBD>K<EFBFBD>O<EFBFBD>O<EFBFBD>,<<3C>e<EFBFBD>D<>E<>U<EFBFBD>
K<EFBFBD>K<>B<EFBFBD><06><18> <20> <0B> <0B>M<EFBFBD>5<EFBFBD>9<>:<3A>; <0B><1B>K<EFBFBD>K<EFBFBD><08>)<29>5<EFBFBD>t<EFBFBD>
4<EFBFBD>5<01> <04> <04>B<EFBFBD> <12>2<EFBFBD>:<3A>r@c <00><><00>tjd<01><02>}|jdd<04><05>|jdd<07><05>|jdd <09><05>|jd
d <0B><05>|jd d <0A><05>|jdd<0F><10>|jdd<0F><10>|j<00>}t j
t j<00>jd<12>}t j
<00>jd<13>}td<14>td|j<00><00><02>td|j<00><00><02>td|<02><00><02>t<00>i}i}i}i}i}|js<>td<18>t|j|j|j<00>}t<00>td<19>t|j|j|j <00>}t<00>td<1A>t#|j|j<00>}t<00>|j$sjtd<1B>t'|j|j<00>}t<00>td<1C>t)|j|j<00>}t<00>t+||||||j|<02>\} }
t-j.|j0d<1D><1E>t,j2j5|j0d|<03>d <20><03>} t,j2j5|j0d|<03>d!<21><03>} t7| d"<22>5} t9j:| | d#<23>$<24>ddd<00>t7| d"<22>5} | j=|
<EFBFBD>ddd<00>td%<25>td&| d'<00><00><02>td(|j?d)d*<2A><00>d+|j?d,d*<2A><00><00><04>td-|j?d)d*<2A><00>d+|j?d,d*<2A><00><00><04>td.|j?d/d*<2A><00><00><02>td0| <0B>d1| <0C><00><04>y#1swY<00><>xYw#1swY<00><>xYw)2NzTurboQuant Full Test Matrix)<01> descriptionz--modelz
qwen2.5:7b)<01>defaultz
--base-urlzhttp://localhost:11434z --promptszbenchmarks/test_prompts.jsonz--corpuszcorpora/wiki.test.rawz --output-dir<69>reportsz--skip-quality<74>
store_true)<01>actionz--skip-performancez%Y-%m-%dT%H:%M:%SZz%Y-%m-%dz=== TurboQuant Test Matrix ===zModel: z Backend: zTime: u&[1/5] Quality — 10 Practical Promptsu%[2/5] Perplexity — WikiText-2 proxyz[3/5] Needle-in-Haystacku)[4/5] Performance — tok/s, TTFT, memoryz[5/5] Context CeilingT)<01>exist_okz test-matrix-z.jsonz.md<6D>wr )<01>indentz=== Results ===z
Go/No-Go: r<>r<>r`rrr_zNeedle: r<>r<>z Reports: z, ) <20>argparse<73>ArgumentParser<65> add_argument<6E>
parse_argsr<00>nowr<00>utc<74>strftimerer r <00> skip_qualityryrpr<>r}r<><00>skip_performancer<65>r<>r<>r<><00>makedirs<72>
output_dirr<EFBFBD>r<>rcr)<00>dump<6D>writer3)<0E>parser<65>argsr<73><00>date_strr<72>r<>r<>r<>rr<>r<><00> json_path<74>md_pathros r><00>mainr<6E>}s<><00><00> <15> $<24> $<24>1N<31> O<>F<EFBFBD>
<EFBFBD><17><17> <09><<3C><17>8<>
<EFBFBD><17><17> <0C>.F<><17>G<>
<EFBFBD><17><17> <0B>-K<><17>L<>
<EFBFBD><17><17>
<EFBFBD>,C<><17>D<>
<EFBFBD><17><17><0E> <09><17>:<3A>
<EFBFBD><17><17>(<28><1C><17>><3E>
<EFBFBD><17><17>,<2C>\<5C><17>B<> <11> <1C> <1C> <1E>D<EFBFBD><18> <0C> <0C>X<EFBFBD>\<5C>\<5C>*<2A>3<>3<>4H<34>I<>I<EFBFBD><17>|<7C>|<7C>~<7E>&<26>&<26>z<EFBFBD>2<>H<EFBFBD> <09> *<2A>,<2C> <09>G<EFBFBD>D<EFBFBD>J<EFBFBD>J<EFBFBD><<3C>
<20>!<21> <09>I<EFBFBD>d<EFBFBD>m<EFBFBD>m<EFBFBD>_<EFBFBD>
%<25>&<26> <09>F<EFBFBD>9<EFBFBD>+<2B>
<1E><1F> <09>G<EFBFBD><10>G<EFBFBD><13>J<EFBFBD> <0F>F<EFBFBD><14>K<EFBFBD><10>G<EFBFBD> <0F> <1C> <1C> <0A>6<>7<>%<25>d<EFBFBD>j<EFBFBD>j<EFBFBD>$<24>-<2D>-<2D><14><1C><1C>N<><07> <0A><07> <0A>5<>6<>#<23>D<EFBFBD>J<EFBFBD>J<EFBFBD><04> <0A> <0A>t<EFBFBD>{<7B>{<7B>K<>
<EFBFBD> <0A><07> <0A>(<28>)<29>'<27><04>
<EFBFBD>
<EFBFBD>D<EFBFBD>M<EFBFBD>M<EFBFBD>B<><06> <0A><07> <0F> <20> <20> <0A>9<>:<3A>%<25>d<EFBFBD>j<EFBFBD>j<EFBFBD>$<24>-<2D>-<2D>@<40> <0B> <0A><07> <0A>%<25>&<26>%<25>d<EFBFBD>j<EFBFBD>j<EFBFBD>$<24>-<2D>-<2D>@<40><07> <0A><07>!<21><17>*<2A>f<EFBFBD>k<EFBFBD>7<EFBFBD>"&<26>*<2A>*<2A>i<EFBFBD>9<>J<EFBFBD>F<EFBFBD>B<EFBFBD><07>K<EFBFBD>K<EFBFBD><04><0F><0F>$<24>/<2F><12><07><07> <0C> <0C>T<EFBFBD>_<EFBFBD>_<EFBFBD> <0C>X<EFBFBD>J<EFBFBD>e<EFBFBD>.L<>M<>I<EFBFBD><10>g<EFBFBD>g<EFBFBD>l<EFBFBD>l<EFBFBD>4<EFBFBD>?<3F>?<3F>l<EFBFBD>8<EFBFBD>*<2A>C<EFBFBD>,H<>I<>G<EFBFBD> <0A>i<EFBFBD><13> <1D>'<27><11> <0C> <09> <09>&<26>!<21>A<EFBFBD>&<26>'<27> <0A>g<EFBFBD>s<EFBFBD> <1B><14>q<EFBFBD> <09><07><07><02> <0B><14>
<EFBFBD>O<EFBFBD><1D> <09>J<EFBFBD>v<EFBFBD>j<EFBFBD>)<29>*<2A>
+<2B>,<2C> <09>I<EFBFBD>g<EFBFBD>k<EFBFBD>k<EFBFBD>(<28>A<EFBFBD>.<2E>/<2F>q<EFBFBD><17><1B><1B>W<EFBFBD>a<EFBFBD>1H<31>0I<30>
J<EFBFBD>K<> <09>H<EFBFBD>V<EFBFBD>Z<EFBFBD>Z<EFBFBD><08>!<21>,<2C>-<2D>Q<EFBFBD>v<EFBFBD>z<EFBFBD>z<EFBFBD>'<27>1<EFBFBD>/E<>.F<>
G<EFBFBD>H<> <09> <1D>g<EFBFBD>k<EFBFBD>k<EFBFBD>-<2D><11>;<3B><<3C>
=<3D>><3E> <09>I<EFBFBD>i<EFBFBD>[<5B><02>7<EFBFBD>)<29>
,<2C>-<2D>'<27>'<27><><14><14>s<00><P1<03>)P=<03>1P:<07>=Q<07>__main__)rJi<00><>)<1D>__doc__r<5F>r)r<>rgr<>r<>r/rr<00>pathlibr<00>typingrrrr rmr<><00>dictr?ryr<>r<>r<>r<>r<>r<><00>__name__r<5F>r@r><00><module>rsu<00><01><04>$<10> <0B> <09> <09><11>
<EFBFBD> <0B>'<27><18>.<2E>.<2E><@<01>#&<26>!<06>C<EFBFBD>!<06><03>!<06>s<EFBFBD>!<06>!$<24>!<06>58<35>!<06> <20>!<06>+/<2F>!<06>N1<06>s<EFBFBD>1<06>c<EFBFBD>1<06><13>1<06><14>1<06>n!<06>#<23>!<06><13>!<06>3<EFBFBD>!<06>4<EFBFBD>!<06>N(<06>#<23>(<06><13>(<06><14>(<06>\'<06>3<EFBFBD>'<06>#<23>'<06>$<24>'<06>Z<06>s<EFBFBD><06>c<EFBFBD><06>d<EFBFBD><06>8X<16>T<EFBFBD>X<16>t<EFBFBD>X<16>T<EFBFBD>X<16>!%<25>X<16>04<30>X<16><1E>X<16>+.<2E>X<16>38<33><14>s<EFBFBD><19>3C<33>X<16>|B.<2E>J <0C>z<EFBFBD><19><08>F<EFBFBD>r@
Reference in New Issue Copy Permalink