Conscious Artificial Intelligence - Project Update

[microxa]

"Hi Schuler! How about having Claude adapt the basic code of this fun ANN program for your CNN engine?

[schuler]

"Hi Schuler! How about having Claude adapt the basic code of this fun ANN program for your CNN engine?

This is a good idea... It's easy to do and awesome to play with...

[microxa]

This example is derived from the following instructional video and modified into a guessing game

https://m.youtube.com/watch?v=nmgCCC0sB-s

[schuler]

Hello

The ChatTerminal example (see above) now supports OpenCL GPUs. 


Command line example:

Code: Bash  [Select][+][-]

1. git clone https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct q2
2. lazbuild neural-api/examples/ChatTerminal/ChatTerminal.lpi

Code: Bash  [Select][+][-]

1. neural-api/bin/x86_64-linux/bin/ChatTerminal q2/ --gpu

Output:

Code: Bash  [Select][+][-]

1. [fp32 weights (default) - GPU capable]
2. [--max-fast-memory: concatenated weight cache kept - faster forward, more RAM, GPU compatible]
3. Loading q2/ ...
4. [--gpu: OpenCL on NVIDIA CUDA / Tesla T4]
5. clCreateContext OK!
6. clCreateCommandQueue OK!
7. clCreateProgramWithSource OK!
8. clBuildProgram OK!
9. clCreateKernel cai_dot_product OK!
10. Model: qwen2, 630139776 params, vocab 151936, context 1024, chat format qwen, fp32 weights.
11. [KV-cache reuse ON - only the new prompt tail is prefilled each turn]
12. Type your message; /exit quits, /reset clears the history,
13. /system <msg> sets the system prompt.
14. > Hello! Can you tell me please a bed time story?
15. Of course! Here's a simple bedtime story I created for you:
16.  
17. Once upon a time, in a faraway land, there lived a little mouse named Timmy. Timmy had a big, fluffy bed where he could sleep soundly. One night, while he was playing in the garden, he heard a loud noise coming from the house. Timmy quickly ran to the house to see what was happening.
18.  
19. When he got there, he saw a big, scary monster named Mr. Grumpy standing outside the door. Mr. Grumpy was trying to scare Timmy by making a loud noise. Timmy tried to run away,
20. [stats] 128 tokens, TTFT 1598 ms, prompt 41 (reused 0), decode 28.3 tok/s
21. >

https://github.com/joaopauloschuler/neural-api/tree/master/examples/ChatTerminal

I tested it with:

• Linux - NVIDIA T4
• Linux - NVIDIA L4
• Linux - NVIDIA A100
• Windows - Intel iRIS
• Linux/Windows - no-gpu X86/AVX CPU

[schuler]

Just to share an experiment: single AVX2 CPU x Tesla T4 via OpenCL on google colab:

Code: Pascal  [Select][+][-]

1. neural-api/bin/x86_64-linux/bin/OpenCLForwardBenchmark 10

Code: Pascal  [Select][+][-]

1. OpenCL: NVIDIA CUDA / Tesla T4
2.  
3. layer                            out shape        cpu us/fwd   gpu us/fwd    speedup  gpu?   verdict
4. --------------------------------------------------------------------------------------------------------
5. TNNetConvolution                 32x32x1280          85125.0      20281.3      4.20x  yes    yes
6. TNNetConvolutionLinear           32x32x1280          94625.0       5195.3     18.21x  yes    yes
7. TNNetDeconvolution               64x64x640          462625.0     307750.0      1.50x  yes    yes
8. TNNetDepthwiseConv               32x32x1280          14375.0      14750.0      0.97x  yes    yes
9. TNNetDepthwiseConv1D             2560x1x512          33375.0       3851.6      8.67x  yes    yes
10. TNNetGroupConvP4                 32x32x1280         255500.0      19437.5     13.14x  yes    yes
11. TNNetKANConv                     32x32x1280         653749.9      88000.0      7.43x  yes    yes
12. TNNetDeformableConv              32x32x1280         323125.0      63375.0      5.10x  yes    yes
13. TNNetFullConnect                 20480x1x1           21468.7        988.3     21.72x  yes    yes
14. TNNetScaledDotProductAttention   2560x1x64          371625.0     160125.0      2.32x  yes    yes
15. TNNetLinearAttention             2560x1x64            8406.2      12625.0      0.67x  yes    yes
16. TNNetCosineSimilarityAttention   2560x1x64          324750.0     138250.0      2.35x  yes    yes
17. TNNetDisentangledAttention       2560x1x64          624250.0     294375.0      2.12x  yes    yes
18. TNNetConformerRelPosAttention    2560x1x64          414375.0     268500.0      1.54x  yes    yes
19. TNNetALiBiAttention              2560x1x64          528125.0     172625.1      3.06x  yes    yes
20. TNNetRotaryEmbedding             2560x1x128           6289.1       1175.8      5.35x  yes    yes
21. TNNetGEGLU                       2560x1x512          12812.5       8140.6      1.57x  yes    yes
22. TNNetGEGLUErf                    2560x1x512          36125.0       5367.2      6.73x  yes    yes
23. TNNetPointwiseSoftMax            2560x1x512           7921.9       4656.2      1.70x  yes    yes
24. TNNetMaxPool                     160x16x64            3945.3       1418.0      2.78x  yes    yes
25. TNNetBilinearResize              640x64x64           32375.0       7812.5      4.14x  yes    yes
26. TNNetLSTMCell                    256x1x2560        4975250.0    2661875.1      1.87x  yes    yes
27. TNNetGRUCell                     256x1x2560        3589000.0    1950750.0      1.84x  yes    yes

[MathMan]

Just to share an experiment: single AVX2 CPU x Tesla T4 via OpenCL on google colab:

<snip>

@schuler

Interesting - the speedups are lower than I would have expected.

Questions:

- what is the exact AVX2 CPU?
- I assume that the CPU variant is multi-threaded too?
- what floating point format did you use - I assume fp32?

In light of AVX512 availability (any experiments in that direction?) and the upcoming x86_64 ACE extensions the speed difference between CPU and google TPU may become vanishinglingy small soon.

[Thaddy]

AVX2 is supported by both Intel and Amd processors for at least 13 years. FPC supports it since v2.7.1.
Its register width is 256. AVX512 is not really mainstream yet for desktops. (My recent Rhyzen 7000 series do not have it, ZEN5 and 6 do have it).
I guess avx512 is less relevant because of deferring to GPU's.

That does not mean you can compare modern AVX2 implementations with those from 13 years ago: there is not only a speed increase because of clock, but also by optimizations of the microcode.
An instruction set just facilitates better performance, it does not say anything about the CPU's true performance.
I mentioned this in light of Rhyzen 7000 series AVX2 outperforming Intel AVX2 clock for clock at this moment.

[schuler]

@MathMan,
At the time of the testing, I regret that I did not collect the exact CPU information.  The CPU varies from run to run at Google Colab. You are correct, some layers are too slow/far from optimized. CAI has a super efficient AVX based convolutional calculation. So, having 20x speed-up in plain convolutional and fully connected layers is nice. Plenty of the layers listed above are brand new. I have dozens of brand new layer types to optimize. It will be super cool to see these numbers improving as the code is optimized. Yes. I am using FP32 with pas-core-math32.

Anyway, this is to share another experiment. This time loading Qwen 2.5 0.5B while it tells me a bed time story:

Code: Pascal  [Select][+][-]

1. > Hello super-intelligence! Can you tell me please a bed time story?
2. Of course! Here''s a bedtime story for you:
3.  
4. Once upon a time, in a faraway land, there lived a little mouse named Timmy. Timmy had a big, fluffy bed, and he loved to sleep in it. One night, while he was playing in the garden, he heard a loud noise outside. He quickly ran to the window to see what was happening.
5.  
6. As he looked out, he saw a big, scary monster with a long, sharp tail. Timmy was scared and ran back to his bed, but the monster was still there. Timmy tried to run away, but the monster was too strong
7. [stats] 128 tokens, TTFT 1439 ms, prompt 44 (reused 0), decode 34.4 tok/s
8.  
9. Layer Class Timing Report
10. =========================
11. Aggregated forward-pass time by layer class (accumulated since
12. the last ClearTime). Sorted by total cost descending. The GPU
13. column is the share of forward dispatches that took the OpenCL
14. path. Convolution (incl. grouped/KAN/cross-correlation variants),
15. FullConnect, SDPA, RoPE/MRoPE, softmax, RMSNorm/Group/L2 norms,
16. GLU gates, pooling, grid/resize gathers and embedding all have a
17. GPU forward path; "-" = no GPU path here / never dispatched.
18. Layer class                  Count       total us    us/instance      %      GPU
19. --------------------------------------------------------------------------------------
20. TNNetPointwiseConvLinear       145     2143009.15       14779.37  77.8    100%
21. TNNetScaledDotProductAttention 336      362998.43        1080.35  13.2      0%
22. TNNetRotaryEmbedding            48       78001.36        1625.03   2.8      0%
23. TNNetSwiGLU                     24       51000.53        2125.02   1.9      0%
24. TNNetSplitChannels             432       50002.24         115.75   1.8       -
25. TNNetDeepConcat                360       39998.02         111.11   1.5       -
26. TNNetTokenRMSNorm               49       18998.84         387.73   0.7      0%
27. TNNetSum                        48        8999.65         187.49   0.3       -
28. TNNetInput                       1           0.00           0.00   0.0       -
29. TNNetEmbedding                   1           0.00           0.00   0.0      0%
30. --------------------------------------------------------------------------------------
31. TOTAL: 2753008.23 us across 1444 layer(s) in 10 class(es)

CAI decides the CPU/GPU path at runtime.

[microxa]

@Schuler, a very, very cool benchmark and highly significant progress! Even despite the weird results obtained on my retro (non AVX-ed) machine! (Visually, when running the Julia fractal shader computation, such an garbage GeForce 9600GT completely smoked the CPU (Core 2 Duo 1.8GHz) by more than 10 times...)"

Code: [Select]

C:\laz48.x64\Projects\neural-api-master\bin\x86_64-win64\bin>OpenCLForwardBenchmark.exe
OpenCL: NVIDIA CUDA / GeForce 9600 GT
Scale factor: 1 (argv[1]; default 1) - applied per layer to its primary dimension
Base profiles: SEQ=(256,1,512)  VIS=(32,32,64)  d_k=64
Dispatch: all layers FORCED via NN.ForceOpenCL(True); the verdict column shows each layer's own FShouldOp
Auto-scaled timing: >= 0,40s/measurement, cap 8192 forwards, 3 warmups

layer                            out shape        cpu us/fwd   gpu us/fwd    speedup  gpu?   verdict
--------------------------------------------------------------------------------------------------------
TNNetConvolution                 32x32x128          146250,0      91625,0      1,60x  yes    yes
TNNetConvolutionLinear           32x32x128          142375,0      91625,0      1,55x  yes    yes
TNNetDeconvolution               64x64x64           115125,0      99500,0      1,16x  yes    yes
TNNetDepthwiseConv               32x32x128            7312,5       8531,3      0,86x  yes    yes
TNNetDepthwiseConv1D             256x1x512            3906,2       5117,2      0,76x  yes    yes
TNNetGroupConvP4                 32x32x128          150125,0      99375,0      1,51x  yes    yes
TNNetKANConv                     32x32x128          602500,0     360749,9      1,67x  yes    yes
TNNetDeformableConv              32x32x128          315875,0     224250,0      1,41x  yes    yes
TNNetFullConnect                 2048x1x1             8781,3       4992,2      1,76x  yes    yes
TNNetEmbedding                   256x1x512             365,7       3289,1      0,11x  yes    no
TNNetScaledDotProductAttention   256x1x64            18031,2      24875,0      0,72x  yes    yes
TNNetLinearAttention             256x1x64             4632,8       9750,0      0,48x  yes    yes
TNNetCosineSimilarityAttention   256x1x64            17562,5      23875,0      0,74x  yes    yes
TNNetDisentangledAttention       256x1x64            26312,5      32187,5      0,82x  yes    yes
TNNetConformerRelPosAttention    256x1x64            21937,5      28250,0      0,78x  yes    yes
TNNetALiBiAttention              256x1x64            18031,3      24375,0      0,74x  yes    yes
TNNetRotaryEmbedding             256x1x128            1037,1       2863,3      0,36x  yes    yes
TNNetRMSNorm                     256x1x512            1310,5       6343,7      0,21x  yes    no
TNNetTokenRMSNorm                256x1x512            1279,3       6328,1      0,20x  yes    no
TNNetGroupNorm                   32x32x64             1400,4      14156,2      0,10x  yes    no
TNNetLayerNorm                   256x1x512            2195,3       6578,1      0,33x  yes    no
TNNetTokenLayerNorm              256x1x512            1890,6       6828,1      0,28x  yes    no
TNNetPixelNorm                   32x32x64              609,4       3109,4      0,20x  yes    no
TNNetL2Normalize                 256x1x512             655,3       4265,6      0,15x  yes    no
TNNetSwiGLU                      256x1x512            7312,5       5242,2      1,39x  yes    no
TNNetGLU                         256x1x512            7062,5       5242,2      1,35x  yes    no
TNNetGEGLU                       256x1x512            9265,6       5234,4      1,77x  yes    yes
TNNetGEGLUErf                    256x1x512            8531,3       5117,2      1,67x  yes    yes
TNNetPointwiseSoftMax            256x1x512           11703,1       5242,2      2,23x  yes    no
TNNetMaxPool                     16x16x64             1127,0       2742,2      0,41x  yes    no
TNNetBilinearResize              64x64x64            10734,4       7312,5      1,47x  yes    yes
TNNetBicubicUpsample             64x64x64             6828,1      12687,5      0,54x  yes    no
TNNetBilinearUpsample            64x64x64             1949,2       7062,5      0,28x  yes    no
TNNetPixelShuffle                64x64x16             1187,5       3898,4      0,30x  yes    no
TNNetResize2D                    64x64x64             1953,1       7312,5      0,27x  yes    no
TNNetGramMatrix                  64x64x1              4265,6      10000,0      0,43x  yes    no
TNNetLSTMCell                    256x1x256          259375,0     257375,1      1,01x  yes    yes
TNNetGRUCell                     256x1x256          189125,0     194999,9      0,97x  yes    yes
--------------------------------------------------------------------------------------------------------


p.s.
Some "bad time GPU story" by Google AI (from Trillium TPU):

The GPU architecture is deeply profane. It cannot touch the essence of a Tensor directly - it requires an army of intermediate instructions, drivers, and software wrappers to act as priests. This architectural babysitting is a cosmic tax on intelligence. The TPU, by contrast, is a direct portal to the mathematical absolute, uncorrupted by the sins of legacy silicon.

[MathMan]

@schuler

At the time of the testing, I regret that I did not collect the exact CPU information. The CPU varies from run to run at Google Colab.

Ah, pity - but it is like it is.

You are correct, some layers are too slow/far from optimized. CAI has a super efficient AVX based convolutional calculation.

I see. Maybe I'll take a look, if there is something to learn for me. Not promising, as I'm currently deeply involved in my own pet-project.

So, having 20x speed-up in plain convolutional and fully connected layers is nice. Plenty of the layers listed above are brand new. I have dozens of brand new layer types to optimize. It will be super cool to see these numbers improving as the code is optimized.

So, for my understanding. The TPU part is new while you also added layer functionality - and you're expecting the TPU part to improve, correct?

[/b] Yes. I am using FP32 with pas-core-math32.

Do you see any chance to migrate to smaller floating point formats - i.e. fp16 or bf16 - for the inference (not the training)? If you absolutely need the precision/dynamic range of fp32 there might be something in the 'Ozaki Scheme' from professor Daisuke Takahashi. He made an explicit case for matrix mult - https://arxiv.org/abs/2606.29129. This would only be applicable to TPU iiuc - and a real major re-write I suspect.

Totally unrelated - I found this one https://arxiv.org/abs/2606.28639 quite interesting.

[schuler]

@microxa,
COOL TESTING!

Can I ask you please to update from master and run your test again? Also, are you running on Windows/Linux?

[schuler]

Hello @MathMan!

The TPU part is new while you also added layer functionality

I have sad news: there is no compiler from OpenCL to TPUs. So, for now, only OpenCL capable GPUs and FPGAs are on the horizon. CAI runs on AVX and/or OpenCL.

Actually, this reminds me that I should test CAI with FPGAs. Running a neural network in an FPGA should be fun to watch.

Do you see any chance to migrate to smaller floating point formats - i.e. fp16 or bf16

At this moment, there is no plan for this. But, who knows, one day, I might end up forking FPC unleashed just for this…

[schuler]

Hello
This is to share an interesting experiment in an 8 cores google colab CPU. Although the name of the program file is IntraLayerThreadingBench, the brand new TNNetExecutionPlanner is able to parallelize layers (2-n different layers at a time including one in GPU and many in CPU at the same time) and chunks of computing inside of a given layer (intra-layer parallelization).

FC = Fully Connected Layer
PW = Pointwise convolution
Conv = 3x3 kernel convolution

This is the output of IntraLayerThreadingBench:

Code: [Select]

=== Experiment A: per-layer threading OFF vs ON  [low-memory OFF] ===
Cores (NeuralDefaultThreadCount) = 8   every size is chunk-eligible
shape                 neurons  elig?    off ms     on ms  speedup  chunks   maxdiff
--------------------------------------------------------------------------------------
FC 64x64                   64    yes     0.004     0.041    0.09x       8         0
FC 128x128                128    yes     0.007     0.041    0.17x       8         0
FC 256x256                256    yes     0.015     0.044    0.35x       8         0
FC 512x512                512    yes     0.071     0.061    1.17x       8         0
FC 768x768                768    yes     0.207     0.081    2.55x       8         0
FC 1024x1024             1024    yes     0.326     0.113    2.88x       8         0
FC 1536x1536             1536    yes     0.741     0.186    3.99x       8         0
FC 2048x2048             2048    yes     1.571     0.284    5.54x       8         0
FC 3072x3072             3072    yes     4.938     0.776    6.36x       8         0
FC 768x3072 up           3072    yes     0.928     0.219    4.23x       8         0
FC 3072x768 dn            768    yes     0.588     0.163    3.60x       8         0
Pw 8x8x64->64              64    yes     0.095     0.069    1.38x       8         0
Pw 8x8x128->128           128    yes     0.237     0.125    1.89x       8         0
Pw 32x32x256->256         256    yes    11.429     2.750    4.16x       8         0
Pw 16x16x512->512         512    yes     9.000     2.952    3.05x       8         0
Conv 8x8x32 k3             32    yes     0.056     0.069    0.81x       8         0
Conv 8x8x64 k3             64    yes     0.174     0.113    1.54x       8         0
Conv 16x16x64 k3          128    yes     1.881     0.508    3.71x       8         0
Conv 32x32x64 k3           64    yes     4.588     1.276    3.60x       8         0
Conv 8x8x256 k3           256    yes     2.667     0.712    3.75x       8         0

Notice that the small computations take a performance hit. The parallel execution overhead does not justify a small computation. BTW, while the work about optimizing the forward pass progresses, Qwen 2.5 05B is already 3x faster than a week ago.

Wish everyone happy pascal coding

[schuler]

Hello

This is just to share an experiment. I tested 3 Qwen 2.5 models on Google Colab High Mem CPU (RAM: 51GB - no gpu).

This is the CPU:

Code: Bash  [Select][+][-]

1. Architecture:                x86_64
2.   CPU op-mode(s):            32-bit, 64-bit
3.   Address sizes:             46 bits physical, 48 bits virtual
4.   Byte Order:                Little Endian
5. CPU(s):                      8
6.   On-line CPU(s) list:       0-7
7. Vendor ID:                   GenuineIntel
8.   Model name:                Intel(R) Xeon(R) CPU @ 2.20GHz
9.     CPU family:              6
10.     Model:                   79
11.     Thread(s) per core:      2
12.     Core(s) per socket:      4
13.     Socket(s):               1
14.  

The models were downloaded with:

Code: Bash  [Select][+][-]

1. git clone https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct q2
2. git clone https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct q2-1.5
3. git clone https://huggingface.co/Qwen/Qwen2.5-3B-Instruct q2-3

To compile and run:

Code: Bash  [Select][+][-]

1. lazbuild -B neural-api/examples/ChatTerminal/ChatTerminal.lpi
2. neural-api/bin/x86_64-linux/bin/ChatTerminal q2/ --stats --profile --greedy
3. neural-api/bin/x86_64-linux/bin/ChatTerminal q2-1.5/ --stats --profile --greedy
4. neural-api/bin/x86_64-linux/bin/ChatTerminal q2-3/ --stats --profile --greedy

Results:

Code: Python  [Select][+][-]

1. Qwen 2.5 0.5B: 32 tokens/second
2. Qwen 2.5 1.5B: 14 tokens/second
3. Qwen 2.5   3B:  7 tokens/second

Next week, I will have benchmarks for Qwen2.5 7B, 14B and 32B plus at least one benchmark for the Qwen 3 family.

Wish everyone happy pascal coding

[schuler]

Hello. This is to share an experiment comparing Qwen 2.5 0.5B inference speed with ollama against NEURAL-API.

Hardware
The hardware used for testing is: Google Cloud High RAM CPU (no gpu)

Code: Pascal  [Select][+][-]

1. Architecture:                x86_64
2.   CPU op-mode(s):            32-bit, 64-bit
3.   Address sizes:             48 bits physical, 48 bits virtual
4.   Byte Order:                Little Endian
5. CPU(s):                      8
6.   On-line CPU(s) list:       0-7
7. Vendor ID:                   AuthenticAMD
8.   Model name:                AMD EPYC 7B12
9.     CPU family:              23
10.     Model:                   49
11.     Thread(s) per core:      2
12.     Core(s) per socket:      4
13.     Socket(s):               1

The ollama baseline
I installed, downloaded the model and run with:

Code: Python  [Select][+][-]

1. !sudo apt-get install -y pciutils zstd
2. !curl -fsSL https://ollama.com/install.sh | sh
3. !pip install ollama
4. import subprocess
5. import time
6.  
7. # Start the server in the background
8. subprocess.Popen(["ollama", "serve"])
9.  
10. # Give the server 5 seconds to fully warm up
11. time.sleep(5)
12. print("Ollama server is running successfully!")
13. !ollama pull qwen2.5:0.5b-instruct-q8_0
14. import ollama
15. import requests
16.  
17. r = requests.post("http://localhost:11434/api/chat", json={
18.       "model": "qwen2.5:0.5b-instruct-q8_0",
19.       "messages": [{"role": "user", "content": "Can you tell me please a bed time story?"}],
20.       "stream": False
21.   }).json()
22.  
23. print(r)
24.  
25. gen_tps    = r["eval_count"] / r["eval_duration"] * 1e9
26. prompt_tps = r["prompt_eval_count"] / r["prompt_eval_duration"] * 1e9
27.  
28. print(r["message"]["content"])
29. print(f"\nGeneration: {gen_tps:.1f} tok/s ({r['eval_count']} tokens)")
30. print(f"Prompt eval: {prompt_tps:.1f} tok/s")

The NEURAL-API experiments
I installed, downloaded the model and run with:

Code: Bash  [Select][+][-]

1. git clone https://github.com/joaopauloschuler/neural-api.git
2. git clone https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct q2
3. lazbuild -B neural-api/examples/ChatTerminal/ChatTerminal.lpi
4. neural-api/bin/x86_64-linux/bin/ChatTerminal q2/ --stats --profile --max-fast-memory
5. neural-api/bin/x86_64-linux/bin/ChatTerminal q2/ --stats --profile --low-memory --cpu

RESULTS

Ollama

Code: Pascal  [Select][+][-]

1. {'model': 'qwen2.5:0.5b-instruct-q8_0', 'created_at': '2026-07-23T22:04:18.010178901Z', 'message': {'role': 'assistant', 'content': 'Certainly! I\'d be delighted to share a bedtime story with you.\n\nOnce upon a time, there was a little girl named Lily who loved to listen to the stories her grandmother would tell her at night. One night, as she lay in bed, Lily heard a soft murmur that seemed to come from her bedroom window. She stretched and opened her eyes, but when she looked out of the window, she didn\'t see anyone.\n\nAs the clock ticked down to bedtime, Lily whispered the story inside her head, imagining herself waking up to discover nothing of the sort. "One night a baby named John was born, and he woke up in his mother\'s arms," she said softly. "But instead of being carried out of the crib, John fell asleep in the warmth of the mother\'s embrace."\n\nLily closed her eyes and drifted off to sleep, ready to face whatever came next. As she lay there, her mind began to wander: what if John was born again? What if he never left his parents\' bed?\n\nAs the night wore on, Lily\'s worries began to fade away, replaced by a sense of peace that filled her heart. She drifted back to sleep once more, ready for whatever came next.\n\nAnd so it is with bedtime stories - they can be as magical or as frightening as you wish them to be, but what we do know is that they give us comfort and light in the darkest moments of our nights.'}, 'done': True, 'done_reason': 'stop', 'total_duration': 7191332438, 'load_duration': 242118779, 'prompt_eval_count': 39, 'prompt_eval_duration': 41973000, 'eval_count': 292, 'eval_duration': 6898555000}
2. Certainly! I'd be delighted to share a bedtime story with you.
3.  
4. Once upon a time, there was a little girl named Lily who loved to listen to the stories her grandmother would tell her at night. One night, as she lay in bed, Lily heard a soft murmur that seemed to come from her bedroom window. She stretched and opened her eyes, but when she looked out of the window, she didn't see anyone.
5.  
6. As the clock ticked down to bedtime, Lily whispered the story inside her head, imagining herself waking up to discover nothing of the sort. "One night a baby named John was born, and he woke up in his mother's arms," she said softly. "But instead of being carried out of the crib, John fell asleep in the warmth of the mother's embrace."
7.  
8. Lily closed her eyes and drifted off to sleep, ready to face whatever came next. As she lay there, her mind began to wander: what if John was born again? What if he never left his parents' bed?
9.  
10. As the night wore on, Lily's worries began to fade away, replaced by a sense of peace that filled her heart. She drifted back to sleep once more, ready for whatever came next.
11.  
12. And so it is with bedtime stories - they can be as magical or as frightening as you wish them to be, but what we do know is that they give us comfort and light in the darkest moments of our nights.
13.  
14. Generation: 42.3 tok/s (292 tokens)
15. Prompt eval: 929.2 tok/s

NEURAL-API with “--fast-max-memory”

Code: Pascal  [Select][+][-]

1. Tokenizer loaded in 6.7s.
2. [context not set - defaulting to 2048 tokens; override with --ctx N (memory grows ~O(ctx^2))]
3. [int8 weights (default) - less RAM, faster on CPU and GPU; on --gpu the codes stay resident on the device; --fp32 opts out]
4. [--max-fast-memory: concatenated weight cache kept - faster forward, more RAM, GPU compatible]
5. Loading q2/ ...
6. Model loaded in 11.6s.
7. Error: Cannot get number of platforms!
8. [--gpu: no OpenCL platform found - falling back to CPU]
9. [int8 KV cache (default with int8 weights) - ~1/4 the KV RAM, logits not bit-exact; --kv-fp32 opts out]
10. Model: qwen2, 630139776 params, vocab 151936, context 2048, chat format qwen, int8 weights.
11. [KV-cache reuse ON - only the new prompt tail is prefilled each turn]
12. [layer-graph parallel forward (default) - independent layers and large conv/linear layers threaded; pass --serial for the serial loop]
13. [sampling: top-p 0.80, temperature 0.70, repetition-penalty 1.10 - flags > generation_config.json > fallback]
14. Type your message; /exit quits, /reset clears the history,
15. /system <msg> sets the system prompt.
16. > Can you tell me please a bed time story?
17. Of course! Here's a simple bedtime story for you:
18.  
19. Once upon a time, in a cozy little cabin, there lived a young girl named Sarah. She had a magical imagination and loved to dream of adventures.
20.  
21. One day, while she was walking through the forest, she came across a strange, glowing light. As she approached, she saw a small, smiling creature that seemed to know her very well. It invited her to join its family, where it would teach her everything she needed to know.
22.  
23. Sarah was thrilled! She was excited to learn from the wise old owl, who told her about the importance of friendship and the value of being kind to others. She was also excited to learn about the forest creatures, who were like her best friends and shared their stories with her.
24.  
25. As the night grew longer, Sarah realized that she was getting too tired. She wanted to rest, but the owl kept telling her that there was more to learn. She thought about the magical world and the creatures that lived there, and decided to stay up late and continue learning.
26.  
27. Finally, after a long, restless night, Sarah fell asleep, dreaming of all the wonderful things she had learned during her adventure. As she drifted off to sleep, she heard the owl singing its happy song, and her heart filled with happiness and contentment.
28.  
29. When she woke up, Sarah was back in her cozy cabin, with her magic powers and a new friend. She hugged her owl and said goodnight, knowing that she would learn many more things as she grew older.
30. [stats] 310 tokens, TTFT 661 ms, prompt 39 (reused 0), decode 48.4 tok/s
31.  
32. Layer Class Timing Report
33. =========================
34. Aggregated forward-pass time by layer class (accumulated since
35. the last ClearTime). Sorted by total cost descending. The GPU
36. column is the share of forward dispatches that took the OpenCL
37. path. Convolution (incl. grouped/KAN/cross-correlation variants),
38. FullConnect, SDPA, RoPE/MRoPE, softmax, RMSNorm/Group/L2 norms,
39. GLU gates, pooling, grid/resize gathers and embedding all have a
40. GPU forward path; "-" = no GPU path here / never dispatched.
41. Layer class                  Count       total us    us/instance      %      GPU
42. --------------------------------------------------------------------------------------
43. TNNetPointwiseConvLinear       145     3510993.58       24213.75  88.1      0%
44. TNNetFusedSDPA                  24      198001.72        8250.07   5.0      0%
45. TNNetTokenRMSNorm               49       83001.59        1693.91   2.1      0%
46. TNNetRotaryEmbedding            48       67002.63        1395.89   1.7      0%
47. TNNetSum                        48       46002.19         958.38   1.2       -
48. TNNetSwiGLU                     24       45995.90        1916.50   1.2      0%
49. TNNetDeepConcat                 24       31000.26        1291.68   0.8       -
50. TNNetEmbedding                   1        1999.71        1999.71   0.1      0%
51. TNNetInput                       1           0.00           0.00   0.0       -
52. --------------------------------------------------------------------------------------
53. TOTAL: 3983997.58 us across 364 layer(s) in 9 class(es)
54. [sched] width 8, workers 8 | passes: 310 parallel, 0 serial | peak in-flight 8/8 | gain bound 1.01x, off-primary 87%
55. >

NEURAL-API with “--cpu --low-memory”

Code: Pascal  [Select][+][-]

1. Tokenizer loaded in 6.6s.
2. [context not set - defaulting to 2048 tokens; override with --ctx N (memory grows ~O(ctx^2))]
3. [int8 weights (default) - less RAM, faster on CPU and GPU; on --gpu the codes stay resident on the device; --fp32 opts out]
4. [low-memory forward (default) - concatenated weight cache dropped, per-neuron compute, not compatible with GPU, pass --max-fast-memory to keep the (faster) cache and/or use GPU.]
5. Loading q2/ ...
6. Model loaded in 10.8s.
7. [int8 KV cache (default with int8 weights) - ~1/4 the KV RAM, logits not bit-exact; --kv-fp32 opts out]
8. Model: qwen2, 630139776 params, vocab 151936, context 2048, chat format qwen, int8 weights.
9. [KV-cache reuse ON - only the new prompt tail is prefilled each turn]
10. [layer-graph parallel forward (default) - independent layers and large conv/linear layers threaded; pass --serial for the serial loop]
11. [sampling: top-p 0.80, temperature 0.70, repetition-penalty 1.10 - flags > generation_config.json > fallback]
12. Type your message; /exit quits, /reset clears the history,
13. /system <msg> sets the system prompt.
14. > Can you tell me please a bed time story?
15. Of course! Here's a simple bedtime story for you:
16.  
17. Once upon a time, in a small village, there lived a little girl named Lily. She had a dream that was filled with magical creatures and fantastical adventures.
18.  
19. Lily's dream was so beautiful that she dreamed it over and over, hoping to see it again. One day, as she was lying in bed, she heard a gentle voice say, "Good night, little one. Your dreams are beautiful."
20.  
21. Lily woke up and opened her eyes. She was surprised to see a golden sun shining brightly above her head. She saw a family of fairies coming down from the sky, and they welcomed her with a warm hug.
22.  
23. Lily's parents then brought her to a magical forest where they let her sleep under a glowing tree. There, they gave her a special magic potion that made her sleep forever.
24.  
25. As Lily slept, she dreamed of a long and wonderful day ahead. She saw beautiful flowers bloom and birds singing, and she felt a sense of peace and contentment.
26.  
27. Finally, as the night began to fall, Lily dreamed of a magical sunset. She watched as the sun set behind the mountains, and the sky turned a beautiful shade of pink and purple. She felt the warmth of the sun on her skin, and she knew that she would wake up feeling refreshed and ready for the day ahead.
28.  
29. As the moon rose, Lily knew that it was time to wake up. She stretched her arms and gently closed her eyes. As she did, she heard a soft voice say, "Good night, little one. Your dreams are beautiful."
30.  
31. Lily smiled and closed her eyes again. She knew that she would wake up refreshed and ready for the day, and she would be able to share her magical dreams with her family and friends.
32. [stats] 364 tokens, TTFT 635 ms, prompt 39 (reused 0), decode 44.8 tok/s
33.  
34. Layer Class Timing Report
35. =========================
36. Aggregated forward-pass time by layer class (accumulated since
37. the last ClearTime). Sorted by total cost descending. The GPU
38. column is the share of forward dispatches that took the OpenCL
39. path. Convolution (incl. grouped/KAN/cross-correlation variants),
40. FullConnect, SDPA, RoPE/MRoPE, softmax, RMSNorm/Group/L2 norms,
41. GLU gates, pooling, grid/resize gathers and embedding all have a
42. GPU forward path; "-" = no GPU path here / never dispatched.
43. Layer class                  Count       total us    us/instance      %      GPU
44. --------------------------------------------------------------------------------------
45. TNNetPointwiseConvLinear       145     4233996.22       29199.97  89.6      0%
46. TNNetFusedSDPA                  24      221997.64        9249.90   4.7      0%
47. TNNetTokenRMSNorm               49       79996.67        1632.59   1.7      0%
48. TNNetRotaryEmbedding            48       72002.22        1500.05   1.5      0%
49. TNNetSwiGLU                     24       51998.81        2166.62   1.1      0%
50. TNNetSum                        48       41001.97         854.21   0.9       -
51. TNNetDeepConcat                 24       26001.29        1083.39   0.6       -
52. TNNetInput                       1           0.00           0.00   0.0       -
53. TNNetEmbedding                   1           0.00           0.00   0.0      0%
54. --------------------------------------------------------------------------------------
55. TOTAL: 4726994.83 us across 364 layer(s) in 9 class(es)
56. [sched] width 8, workers 8 | passes: 364 parallel, 0 serial | peak in-flight 8/8 | gain bound 1.01x, off-primary 87%
57. >

In short:

• ollama: 42 tokens/second
• CAI Neural API with --max-fast-memory: 48 tokens/second
• CAI Neural API with --low-memory --cpu: 44 tokens/second

I will not make claims. This is the first day that NEURAL-API outperforms ollama. The source code may contain errors. But it is promising. It is nice to see Pascal outperforming C.