Conscious Artificial Intelligence - Project Update

[kapibara]

Following with interest.

/Roger

[schuler]

@Dzandaa and @kapibara, many thanks for your support!

In the case that you would like to compile/test your existing projects with the new branch (a1), this is the pull request:
https://github.com/joaopauloschuler/neural-api/pull/177

It will be nice to know if your models converge faster/slower/same with the new version. This new version works better with the AVX2 define instead of the previous default for AVX. I intend to fix bugs before merging into master.

Wish everyone happy pascal coding

[kapibara]

Here's my results. I tested master vs PR #177/a1 on a synthetic sequential geometry-detection task.

Environment:
  Linux x86_64
  FPC 3.3.1
  Laz trunk
  11th Gen Intel® Core™ i5-11600K @ 3.90GHz
  15,4 GiB of RAM

The task is binary classification over sequential numeric windows. Each sample is a 300-step window with 5 numeric features per step. The positive class represents a generated geometric shape/pattern inside the sequence; the negative class is background/no-pattern data.

Dataset:

Code: Pascal  [Select][+][-]

1. Total windows: 9,700
2. Train/Validation/Test: 5,820 / 1,940 / 1,940
3. Input shape: 300 x 5 x 1
4. Classes: 2
5. Batch size: 64
6. Epochs: 10
7.  

Training time

Code: Pascal  [Select][+][-]

1. master, no switches:             41.68 sec
2. master + -dAVX2:                 27.72 sec
3. master + -dAVX2 -O3:             27.31 sec
4.  
5. PR #177/a1, no switches:         34.03 sec
6. PR #177/a1 + -dAVX2:             19.30 sec
7. PR #177/a1 + -dAVX2 -O3:         13.09 sec
8.  

Evaluation time

Code: Pascal  [Select][+][-]

1. master, no switches:             ~7.30 sec
2. master + -O3:                     5.95 sec
3. master + -dAVX2:                  ~4.62 sec
4. master + -dAVX2 -O3:              4.25 sec
5.  
6. PR #177/a1, no switches:          5.42 sec
7. PR #177/a1 + -O3:                 3.23 sec
8. PR #177/a1 + -dAVX2:              1.74 sec
9. PR #177/a1 + -dAVX2 -O3:          0.95 sec
10.  

Quality

Code: Pascal  [Select][+][-]

1. master + -dAVX2 -O3:
2.   Accuracy:   99.02%
3.   Precision:  98.97%
4.   Recall:    100.00%
5.   F1:         99.48%
6.   FP/FN:      19 / 0
7.  
8. PR #177/a1 + -dAVX2 -O3:
9.   Accuracy:   99.28%
10.   Precision: 100.00%
11.   Recall:     99.23%
12.   F1:         99.62%
13.   FP/FN:       0 / 14
14.  

Main result:
With -dAVX2 -O3, PR #177/a1 trained about 2.1x faster and evaluated about 4.5x faster than master on this test, with similar or slightly better F1.

[schuler]

@kapibara, as a way to say “thank you”, if you have any feature that you would like coded, please feel free to ask. If not too difficult, AI will code it :-) .

[Dzandaa]

Hi Joao-Paulo,

Can me explain me how to create a semantic segmentation of an image?

Is it possible with CAI?

Thank you.

B->

[schuler]

@Dzandaa,
I will add to my task list coding an example for this.

[microxa]

Hi Paolo! Please take a look at the StringManipulation example — there is still some weird bug there (potentially a glitch in 32-bit FPC 3.2.2). In an earlier version (2.0), this was resolved by reverting to the previous class, i.e., TNNetSoftMax = class(TNNetIdentify).After doing that, the example needed only 5 epochs to reach 100% training accuracy and correct output. Good luck an regards!

[schuler]

@microca,
Thank you for the bug report. I will look at it.

In the case that you are interested, I asked Claude to produce a list of interesting NLP related examples in this API. The below AI generated list has been coded but not yet properly tested.

In the list: Model — Its Building Block: Example showing the building block.

  - DeepSeek — MLA + decoupled-RoPE slice (V2): examples/LatentAttention/LatentAttention.lpr (builds MHA vs MLA vs MLA+decoupled-RoPE arms side by side); multi-token prediction (V3): examples/MultiTokenPrediction/MultiTokenPrediction.lpr (also used by examples/SelfSpeculativeDecoding/SelfSpeculativeDecoding.lpr).
  - LLaMA — pre-norm RMSNorm residual blocks: examples/PreNormVsPostNorm/PreNormVsPostNorm.lpr; SwiGLU: examples/SwiGLUFeedForward/SwiGLUFeedForward.lpr; RoPE: examples/RoPEBaseFrequencySweep/RoPEBaseFrequencySweep.lpr (also compared in examples/PositionEncodingBakeoff/PositionEncodingBakeoff.lpr).
  - Mistral / Longformer — sliding-window causal mask: examples/SlidingWindowBakeoff/SlidingWindowBakeoff.lpr.
  - Gemma — logit soft-capping: examples/SoftCappingStability/SoftCappingStability.lpr and examples/SoftCappingSweep/SoftCappingSweep.lpr; GeGLU: examples/GEGLUFeedForward/GEGLUFeedForward.lpr (also in examples/GatedFFNBakeoff/GatedFFNBakeoff.lpr).
  - BERT / GPT — GELU: examples/ActivationBakeoff/ActivationBakeoff.lpr; GPT-style decoder blocks: examples/TransformerDecoderBlock/TransformerDecoderBlock.lpr (full pipeline in examples/SimpleNLP/TransformerWithTokenizer.lpr).
  - Mamba — TNNetSelectiveSSM: examples/SelectiveSSM/SelectiveSSM.lpr.
  - Modern recurrent family — RWKV-4: examples/RWKV/RWKV.lpr (cross variant: examples/CrossWKV/CrossWKV.lpr); xLSTM sLSTM: examples/SLSTMvsCfC/SLSTMvsCfC.lpr; xLSTM mLSTM: no example yet (TNNetMLSTMCell has tests but no examples/ program); RetNet: examples/RetentionDualForm/RetentionDualForm.lpr; Titans: examples/TitansMemory/TitansMemory.lpr; DeltaNet: examples/DeltaNet/DeltaNet.lpr; GLA: examples/GatedLinearAttention/GatedLinearAttention.lpr (block form: examples/GatedLinearAttentionBlock/GatedLinearAttentionBlock.lpr).
  - Switch Transformer (load-balance loss): examples/TopKMoE/TopKMoE.lpr (contrast without aux loss: examples/ExpertChoiceMoE/ExpertChoiceMoE.lpr).
  - DIFF Transformer: examples/DifferentialAttentionNoise/DifferentialAttentionNoise.lpr.
  - StreamingLLM (attention sinks): examples/SinkAttentionStability/SinkAttentionStability.lpr.
  - BLOOM-lineage ALiBi: examples/ALiBiSlopeSweep/ALiBiSlopeSweep.lpr (also in examples/PositionEncodingBakeoff/PositionEncodingBakeoff.lpr).
  - GQA/MQA via KVHeads: no example yet — the KVHeads parameter exists on the MHA builders (neural/neuralnetwork.pas:11123) and GQA is discussed in examples/LatentAttention/README.md as a comparison point, but no example program actually passes KVHeads <> Heads.

This is a good starting point: https://github.com/joaopauloschuler/neural-api/blob/master/examples/SimpleNLP/DecodeFeaturesBakeoff.lpr (includes some building blocks for DeepSeek v2 and v3)

@all, the list of examples is located at: https://github.com/joaopauloschuler/neural-api/blob/master/examples/README.md

[schuler]

Hi Joao-Paulo,

Can me explain me how to create a semantic segmentation of an image?

Is it possible with CAI?

Thank you.

B->

@Dzandaa, as a source of inspiration for both of us, have a look at please: https://github.com/joaopauloschuler/neural-api/blob/master/neural/neuralpretrained.pas

The master branch is under heavy coding. Therefore, expect bugs. Anyway, I could extend neuralpretrained.pas to support a vision model or models that will do your segmentation.

Is there any image segmentation model from hugging face that you would like supported in this API?

[Dzandaa]

Hi Joao-Paulo

Thank you, I'll read that next week.

I had found an interesting web page that explained the different parts of a Hugging Face segmentation, with images of the different intermediate layers, but I can't find it anymore, sorry.

Here is 2 interesting links:

https://github.com/huggingface/blog/blob/main/mask2former.md

https://medium.com/@az.tayyebi/segformer-model-training-from-scratch-vs-using-hugging-face-f5234f31399a


B->