SedaiBasic2 vs. the FreeBASIC compiler's own test suite
I run my BASIC implementation against fbc's own regression suite — unmodified, at tag 1.10.1. Here is where it stands today.
The suite's -lang fb set holds 2,292 files; 38 carry no test tag and aren't tests. Of the remaining 2,254 tests, 1,334 pass — 59%.
| what the test asks | tests | pass | |
|----------------------------------------|-------|------|---------------|
| must compile | 320 | 319 | 99.7% |
| must be rejected (with a diagnostic) | 1,238 | 574 | 46.4% |
| must compile and run correctly | 21 | 12 | 57% |
| every assertion must hold (unit tests) | 655 | 429 | 65.5% |
| multi-module link | 20 | — | not wired yet |
The number worth reading twice is the second row: 55% of the suite is not about features at all — those tests hand the compiler an invalid program and require it to be refused, with the right exit code. That is a type checker, and it's a project of its own. Counting only the tests that ask the language to actually do something: 760 / 1,016 = 75%.
One honest caveat: fbc's harness has no percentage. make unit-tests links every unit test into a single binary that exits 0 only if every assertion of every test passes; make log-tests wants zero failures. So 59% is my own measure of partial progress, not a grade the suite hands out. (For scale: fbc itself doesn't pass its suite either — 18 files carry known-bug blocks that are switched off by default.)
Next milestones, in order: every unit-test file at least runs (73 still don't), then every assertion holds.
