Object Pascal, Memory Safety, the US Whitehouse and future programming

[AlexK]

The point is that nil pointer exception is not the most dangerous exception in the system like those managed/safe languages that you mirror. It is dereferencing a stale pointer/reference.

In the grand scheme of things for a language to pass as "memory safe"?
"Dangling pointer" in C parlance.

This problem in non-GC languages can be considered already solved by smart pointers. (C++ shared_ptr)
Main requirement for smart pointers implementation is the ability to overload pointer dereference operator. (C++ https://www.aristeia.com/Papers/DDJ_Oct_1999.pdf )
It's needed for smart pointers to be interchangeable with raw pointers in code.

[marcov]

Yes, basically ARC, but without cycle busting. But, like all things ARC is a tradeoff, performance the ability to do lowlevel buffer management.

I wasn't aware that that also solved the problem when a pointer points into a buffer. (Does C++ shared ptr also increase refcount then?)

In short, it depends how much emphasis you put on the "problem" being a problem. I don't like the tradeoff, at least not for the applications I'm currently making.

[AlexK]

"Sound Null Safety" considers Nil useful but gives full control to programmer and compiler that guarantees there'll be no unexpected runtime "NullPointerException"s.

Good. But this implies that nil/null is considered of limited utility, a "dangerous, handle with care" concept.

With "Sound Nil Safety" Nil is guaranteed no longer dangerous or harmful.
There are no places in the code where Nil can fly in unexpectedly and compiler ensures that all checks for Nil value are present, where it's required only.
Other possible benefits, like Self can never be Nil.

[AlexK]

Yes, basically ARC, but without cycle busting. But, like all things ARC is a tradeoff, performance the ability to do lowlevel buffer management.

I wasn't aware that that also solved the problem when a pointer points into a buffer. (Does C++ shared ptr also increase refcount then?)

shared_ptr can own(refcount) an object but point into that object or other object somewhere.
There's no automatic cycle busting, but weak_ptr. Refcounting is thread safe.

It seems to me that TComponent memory management facilities are easier to use than shared_ptr in C++.

unique_ptr provides move semantics at compile time, similar to Rust.
Useful to pass ownership between threads.

"Sound Null Safety" is a boon by itself for a language/compiler. Not tied to memory management.

[Warfley]

"Sound Null Safety" is the most novel concept-solution in languages/compilers, but fixes half-century problem that wasn't considered harmful.

"Sound null safety" is just a fancy name the Dart programming language gave a typetheoretical concept that is at least around since the first implementation of the maybe monad in ML in the 70s, and us exactly the same concept that is used by Haskell, Rust, Swift, Kotlin, etc.

The idea is basically instead of having a magic null value that is implicitly part of the domain of some (or all) types, you have an explicit null type and if you want to make a variable nullable by forming a union type over the data type with the null type.
With this the typechecker can enforce that if you access the value of the type you check the domain of the value first, and thereby enforce checking for null.

Languages like Rust or Haskell which come from ML make this explicit as they try to not have any magic types. Languages like Dart, Swift, typescript or Kotlin on the other hand have special syntactic sugar for it with ? And ! Operators. But in the end its the same thing, one is just self defined based on language primitives while the other is syntactic sugar. Other than that they work identical

[AlexK]

"Sound null safety" is just a fancy name the Dart programming language gave a typetheoretical concept that is at least around since the first implementation of the maybe monad in ML in the 70s, and us exactly the same concept that is used by Haskell, Rust, Swift, Kotlin, etc.

Dart and Swift currently are the only imperative languages that acquired Sound Null Safety (no opt-out from null safety for libraries/dependencies). Not certain about Eiffel.
Rust is imperative but it chose some explicit functional features from Ocaml. In the beginning Rust even had a GC.

The usual defensive programming style using Nil is a pragmatic approach also worth adhering to.
But due to the lack of static analyzer tool all the burden is always on a programmer.
Pascal was conceptualized as a language amenable to program-wise static analysis.

Why would language users conform with Self to be Nil...
I'm searching what causes Self to be Nil sometimes in Object Pascal, this is the case(a bug):

Code: Pascal  [Select][+][-]

1. Writeln(SomeClass(nil).SomeMethod);

In this particular case to prevent Self to be Nil inside SomeMethod compiler can only insert a run-time check for Nil that raises exception and advise programmer to wrap it in try block.

[creaothceann]

I'm searching what causes Self to be Nil sometimes in Object Pascal

In a constructor perhaps, before the inherited Create is called?

[AlexK]

I'm searching what causes Self to be Nil sometimes in Object Pascal

In a constructor perhaps, before the inherited Create is called?

Self in constructor already points to allocated data in memory.
Documentation https://www.freepascal.org/docs-html/ref/refse38.html  :

Calling the constructor will provoke a call to the virtual class method NewInstance, which, in its default implementation, calls GetMem, to allocate enough space to hold the class instance data, and then zeroes out the memory.

After that, the constructor’s code is executed. The constructor has a pointer to its data, in Self.

For some reason TObject.NewInstance is virtual, which means it can be overridden without calling default implementation in TObject.
https://lazarus-ccr.sourceforge.io/docs/rtl/system/tobject.newinstance.html

But class constructor doc states that they are not allowed be virtual, "Neither constructor nor destructor can be virtual" :
https://www.freepascal.org/docs-html/ref/refsu29.html

[Warfley]

Dart and Swift currently are the only imperative languages that acquired Sound Null Safety (no opt-out from null safety for libraries/dependencies). Not certain about Eiffel.

What about Typescript? Sure it's annyoing that it has both null and undefined, but unless you use "any" which is explicitly saying "please deactivate all typechecking" you cannot have nullpointer exceptions. And I mean in both swift and dart you can also use ! to circumvent the typechecker.

Rust is imperative but it chose some explicit functional features from Ocaml. In the beginning Rust even had a GC.

It's more Haskell not OCaml, but doesn't really matter because the underlying concepts mostly stem from ML anyway.

But anyways, my point is still that the ? and ! is just syntactic sugar around optional typing. For example, if the DFA in the fpc would be augmented with variant selector tracking, and TNullable would be defined as:

Code: Pascal  [Select][+][-]

1. type
2.   generic TNullable<T> = record
3.   case HasValue: Boolean of
4.   True: (Value: T);
5.   False: ();
6.   end;

Then it would behave exactly like the optional types in Swift, Dart, etc. completely in language. The operators ? and ! could then be just added as syntactic sugar.
So basically all this functionality could be "easily" (still a lot of work to implement, maybe even invovling SMT solving) added to the FPC with the existing structures. Even better than in Swift or Dart, this can be extended to union types or any other variant type, similar to how TypeScript handles these things and not being limited to only nil checking.
Making this possible was btw exactly why I built this feature last year, because it's a prequesite for building something like this. But the MR sits around since then

Why would language users conform with Self to be Nil...
I'm searching what causes Self to be Nil sometimes in Object Pascal, this is the case(a bug):

Code: Pascal  [Select][+][-]

1. Writeln(SomeClass(nil).SomeMethod);

I personally think nil should not automatically be a pointer value but instead there should be a bottom type "nil" which has only one value "nil" which can be used for operator overloading such as TNullable (basically like TNullPtr which is used by TNullable). This would be safe but also consistent in the typesystem and can be used to extend the concept to other types.

I also want to note that I think using Free is terrible, because it working on nil can easily hide double free errors which otherwise would be discoverable with heaptrc or valgrind. Calling Free on a nil pointer is probably in 99% of cases not intended and a sign of an error. That said, there are definitely cases where calling a method on nil can make sense.

[PascalDragon]

It's literally the only concept in Pascal where the right hand side of an expression is dependent on the left hand side, and kindof undermines the typesystem.

That's not the only concept. It's also the case with operator overloading so that the correct overload is picked and when assiging a method or function to a method or function pointer so that the correct overload is picked there.

For some reason TObject.NewInstance is virtual, which means it can be overridden without calling default implementation in TObject.
https://lazarus-ccr.sourceforge.io/docs/rtl/system/tobject.newinstance.html

Correct. This way one can implement for example some memory pool or something if one desires.

But class constructor doc states that they are not allowed be virtual, "Neither constructor nor destructor can be virtual" :
https://www.freepascal.org/docs-html/ref/refsu29.html

Class constructors are not instance constructors. The later can be virtual as well. The former are only called once per program per type.

[AlexK]

Dart and Swift currently are the only imperative languages that acquired Sound Null Safety (no opt-out from null safety for libraries/dependencies). Not certain about Eiffel.

What about Typescript? Sure it's annyoing that it has both null and undefined, but unless you use "any" which is explicitly saying "please deactivate all typechecking" you cannot have nullpointer exceptions. And I mean in both swift and dart you can also use ! to circumvent the typechecker.

Dart language developers think that Typescript only imitates statically typed language. They don't say it in a direct manner.

But anyways, my point is still that the ? and ! is just syntactic sugar around optional typing.
...
Making this possible was btw exactly why I built this feature last year, because it's a prequesite for building something like this. But the MR sits around since then

In Dart it's not just syntax, but a guaranty from compiler enforcing its well defined type system.

That MR requires quite a time investment to evaluate it, also it depends on other MR.
I noticed it proposes run-time checks insertion. For a language like Object Pascal such run-time checks must be defined on a Language Theory level, otherwise compiler implementers/maintainers will have big problems reasoning about compiler's code and requirements over time.

I discovered that in order to get rid of "Self can be Nil" a run-time check insertion is required. It must be well-defined for the Language before even be deemed worth considering for compiler implementation. Currently(since the 1990-s) Object Pascal just copies C++ undefined behavior allowing Self to be Nil.

I also want to note that I think using Free is terrible, because it working on nil can easily hide double free errors which otherwise would be discoverable with heaptrc or valgrind. Calling Free on a nil pointer is probably in 99% of cases not intended and a sign of an error. That said, there are definitely cases where calling a method on nil can make sense.

It led me to try to investigate what's the original reason to have Free method at all.
The reason is "Self can be Nil" in Object Pascal.
It looks like devs(Borland-Delphi?) just took C++ particular undefined behavior and applied it in RTL.

[AlexK]

For some reason TObject.NewInstance is virtual, which means it can be overridden without calling default implementation in TObject.
https://lazarus-ccr.sourceforge.io/docs/rtl/system/tobject.newinstance.html

Correct. This way one can implement for example some memory pool or something if one desires.

Class constructors are not instance constructors. The later can be virtual as well. The former are only called once per program per type.

Yes, this behaviour introduced "Self can be Nil" to the language.
If some class overrides TObject.NewInstance and doesn't call inherited; then there's no guarantee that Self points to a valid memory block. So "Self can be Nil".

Another entrance for "Self can be Nil" in the language is this:

Code: Pascal  [Select][+][-]

1. Writeln(SomeClass(nil).SomeMethod);

Compiler inserts a call to SomeMethod that will have Self equal to Nil.

To fix it without run-time check insertion compiler must advise/demand with error "You should wrap it in an IF block checking for Nil value".

The more I look into such issues the more it shows that's a use case for a Static Analyzer tool.
Indeed, requesting/proposing some features(guarantees) from/to a compiler is a usual road towards "feature bloat", other new bugs, uncertainty whether something is a bug or a feature, slower compilation.

TObject.NewInstance is pluggable and it leads to "Self can be Nil" sometimes. It seems so.
In many other languages one usually doesn't come up with a thought-train like - "what if I could implement a heap without an OS by one class on a 32-bit microcontroller with 0.5 Mb SRAM", or something like that.

[Warfley]

Dart language developers think that Typescript only imitates statically typed language. They don't say it in a direct manner.

Typescript is a statically typed language, built on a dynamically typed runtime. If you stay within typescript it's mostly typesafe (there is exactly one case where unions are treated non conservatively which can lead to bugs).
Most issues arise in Typescript because it usually interfaces with plain javascript, so while typescript is typesafe, any interface with raw Javascript isn't.

It's the same problem with Kotlin and Java integration, your Kotlin null checker doesn't matter if you receive data from Java which could be null (even if it's tagged as @NotNull).

That MR requires quite a time investment to evaluate it, also it depends on other MR.
I noticed it proposes run-time checks insertion. For a language like Object Pascal such run-time checks must be defined on a Language Theory level, otherwise compiler implementers/maintainers will have big problems reasoning about compiler's code and requirements over time.

Yes the compiletime checks as part of the DFA (Data Flow Analysis, when compiling with -Oodfa) would be the next step. This MR is just the first step, as in the current implementation FPC does not really map variant parts to the selector fields and things like this are possible:

Code: Pascal  [Select][+][-]

1. TTest = record
2. case b: Boolean of
3. False: (A: Integer);
4. False: (B: Double);
5. end;

Also currently variant parts cannot have managed types, and as String is probably the most commonly used type in Pascal, not being able to have string usable would be kinda bad.
So this MR tries to do at first that (well and also requires the other MR by me to do the groundwork in the compiler to make this possible to begin with).

Once those are implemented the next step would be to do selector domain tracking in the DFA, basically trying to map which values the selector can have at which point in the code. With this static compiletime checks can be done like:

Code: Pascal  [Select][+][-]

1. n:=GetPotenitalNull;
2. WriteLn(n.Value); // Error selector might be False
3. if n.HasValue then
4.   WriteLn(n.Value); // no error because selector is guaranteed to be True

Note this would also allow the following:

Code: Pascal  [Select][+][-]

1. type
2.   TUnion<T, U> = record
3.     case Selector: (First, Second) of
4.     First: (First: T);
5.     Second: (Second: U);
6.   end;
7.  
8. var
9.   u: TUnion<String, Integer>;
10. begin
11.   u:=GetStringOrInt;
12.   WriteLn(u.First); // Error selector may be second
13.   if u.Selector = First then
14.     WriteLn(u.First) // No error because selector selects the correct branch
15.   else
16.     WriteLn(u.Second); // No error because in else only the other branch can be selected
17. end;

Having a conservative check like the above wouldn't be too difficult to implement, basically just track a set of possible values (e.g. based on a range set) and make a set difference with the set of ranges that select a certain branch. If the difference is non empty it is not provable that the branch you are trying to access is actually selected and the compiler can throw an error.
This of course only works easily for simple comparison operators and maybe some basic arithmetic. Anything more complicated requires basically an SMT solver like Z3 (which is also why TypeScript has such an advanced typechecker, because Microsoft has a lot of in-house knowledge from their Z3 and formal analysis development teams)

[440bx]

Code: Pascal  [Select][+][-]

1. TTest = record
2. case b: Boolean of
3. False: (A: Integer);
4. False: (B: Double);
5. end;

That's considered a bug.  There is a ticket for it but, I didn't search for it.

The compiler should not accept duplicate values in a variant (obvious ambiguity.)  The bug is still present in v3.2.2 but it might have been fixed in v3.3.1 (I don't know if @PascalDragon got to it.)

[Thaddy]

the correct syntax is:

Code: Pascal  [Select][+][-]

1. TTest = record
2. case Boolean of
3. False: (A: Integer);
4. true: (B: Double);
5. end;

No bug, but wait: What is the bug is the duplicate case lable with false, false and the example is wrong.
Because this by 440bx:

Code: Pascal  [Select][+][-]

1. TTest = record
2. case b:Boolean of  // b one
3. False: (A: Integer);
4. false: (B: Double); // b two
5. end;

has two b' s and won't compile with Error: Duplicate identifier "b"

The actual error is the duplicate case label, though, and that is ignored. That is the bug.
I would read the compiler output first..

Code: Pascal  [Select][+][-]

1. TTest = record
2. case Boolean of
3. False: (A: Integer);
4. False: (B: Double); //??????
5. end;

There's your bug and that has indeed been reported fairly recently.
(tested with {$B-}/{$B+} makes no difference)

And that is not fixed (yet)