Reimplementing TObject

[damieiro]

BeniBela: I would love that redirection would be done before any execution of other units (i would put that redirection in a "debug unit", so that "initialization" should be the first one. Is there a way or to force it some how? (for example, being the first in the uses clause on main program). <Edited: I see documentations and doubts in forum that it should be the first unit and not have nested units with initilization or will be the more nested initialization first).

For such tricks it should indeed be one of the first units and should not use other units that have initialization sections that instantiate objects. Please note that you'll have to pull that trick for each and every class separately, because each class has a full copy of the VMT.

Yes, it's clear it should be the first unit and no nested initializations in that. For that reason its simplicity and not doing other weird things.

Oh, i hate these things. I hate when a compiler thinks that programmers are stupid or we don't know what we are doing.

The main purpose of this is to avoid accidental mistakes. 99.99% of developers don't ever need to modify the VMT thus it makes sense to keep it protected just like any other readonly data. On ELF (e.g. Linux) and Mach-O (macOS) based targets that is simply not the case due to some problems with relocations in these files for readonly sections otherwise it would be blocked there as well.
[/quote]

Well, i understand it too, but  i think that .text section (no matter what languaje is used) should the developer's choice if rw or r.

is there any way (compiler/linker option) to avoid this?

No documented one. Change the protection at runtime if you need it, e.g. using VirtualProtect on Windows as 440bx suggested.

Yes, but i dislike it a lot. It won't be portable between OSes. And if this could be avoided by system calls... should not be avoided by user compilation/linker option as a preferred choice?

Helpers do not replace or override the methods, instead they just allow to add additional methods.

Why not? It has no sense for me.

Because that is simply not the purpose of a helper. They don't modify the existing type, they add to it at compile time. It is essentially the Object Pascal variant of extension methods in C# and that's where they originally came from (Delphi.NET introduced them originally).

Ok, i understand. So helpers are like in c# and is for a compatibility issue. Well i would prefer to mimic c++ for some things, but now i know why are these here. (Really i never used them, i simply recompile all my code or expect other's code shouldn't use it)

I think i have explained really bad. I hate prototype and i like low-level. For me, Pascal is same low level as C. In pascal, i watn make my own memory manager and trace pointers (i think it's a good low level feature). But with a high level structure i cannot do it. And i love have in my coding way a Debug unit that if used or activated by a compiler optional can do bookeeping because i will do some low-dirty-or desirable to trace for the sanity of the code and not propagate memory holes and make a more readable dumps. Not a javascrip thing. . I think i'm requesting low level control of facilities, not prototype ones.

Pascal might be as low level as C, but it simply is not C. The language simply introduced some concepts that protects the programmer from common mistakes in C and in addition to that classes are not something that is part of C. So expecting class metatypes or virtual methods of already declared classes to be easily changeable is simply wrong from a language point.

Well, here i disagree with you. It's a nonsense that one can do (and free pascal documents it) how to make a memory manager (quite lower level) by redirecting all, but doesn't allow for a higher level to redirect Tobject having the middle-level structures to do it easily in methods (like afterconstruction and beforeconstruction that  are really suited for it)
And if even these structures were to be read-only, not using a method for TObject to pass him a Procedure for being called for TObject (like a method TObject.SetAfterCreationProcedure (MyProcedure)). TObject.AfterCreation is like abstract method. It can be never executable for user (it's empty), only for a descendant class who overrides it. So all TClass=class  definitions (without anything) or Tclass  = class (TObject) would do nothing (and empty call if not optimiced by compiler) and a lot of code that could be using this great potential debug facility  is simply evaporated. I can't agree with that.
Even it's worse. AfterCreation is triggered by compiler, not dev. So i cannot accept "The way of doing pascal things" as a dogma. I think it has potential beneficts if redirected by a developer who knows. And it can be done and it can be done even in pascal way (as explained in the example) for only this method accept redirection.

I think that if free pascal compiler makes a call to After Construction for ALL classes, and all could be debugged with this facility, TObject must be a logical target. And can be done within languaje philosophy (like pasing a procedure with self parameter configurable between the Tobject definition).Same for Before Destruction.
I think these methods have the big potential, but not are fully accessible.

And, I insists that it's not a problem for my own work. I derived from Tobject and override these methods that works with optional debug and i have good dumps (far better than heatptrc). But it's nearly impossible to get a classes metric (other than mere from mere allocs or not). With a class "captured" afterconstruction you can log text with info like class name, timestamp, etc. etc.
On other's project i usually take all TObject definitions  and derive to my XClass. But it's tedious and error-prone and somehow, absurd boilerplating..

[damieiro]

On the other hand (from the teoretically options how it could be done) Is it possible intercalate a new class  Like TObject -> TXclass -> TPersistent?. I think no for obvious reasons (Tpersitent would point to Tobject, but at least TXclass could override and add methods), and even it could be a bad decision (all new TObject descendants will not have the override)

Yes and no!
If you want to "build in" TXClass before TPersistant what you need is to modify Classes unit. (the unit where TPersisent is declared)
And either explicitly specify TPresistant to inherit from TXClass
OR
add a unit to the uses clause of Classes: (and be careful as the it's OS specific. i.e. Win32 has it's own classes.pp, Linux has its own ... etc)
In the new unit you'd redeclare TObject type to point to TXClass.

Code: Pascal  [Select][+][-]

1. unit XclassHijack;
2.  
3. inteface
4.  
5. uses xclassobj;
6.  
7. type
8.   TObject = TXClass;
9.  

Code: Pascal  [Select][+][-]

1. unit Classes;
2.  
3. inteface
4.  
5. uses
6.   rtlconsts,
7.   sysutils,
8.   types,
9. {$ifdef FPC_TESTGENERICS}
10.   fgl,
11. {$endif}
12.   typinfo,
13.   windows
14.   , XclassHijack;
15.  

Next step is to recompile RTL. and this is something you might want or not want to do

Alternative ways (w/o recompiling RTL) is actually to modify VMT of TObject class in runtime.
So, instead of running default methods it would run your methods. Pretty much a hack.

However, if XClass also has some fields, it might not work. Accessing those fields will likely cause a crash.

Well, i'm seriously thinking about clone the unit that declares Tobject and then add a conditional compiler directive that "afterconstruction" do a calling for my debug bookeeping and test these on code not created for me.
Is needed to recompile RTL or objpas (et all..) is enough? I have never did it. And it seems like killing flyes with cannons.

[damieiro]

Hi Thaddy

Indeed.
Actually TObject is already overbloated.
If you need extensions, please derive and develop your own framework. Like everybody else does.....

@Taddy: I think i understand better what you said. But it can't do the job for this problem.
I have my own framework and my first class is a TObject derived TXclass with makes AfterConstruction redirects. And i like it because it's rock solid in my tests and gives me good dumps.

But third party codes (obviously) can't derive from TXclass, these derive from TObject. I could have several choices:
1.- Make all Tobject descendants being descendants of TXclass -> It could be a nightmare in other's code.
2.- Make all Afterconstruction in TObject. Side effects could be (example an overridden Tobject by descendants, but it seldom happens and if were well coded (with inherited) should be minimal).
3.- TObject would accept a passed procedure to execute because these are methods called by compiler, in this Object only.

Then there are the more (philosophical) questions: If FPC would allow dinamically redirects (like C++) or not. And i don't seem it as low-level or things like that. I think it should be viable like other tool or like passing procedures as parameters.
And i think this it could be done by :
1.- hijacks, tricks
2.- by a well-documented feature (for example and in-system call that could change the VMT).

[PascalDragon]

Oh, i hate these things. I hate when a compiler thinks that programmers are stupid or we don't know what we are doing.

The main purpose of this is to avoid accidental mistakes. 99.99% of developers don't ever need to modify the VMT thus it makes sense to keep it protected just like any other readonly data. On ELF (e.g. Linux) and Mach-O (macOS) based targets that is simply not the case due to some problems with relocations in these files for readonly sections otherwise it would be blocked there as well.

Well, i understand it too, but  i think that .text section (no matter what languaje is used) should the developer's choice if rw or r.

No, because more often then not the OS itself wants to have sections containing code to be read only (especially if some security checks are enabled).

is there any way (compiler/linker option) to avoid this?

No documented one. Change the protection at runtime if you need it, e.g. using VirtualProtect on Windows as 440bx suggested.

Yes, but i dislike it a lot. It won't be portable between OSes. And if this could be avoided by system calls... should not be avoided by user compilation/linker option as a preferred choice?

Then use conditional compilation. I've done the same for the thunk generation code for TVirtualInterface in unit Rtti (see here). If you're working on such a low level you simply can't work around doing OS-specific things.

I think i have explained really bad. I hate prototype and i like low-level. For me, Pascal is same low level as C. In pascal, i watn make my own memory manager and trace pointers (i think it's a good low level feature). But with a high level structure i cannot do it. And i love have in my coding way a Debug unit that if used or activated by a compiler optional can do bookeeping because i will do some low-dirty-or desirable to trace for the sanity of the code and not propagate memory holes and make a more readable dumps. Not a javascrip thing. . I think i'm requesting low level control of facilities, not prototype ones.

Pascal might be as low level as C, but it simply is not C. The language simply introduced some concepts that protects the programmer from common mistakes in C and in addition to that classes are not something that is part of C. So expecting class metatypes or virtual methods of already declared classes to be easily changeable is simply wrong from a language point.

Well, here i disagree with you. It's a nonsense that one can do (and free pascal documents it) how to make a memory manager (quite lower level) by redirecting all, but doesn't allow for a higher level to redirect Tobject having the middle-level structures to do it easily in methods (like afterconstruction and beforeconstruction that  are really suited for it)

Because the memory manager is something that was also provided by TP and Delphi. Especially the later used it to allow common memory use across library boundaries.

And if even these structures were to be read-only, not using a method for TObject to pass him a Procedure for being called for TObject (like a method TObject.SetAfterCreationProcedure (MyProcedure)). TObject.AfterCreation is like abstract method. It can be never executable for user (it's empty), only for a descendant class who overrides it. So all TClass=class  definitions (without anything) or Tclass  = class (TObject) would do nothing (and empty call if not optimiced by compiler) and a lot of code that could be using this great potential debug facility  is simply evaporated. I can't agree with that.
Even it's worse. AfterCreation is triggered by compiler, not dev. So i cannot accept "The way of doing pascal things" as a dogma. I think it has potential beneficts if redirected by a developer who knows. And it can be done and it can be done even in pascal way (as explained in the example) for only this method accept redirection.

It is simply not the desired functionality to be able to modify TObject's behavior in such a way. You had the example of setting a function variable so that AfterConstruction calls it, but what if two different units want to do that? One could solve this of courrse by either saying that such a function needs to call the previously set function or that there'll be a list of function variables, but what if the behavior of these functions is mutually exclusive? And even if one says that this function variable is write-once then this would leave the problem that the order of unit initialization is not necessarily guaranteed and thus adding or removing a unrelated unit might lead to a complete different behavior of the complete class hierarchy.

So, no, there is no desire to change this.

Then there are the more (philosophical) questions: If FPC would allow dinamically redirects (like C++) or not.

Please highlight where C++ does have dynamic redirects.

2.- by a well-documented feature (for example and in-system call that could change the VMT).

There is such a feature in Delphi that is not yet implemented in FPC: TVirtualMethodInterceptor
However it is rather high level (if you look at this example you might notice what I mean) and also it only replaces the VMT of an instance instead of adjusting the VMT of the whole class type. Though once we implement this as well we could in theory expose the low level functionality as well and make it possible to adjust the VMT of a class type instead of an instance.

[MarkMLl]

The bottom line is that if you want the ability of making unrestricted modifications to low-level classes which are immediately accessible to the entire system you should be using Smalltalk.

And Smalltalk (together with other comparable environments) was even in the early days recognised as a maintenance nightmare since there was no way of determining whether two systems really did have exactly equivalent environments (e.g. on one of them somebody might have redefined the array base index).

Microsoft much later rediscovered that, and embraced it as "DLL Hell" In fact they found ways of extending it, and were never able to extinguish it...

MarkMLl

[damieiro]

    Well, i understand it too, but  i think that .text section (no matter what languaje is used) should the developer's choice if rw or r.

No, because more often then not the OS itself wants to have sections containing code to be read only (especially if some security checks are enabled).

Yes, but we are talking about debugging things and tool specific. Production code will be rip off this. And this is formal case of use and it's supported by linkers (but i haven't got it ). But it's a BAD idea to do read-only sections of .text (see above, we agree) as read-write even on debug (it can conceal other errors of protected access).

 

  Yes, but i dislike it a lot. It won't be portable between OSes. And if this could be avoided by system calls... should not be avoided by user compilation/linker option as a preferred choice?

Then use conditional compilation. I've done the same for the thunk generation code for TVirtualInterface in unit Rtti (see here). If you're working on such a low level you simply can't work around doing OS-specific things.

i will see, but it's same problem with differente aproach. It would ok for all OSes support, not OS-specific. I think it's bad idea.


 

   Well, here i disagree with you. It's a nonsense that one can do (and free pascal documents it) how to make a memory manager (quite lower level) by redirecting all, but doesn't allow for a higher level to redirect Tobject having the middle-level structures to do it easily in methods (like afterconstruction and beforeconstruction that  are really suited for it)

Because the memory manager is something that was also provided by TP and Delphi. Especially the later used it to allow common memory use across library boundaries.

So if delphi and TP would give the TObject facility, will be doing it?.  I think it's not the only way for doing things.

   

And if even these structures were to be read-only, not using a method for TObject to pass him a Procedure for being called for TObject (like a method TObject.SetAfterCreationProcedure (MyProcedure)). TObject.AfterCreation is like abstract method. It can be never executable for user (it's empty), only for a descendant class who overrides it. So all TClass=class  definitions (without anything) or Tclass  = class (TObject) would do nothing (and empty call if not optimiced by compiler) and a lot of code that could be using this great potential debug facility  is simply evaporated. I can't agree with that.
    Even it's worse. AfterCreation is triggered by compiler, not dev. So i cannot accept "The way of doing pascal things" as a dogma. I think it has potential beneficts if redirected by a developer who knows. And it can be done and it can be done even in pascal way (as explained in the example) for only this method accept redirection.

It is simply not the desired functionality to be able to modify TObject's behavior in such a way. You had the example of setting a function variable so that AfterConstruction calls it, but what if two different units want to do that? One could solve this of courrse by either saying that such a function needs to call the previously set function or that there'll be a list of function variables, but what if the behavior of these functions is mutually exclusive? And even if one says that this function variable is write-once then this would leave the problem that the order of unit initialization is not necessarily guaranteed and thus adding or removing a unrelated unit might lead to a complete different behavior of the complete class hierarchy.

So, no, there is no desire to change this.

I agree, but same reasoning could be made for a memory manager. If two units "race" for a memory manager, will crash. So i don't like that one argument is used for one thing, but not for others.
And also, i agree with this, but should be programmer's choice. A compiler cannot prevent all bad coding. It's like if i have a hammer with a plastic head for not making damage in my fingers. Give me a complete hammer, and i will take my gloves and do the job better.


 

  Then there are the more (philosophical) questions: If FPC would allow dinamically redirects (like C++) or not.

Please highlight where C++ does have dynamic redirects.

An example:https://www.usenix.org/legacy/publications/library/proceedings/usenix98/full_papers/hjalmtysson/hjalmtysson.pdf

there are differente ways to to similar things in C++. And there is literature in c++ for dynamic changes. But it shouldn't be the pascal way. Class in pascal inherits from TObject, not from scratch. And debugging from TObject has sense by how its classes implemented in pascal, but the same thing has not the same sense and utility in C++

here is such a feature in Delphi that is not yet implemented in FPC: TVirtualMethodInterceptor
However it is rather high level (if you look at this example you might notice what I mean) and also it only replaces the VMT of an instance instead of adjusting the VMT of the whole class type. Though once we implement this as well we could in theory expose the low level functionality as well and make it possible to adjust the VMT of a class type instead of an instance.

YES! this is the common ground i'm talking. It's a middle-level, full OSs support and could be expanded to class type as a TOOL, not as a way of doing things... And yes, it can do break havoc, but these are for very specific questions.

When i was reading this i wanted to give (to PascalDragon) a big hug full of kisses. Really thanks for all the responses for a (perhaps) too enthusiasthic dev  . This is the behaviour i think will be ok and not all the "tricks" and "patching ideas". It could be done or not by fpc devs, but i won't code hijacks unless Really Really Needed.

The bottom line is that if you want the ability of making unrestricted modifications to low-level classes which are immediately accessible to the entire system you should be using Smalltalk.

No, i'm talking of doing limited access from dev's view on code by restricted means Like TVirtualMethodInterceptor. I do not like to expose the read only memory in production. And even is a bad idea to expose it in this case.
It's, by far the best strategy:
1.- Being higher level
2.- Valid for all OSes.
3.- Cleaner syntax.
4.- Narrow access.
5.- Protected memory
6.- Changes in internal VMT would be supported by changes on the method, no FPC compiler version hell


But this can't be done by developers, because is on compiler ground and his internals and only for FPC devs.

@ToAll: I have learned a lot with this thread. Many thanks..!!

[damieiro]

here is such a feature in Delphi that is not yet implemented in FPC: TVirtualMethodInterceptor
However it is rather high level (if you look at this example you might notice what I mean) and also it only replaces the VMT of an instance instead of adjusting the VMT of the whole class type. Though once we implement this as well we could in theory expose the low level functionality as well and make it possible to adjust the VMT of a class type instead of an instance.

Is there any road or wishlist to ask for this? -> To intercept after/before construction for debugging?