[SOLVED] Specification of the binary serialization format ?

[guest58172]

Hi, where can I find the specifications of native binary serialization format ? ("component streaming" if you prefer this naming).

I use the text format in several performance-sensitive contexts and I'd like to get rid of the ObjectTextToBinary convertion especially since the serialization is done from the outside (so there's already too much streams involved: external process output -> read output in str1 -> convert str1 in str2 -> destream str2 in target component instance.)

[guest58172]

E.g I do this in D:

Code: D  [Select][+][-]

1. static if (Fmt == ListFmt.Pas)
2. {
3.     pasStream.put("\ritem\r");
4.     pasStream.put(format("line=%d\r", dt.name.line));
5.     pasStream.put(format("col=%d\r", dt.name.column));
6.     pasStream.put(format("name='%s'\r", dt.name.text));
7.     pasStream.put("symType=" ~ symbolTypeStrings[st] ~ "\r");
8.     static if (dig) if (deep)
9.     {
10.         pasStream.put("subs = <");
11.         dt.accept(this);
12.         pasStream.put(">\r");
13.     }
14.     pasStream.put("end");
15. }

to write something that can be loaded in a TComponent (you'll even see that's a TCollection that's written here...). I'm looking for the specifications of the binary format. , to gain a few millisecs at each execution...

[engkin]

You most likely have a better chance finding what you want if you see how the RTL saves TComponent in binary form:

Code: Pascal  [Select][+][-]

1. //\rtl\objpas\classes\writer.inc
2. procedure TWriter.WriteComponentData(Instance: TComponent);
3. ...
4.   FDriver.BeginComponent(Instance,Flags,FCurrentPos);
5.   If (FAncestors<>Nil) then
6.     Inc(FCurrentPos);
7.   WriteProperties(Instance);
8.   WriteListEnd;
9.   // Needs special handling of ancestor.
10.   If not IgnoreChildren then
11.     WriteChildren(Instance);
12. ...
13.  

For an example of FDriver check TBinaryObjectWriter in \rtl\objpas\classes\classesh.inc


You'll get more answers, if you have a specific problem that you can demonstrate in code.

[marcov]

The D way you show is the "old" way, like it was done in Pascal in Turbo Vision times (before say 1995).

Delphi streaming relies heavily on the RTTI of properties, which is why general methods like WriteProperties (in Engkin's post) can stream them without class specific code.

Maybe, very maybe you could cache some of the RTTI querying, but I don't expect much benefit.

[guest58172]

The D side of things doesn't matter, it's done like that because it hasn't RTTI like Object Pascal (it has compile time reflection), so I write the stuffs by hand (it could be done automatically but then the whole structure of the collection would have to be reproduced, which would be slower in ths particular case, i.e visitor pattern: i can write directly without building a tree of aggregates).

Anyway, so no document exist.

[Thaddy]

Anyway, so no document exist.

Well, in such low level cases the sourcecode itself and the comments in the sourcecode are the documentation.
These are usually considered implementation specifics of a higher level functional requirement specification.
Also note that because of that such low-level formats are subject to change w/o warning so is highly specific to an FPC  version.
So what you gain in performance may lead to extremely difficult to maintain code.
The recommended way is through RTTI which is documented, but also highly subject to change atm.
The format can be deduced from the RTTI related code.

I usually do what you do: when speed is a requirement I write my own streaming for my own specific classes. Usually separating just state and content/payload.