[SOLVED] Help reading .tmx file

[pbk]

Hello

This is a question from a rank amateur. I am using Lazarus version 1.0.10 on a computer running Windows 7, 64-bit. About 30 years ago I took some programming courses and was completely taken by Pascal, so I decided to try my hand at using it to do a task at hand. I’m pretty sure I can figure out how to handle the data manipulation, but I am stumped by file operations.

I want to read a very large *.tmx file (ca. 1.3 million lines) with over 100k records that look roughly as follows:

/begin sample input
<tu usagecount="3" lastusagedate="20130219T195424Z" creationdate="20130219T113048Z" creationid="PBK">
<prop type="Att::Client">BogusClientName</prop>
<prop type="Att::Subject">Experimental Economics, Economics, Insurance</prop>
<prop type="Txt::Job ID">13-009</prop>
<prop type="Txt::ProjectName">Demande d_assurance papier Final.doc</prop>
<tuv xml:lang="FR-CA">
<seg>Au plan expérimental, nous avons montré que les agents riscophiles se démarquent très significativement, dans leurs choix assurantiels, des agents riscophobes, puisque des variations du prix ou du coût fixe d’assurance n’ont pas les mêmes impacts sur les deux types de population.</seg>
</tuv>
<tuv xml:lang="EN-CA">
<seg>At the experimental level, we have observed that risk-loving individuals differ significantly from risk-averse agents in their insurance choices, because changes in the price or the fixed cost do not have the same impact on the two groups.</seg>
</tuv>
</tu>
/end sample input

This, as you see, this is in the xml family. Not all records have all the same fields. I hope to be able to read all the records into Pascal Record type variables, then filter and generate various reports. I am stumped by the declarations I need to create a handle to read an xml file type and the variables / procedures necessary to get this information into records. I have been unable to find documentation online that is sufficiently detailed in addressing this particular issue.

Any help would be very much appreciated.

Paul

[jwdietrich]

Do you need to evaluate every tag of the XML file or only parts (e.g. the tuv and seg tags)?

And did you read the articles in the wiki category XML? Most of them are very helpful.

[pbk]

The information I need is the field name (i.e. Job ID, or Subject) and the corresponding field value (i.e. 13-009, or economics). I don't think I will ever need to edit the .tmx file, but I need the data. For example, only a handful of the records have a Subject field, and those that do can have one or more values (separated by commas). I want to find the records that have Subjects, associated them with the corresponding Job ID, and create a database (I'm developing that in Access) in which they are associated with Subjects (many-to-many). In general, I want to extract info that should be Job level and put it somewhere useful.

I had seen some of the articles that you refer to, and had read the xml tutorial and tried to run some of the sample code. However, my ignorance is too great. For example, I know that I need 'uses,' but am not sure what components I need to use, why, and what to do next. I assume that these provide type definitions, but don't know that. Also, in the xml-world I am not sure what types do what, and how to use the variables they define. What I would find most useful would be an example of code that does something like what I'm trying to do.

Thanks for your feedback

[jwdietrich]

Do you have a smaller version of your tmx file at hand? A version that is much smaller but contains about three examples with the relevant tags would be helpful...

[pbk]

Here:

\begin sample text
<?xml version="1.0" ?>
<!DOCTYPE tmx SYSTEM "tmx14.dtd">
<tmx version="1.4">
<header
    creationtool="TRADOS Translator's Workbench for Windows"
    creationtoolversion="Edition 8 Build 835"
    segtype="sentence"
    o-tmf="TW4Win 2.0 Format"
    adminlang="EN-US"
    srclang="FR-CA"
    datatype="rtf"
    creationdate="20120828T184721Z"
    creationid="PBK"

    [SNIP]

    <prop type="Name">cf2ce</prop>
</header>

<body>
<tu usagecount="3" lastusagedate="20130219T195045Z" creationdate="20130219T153636Z" creationid="PBK">
<prop type="Att::Client">client1</prop>
<prop type="Att::Subject">Experimental Economics, Economics, Insurance</prop>
<prop type="Txt::Job ID">13-009</prop>
<prop type="Txt::ProjectName">Demande d_assurance papier Final.doc</prop>
<tuv xml:lang="FR-CA">
<seg>Aversion au risque</seg>
</tuv>
<tuv xml:lang="EN-CA">
<seg>Risk averse</seg>
</tuv>
</tu>

<tu creationdate="20041110T014437Z" creationid="pbk" changedate="20041111T231111Z">
<tuv xml:lang="FR-CA">
<seg>Nous montrons en fait que, dans une optique de long terme, les entrepreneurs, pourtant mieux informés que le public, peuvent avoir intérêt à de telles limitations.</seg>
</tuv>
<tuv xml:lang="EN-CA">
<seg>We shall demonstrate that, from a long-term perspective, entrepreneurs may well benefit from these types of limitations, even though they are better informed than the public.</seg>
</tuv>
</tu>

<tu usagecount="1" lastusagedate="20100427T211536Z" creationdate="20100426T223853Z" creationid="pbk" changedate="20100427T211559Z">
<prop type="Att::Client">client2</prop>
<prop type="Txt::Job ID">10-015</prop>
<prop type="Txt::ProjectName">35198</prop>
<tuv xml:lang="FR-CA">
<seg><it pos="end">&lt;/cf&gt;</it><ph>&lt;cf size=&quot;18&quot;&gt;</ph>Terrain :</seg>
</tuv>
<tuv xml:lang="EN-CA">
<seg><it pos="end">&lt;/cf&gt;</it><ph>&lt;cf size=&quot;18&quot;&gt;</ph>Field:</seg>
</tuv>
</tu>

</body>
</tmx>
\end sample text

There are pages of junk in the header that I will just delete before reading the file (or tell the program to ignore).
These three records are from various periods, over time I became more conscientious in filling out the fields.

[jwdietrich]

OK, I played a bit around with your TMX file and the XML routines in the FCL. The attached sample solution implements a very simple TMX parser. It has a very plain user interface. After clicking the button "Read file..." you may select a tmx file in your file system. Its raw contents are shown in the memo field in the window's top. By clicking "Parse and build tree" the XML file is parsed and visualized in a tree in the bottom left of the window. As you wanted to read field names they are extracted and shown in the Grid in the lower right of the window.

The main parsing routines are:

Code: [Select]

function GetNodeAttributesAsString(pNode: TDOMNode): string;
{Deliver all XML node attributes as string}
var
  i: integer;
begin
  Result := '';
  if pNode.HasAttributes then
    for i := 0 to pNode.Attributes.Length - 1 do
      with pNode.Attributes[i] do
        Result := Result + format(' %s = "%s"', [NodeName, NodeValue]);
  Result := Trim(Result); {remove leading and trailing spaces}
end;

procedure ParseTMXFile(Node: TDOMNode; TreeNode: TTreeNode);
{Process a node and all child nodes}
begin
  if node = nil then exit; {exit procedure if all nodes have been processed}
  TreeNode := Form1.XMLTreeView.Items.AddChild(TreeNode, Trim(Node.NodeName + ' ' + GetNodeAttributesAsString(Node) + Node.NodeValue));
  Node := Node.FirstChild;
  while Node <> nil do  {repeat as long as there are contents}
  begin
    if Node.NodeName = 'prop' then  {Add contents of prop tag to ValueListEditor}
    begin
      Form1.FieldListEditor.Strings.Add(format('%s=%s',[Node.Attributes[0].TextContent, Node.FirstChild.TextContent]));
    end;
    ParseTMXFile(Node, TreeNode);  {recursive call enabling analysis of next tag}
    Node := Node.NextSibling;  {next tag}
  end;
end;

ParseTMXFile is a routine that is called recursively. It continues to process XML tags and their respective child tags until the end of the file is reached. GetNodeAttributesAsString extracts the attributes of a XML tag and returns them as string.

If ParseTMXFile finds a prop tag its contents are read and inserted into the grid.

You might want to assign the results to a variable instead or insert them into a database.

[pbk]

Wow, this is amazing. Thank you very much. I've been cobbling together some routines for working with this data, in the hope I'd be able to read it. I'm going to have to spend a little time with this, but I'm really grateful because I could not have created it.

[pbk]

I had to focus on some other stuff, so was not able to get back onto this right away.

I could not figure out how to return the values from the unit to the main program. Eventually, I realized that I could incorporate the procedures from the unit into my program, and have the variables declared and available right there. That works fine. So, thanks again for your immense help.

PS I'd close this thread and mark it "solved," but don't know how!

[BigChimp]

No need to close it, really, though apparently there is some way to disallow further posts to a thread - which I find very annoying though..

You can mark the thread solved by editing the title of your first post and inserting [SOLVED] at the beginning...

Glad it works for you