How to store Byte Arrays in a Big List

[Gizmo]

Hi

I have a series of 8 byte values that get put into an 8 byte byte array. I have a function that repeatedly scans for certain values and each time it finds a value, it stores it in the array, obviously replacing what was there from the time before. 

So I need to store these values in a seperate list each time they are found, but I need to store them in their binary byte form, and later on, via another function, I need to then query the list and do lookups.

So, lets say the value 0xFE1200AAAE020405 found on the first scan, and then the next time round 0xAABB0C0104010ABB was found. So my list would contain the binary value of

FE1200AAAE020405
AABB0C0104010ABB

Then later on, I need to qury this list to say "is FE1200AAAE020405 in the list and if so where?" and "is AABB0C0104010ABB in the list and if so where".

I seem to recall TList can be used for this, but the example I have makes use of the storage of 64-bit integer numbers, not 8-byte byte arrays. So I'm unsure if TList can be used for this or not? If so, how? Is it simply MyTList.Add(ArrayVal[0])? Or more complex?   

[molly]

The quickest/easiest i was able to come up with atm:

Code: Pascal  [Select][+][-]

1. program bytearraylist;
2.  
3. {$MODE OBJFPC}
4.  
5. uses
6.   heaptrc, fgl;
7.  
8. type
9.   TMyByteArray = String[8];
10.  
11.  
12. type
13.   TMyByteArrayList = specialize TFPGList<TMyByteArray>;
14.  
15. var
16.   MyByteArrayList : TMyByteArrayList;
17.  
18.  
19. Function MakeByteArray(b1,b2,b3,b4,b5,b6,b7,b8: Byte): TMyByteArray;
20. begin
21.   Result[0] := #8;
22.   Result[1] := Char(b1);
23.   Result[2] := Char(b2);
24.   Result[3] := Char(b3);
25.   Result[4] := Char(b4);
26.   Result[5] := Char(b5);
27.   Result[6] := Char(b6);
28.   Result[7] := Char(b7);
29.   Result[8] := Char(b8);
30. end;
31.  
32. procedure Test;
33. var
34.   i         : integer;
35. begin
36.   MyByteArrayList := TMyByteArrayList.Create;
37.  
38.   for i := 1 to 10 do
39.   begin
40.     MyByteArrayList.Add(MakeByteArray(1,2,3,4,5,6,7,8*i));
41.   end;
42.  
43.   i := MyByteArrayList.IndexOf(MakeByteArray(1,2,3,4,5,6,7,8*3));
44.   WriteLn('8*3 found at index = ', i);
45.  
46.   MyByteArrayList.Free;
47. end;
48.  
49. begin
50.   Test;
51. end.
52.  

PS:you sure you can't simply use a qword/int64 ?

[Gizmo]

Molly

I'm never sure about anything! You've helped me enough to know that by now.

Below is a sanitised paste of the segment of code in question. I can't paste the full thing but it shows what I am trying to do. Basically a procedure reads the raw binary data into 64Kb buffers, which are passed to ScanA. ScanA then looks for a particular 8 byte signature, and where it is found, the 8 bytes that follow it need to be stored into memory in some kind of fast, searchable list, so that later on another function, called ScanB, can search through some other data looking for the values found in ScanA. I could of course just convert the numbers to strings, hold them in a stringlist, and then convert them back to bytes again. But that seems very inefficient and unnecessary.   

Code: [Select]


uses
   ...
  // the fgl unit is for the 64-bit TList implementation TFPGList<Int64>
  fgl,
 
type
  {64-bit TList declaration type, from the fgl unit }
  TListInt64 = specialize TFPGList<Int64>;
 
 
var
MyBinaryList : TListInt64;

procedure ScanA(Buffer : array of byte; NoOfBytes : integer);

...

// ScanA is called by another function that reads the source binary data in 65K buffer reads.
// The buffer is passed to ScanA to search for an 8 byte signature, and the 8 bytes that follow
// the signature.
procedure ScanA(Buffer : array of byte; NoOfBytes : integer);
var
  i, k : integer;
  MyArray : array [0..7] of byte;
  strMyArray : ansistring;

begin
  for i := 0 to NoOfBytes do
  begin
   if PByte(@buffer [i])^ <> $00 then continue;   // these hex bytes are just ficticious values
     if PByte(@buffer [i + 1])^ <> $01 then continue;
       if PByte(@buffer [i + 2])^ <> $02 then continue;
         if PByte(@buffer [i + 3])^ <> $03 then continue;
           if PByte(@buffer [i + 4])^ <> $04 then continue;
             if PByte(@buffer [i + 5])^ <> $05 then continue;
               if PByte(@buffer [i + 6])^ <> $FE then continue;
                 if PByte(@buffer [i + 7])^ <> $FF then continue;

                 // Having found these 8 bytes, the 8 bytes that follow them are of importance.   
// I need to keep them all, as they are found, in a list in memory.
                 for k := 8 to 15 do
                 begin
                   MyArray[k] := Buffer[i+k]; // So the 8 bytes that follow the sig are now inserted into an array one by one
                   strInterfaceID := strInterfaceID + IntToHex(MyArray[k], 2); // And now the 8 bytes are each converted to string and held in a string for display elsewhere

                   // Now I must store the values in MyArray in their binary form into a list for fast lookups
   // because they will be lost the next loop round.
   // No point converting the values to strings, then back to integers\bytes later when they are needed
   // by the next function, ScanB

                   MyBinaryList.Add(MyArray[0]); // this is wrong I think.???
                 end;                 
  end;
end;

initialization
// Create the 64-bit Tlist for storing the important 8 byte sequences
MyBinaryList := TListInt64.Create;



[Thaddy]

Molly: should be packed...

[Zoran]

Pascal AnsiString (from fpc 3.0, use RawByteString) is very good type to chose for storing array of bytes - you can search it char by char (which means byte by byte) and you can store any byte inside it.

[molly]

@gizmo:
The reason to ask for being able to use 64-bit integer is that it makes things easier as you are able to cast data as an int64 (or qword).

Thank you very much for your snippet as that make things more clear.

MyBinaryList.Add(MyArray[0]); // this is wrong I think.???

Well not wrong. Simply adding a bytevalue that is converted to int64 :-)

Code: [Select]

Var
  MyTempInt64: int64;

 ...
  MyTempInt64 := PInt64(@Buffer[i+8])^;
  MyBinaryList.Add(MyTempInt64);
 ...

As you might notice, you can apply this directly to your scanbuffer, as well as copy the items first into your own array (if that is what you really wish).
(Note that the storage into MyTempInt64 is not really required. It is just there to make things more clear)

In case the latter wanting to use MyArray, consider the following snippet:

Code: [Select]

var
  MyArray   : packed array[0..7] of byte;
  MyInt64    : Int64 absolute MyArray;
In which case you can use the MyInt64 variable instead of having to cast manually, e.g. you can directly feed that to the add() method of your list.

edit: PS: I have no idea where your data is originating from but please be aware of the endianess of the read vs stored data.

@Thaddy, i think you  meant gizmo :-) Either way the packing alone is not going to help him, see my reply as above.

[Gizmo]

Molly

Your help and tips are, as ever, awesome.

In between my last reply and your latest response, I was instead working on a different approach of recording the offsets in the TList of where the 8 byte ranges were found, with the intention of then going through that list, and for each recorded offset, navigating to that in the source data and then reading a byte range from each value. I nearly got that working but then read your reply which will serve me better long term.

And your solution looks better!

So, now what I have is something like this :

Code: [Select]

for k := 8 to 15 do  // Read the 8 bytes that follow the first 8 byte signature
  begin
    MyArray[k] := Buffer[i+k]; // So the 8 bytes that follow the sig are now inserted into an array one by one
    strInterfaceID := strInterfaceID + IntToHex(MyArray[k], 2); // And now the 8 bytes are each converted to string and held in a string for display elsewhere
 end;

So I have a packed byte array (MyArray) with 8 addressable portions; for example, the following byte values in it : 02 08 74 FF FE B0 0B 0B

And thanks to your tip Molly, I can now add that to my TList as an integer as follows if I need to :

Code: [Select]

MyBinaryList.Add(PInt64(@MyArray[0])^);

And my list contains, for example, the value 795,924,368,415,000,578, which is indeed the 64-bit integer val of 0x020874FFFEB00B0B

So that's all good.

HOWEVER, what I actually need to store and lookup later on are THE BYTES. The 64-bit value of these 8 bytes is not what I need to record or use. Each of the 8 bytes are, in and of themselves, individually important. That's why I needed to use a byte array so that I can examine\test\query MyArray[X]. Sorry if I didn't explain that terribly well and perhaps it doesn't matter anyway if you know a way? By putting them in the TList, I lose the bytes and get just a long 64-bit integer. And as you identify Molly...Endianess comes into play when using anything more than 2 bytes, whereas for individual bytes, it does not.

What I need to be able to do is to store the binary representation of the 8 seperate bytes (such as 0x020874FFFEB00B0B and many more like it) in a list.

So, the first scan, MyArray will contain

Byte 0 holds : 02 = 2
Byte 1 holds : 08 = 8
Byte 2 holds : 74 =116
Byte 3 holds : FF = 255
Byte 4 holds : FE = 254
...
Byte 7 holds : 0B = 11

So I then need the content of MyArray to be recorded in a list (like TList, MyBinaryList) before the content gets overwritten with new data for the second pass, and when that second pass takes place, the next sequence of bytes need to be recorded in the TList.
 
Then, later on, I need to be able to go to the list, and for each entry, I need to be able to go down the list and ask for Byte 0, Byte1, Byte 2 and so on of each line in it. The equivalent of asking MyByteArray every time it is populated, except I need to do it later. So from line one of my list (using the example above) when I ask for the equivalent of MyByteArray[3], I need to get 255 (0xFF) returned. When I ask for MyByteArray[7] I need to get 11 (0xFF) returned.

I'm kind of confusing myself now, but I hope that makes some sense? 

[molly]

In between my last reply and your latest response, I was instead working on a different approach of recording the offsets in the TList of where the 8 byte ranges were found, with the intention of then going through that list, and for each recorded offset, navigating to that in the source data and then reading a byte range from each value. I nearly got that working but then read your reply which will serve me better long term.

as a matter of fact i was thinking of suggesting that but, i did not know if your data was static or not. In case static, it might be quicker to store the offsets (or pointers).

In case your data is static you also might want to consider using a stream instead of muddling with passing along byte arrays. You could then use pointers instead (but perhaps you wanted to avoid that).

Truth is, every solution (and its usefulness) depends on how the data is stored/structured and as such there are many approaches possible.

HOWEVER, what I actually need to store and lookup later on are THE BYTES.

I understand. It is just a cast the other way around ? :-)

In case you really need to have each indivisual byte accesible (instead of converting to Int64 and comparing) then you can cast the other way around.

Admittingly, the use of the list is then becoming a bit awkward as you can't simply use the IndexOf() method. In that case you could consider adding your own custom compare/search method to the list if you like. But there is nothing preventing you to do things 'outside' your list (in fact it is a good exercise to see what would be required to implement such a method for the list, if you wish to do so later on).

You could perhaps consider the following approach:

Code: [Select]

Var
 i : Integer;
  StoredByteArray : array[0..7] of byte;
  StoredInt64 : absolute StoredByteArray;
begin
  for i := 0 to Pred(MyByteArrayList.Count) do
  begin
    StoredInt64 := MyByteArrayList.Items[i];

    If StoredByteArray[0] = 3 then
    If StoredByteArray[1] = 8 then
    If StoredByteArray[2] = 116 then
    If StoredByteArray[3] = 255 then
   etc...
  end;
end;

An alternative method could be to use type int64rec for the cast:

Code: [Select]

Var
  StoredByteArray : array[0..7] of byte;
  StoredInt64 : absolute StoredByteArray;
begin
  for StoredInt64 in MyByteArrayList do
  begin
    if (Int64Rec(StoredInt64).Bytes[3] = 255)) then
    begin
      // do something important
    end;
    // or:   
    if (StoredByteArray[3] = 255) then
    begin
      // do something important
    end;
    etc....
  end;
end;

Many ways that lead to Rome :-)

edit: oops, copy-paste error corrected in first example

[molly]

Code: [Select]

for k := 8 to 15 do  // Read the 8 bytes that follow the first 8 byte signature
  begin
    MyArray[k] := Buffer[i+k]; // So the 8 bytes that follow the sig are now inserted into an array one by one
    strInterfaceID := strInterfaceID + IntToHex(MyArray[k], 2); // And now the 8 bytes are each converted to string and held in a string for display elsewhere
 end;


I believe an error is in there ?

Assuming MyArray is an array containing 7 bytes (and yes i am aware you might have declared that as MyArray: array[8..15] of byte) then i believe that is wrong, consider the following:

Code: [Select]

type
  TMyByteArrayList = specialize TFPGList<Int64>;

  T8Bytes          = packed array[0..7] of Byte;
 
var
  MyByteArrayList  : TMyByteArrayList;
  RandomBytes      : Packed array of Byte;


procedure ScanA(Buffer : array of Byte; NoOfBytes : integer);
var
  i, k       : integer;
  MyArray    : T8Bytes;
  MyInt64    : Int64 absolute MyArray;
  strMyArray : ansistring;
  strInterfaceID : String;
begin
  Writeln('scanning buffer from ', Low(Buffer), ' to ', High(buffer));
  for i := Low(Buffer) to High(Buffer) - Length(MyArray) do
  begin
    if (Buffer[i+0] <> $6f) then continue;
    if (Buffer[i+1] <> $63) then continue;
    if (Buffer[i+2] <> $65) then continue;
    if (Buffer[i+3] <> $64) then continue;
    if (Buffer[i+4] <> $75) then continue;
    if (Buffer[i+5] <> $72) then continue;
    if (Buffer[i+6] <> $65) then continue;
    if (Buffer[i+7] <> $20) then continue;
   
    strInterfaceID := '';
    // k is an index to an array with indexes from 0 to 7 (and not 8 till 15)
    for k := Low(MyArray) to High(MyArray) do
    begin
      MyArray[k] := Buffer[i+8+k];  // note the difference, the 8 offset is only required for the buffer.
      strInterfaceID := strInterfaceID + IntToHex(MyArray[k], 2);
    end;
    MyByteArrayList.Add(MyInt64);
  end;
end;
Please forgive the byte sequence values i've used (i had to come up with some random data ;-) ).

Edit: And a small working example (hopefully it is enough situated in the spirit of your program):

Code: [Select]

program ByteArrayLister;

{$MODE OBJFPC}

uses
  heaptrc, fgl, classes, SysUtils;

type
  TMyByteArrayList = specialize TFPGList<Int64>;
  T8Bytes          = packed array[0..7] of Byte;
 
var
  MyByteArrayList  : TMyByteArrayList;
  RandomBytes      : Packed array of Byte;

{*
procedure SeeminglyRecognized()
procedure AnotherTest;
procedure WhyAreWeHere?
procedure Answer:
procedure SoWeCanTestThingsBetterThisWay()
*}

procedure ScanA(Buffer: array of Byte; NoOfBytes : integer);
var
  i, k           : integer;
  MyArray        : T8Bytes;
  MyInt64        : Int64 absolute MyArray;
  strMyArray     : ansistring;
  strInterfaceID : String;
begin
  Writeln('scanning buffer from ', Low(Buffer), ' to ', High(buffer));
  // beware of buffer shortage.
  // There must be overlap consisting of the extra bytes you read after recognition + the number of recognition bytes.
  // so the endvalue used below is (also) not entirely 100% correct (the example does not suffer from it unless you know how to make it crash).
  for i := Low(Buffer) to High(Buffer) - Length(MyArray) do
  begin
    // Oh boy, what would we be looking for ? :-)
    if (Buffer[i+0] <> $6f) then continue;
    if (Buffer[i+1] <> $63) then continue;
    if (Buffer[i+2] <> $65) then continue;
    if (Buffer[i+3] <> $64) then continue;
    if (Buffer[i+4] <> $75) then continue;
    if (Buffer[i+5] <> $72) then continue;
    if (Buffer[i+6] <> $65) then continue;
    if (Buffer[i+7] <> $20) then continue;
   
    strInterfaceID := '';
    // k is an index to an array with indexes from 0 to 7 (and not 8 till 15)
    for k := Low(MyArray) to High(MyArray) do
    begin
      MyArray[k] := Buffer[i+8+k];  // note the difference
      strInterfaceID := strInterfaceID + IntToHex(MyArray[k], 2);
    end;
    MyByteArrayList.Add(MyInt64);
  end;
end;

Procedure SimulateRandomBytes;
const
  FileName = 'ByteArrayLister.pas';
var
  FS : TFileStream;
  i  : LongInt;
begin
  FS := TFileStream.Create(FileName, fmOpenRead);
  FS.Position := 0;

  SetLength(RandomBytes, FS.Size);
  FS.ReadBuffer(RandomBytes[0], Length(RandomBytes));
 
  FS.Free;
end;

procedure ProcessData;
var
  i           : integer;
  StoredArray : T8Bytes;
  StoredInt64 : Int64 absolute StoredArray;
begin
  // find in list all procedures starting with "S" and print their names
  // of course we have only stored 8 bytes in the list so some names may end
  // unexpectedly.
  for StoredInt64 in MyByteArrayList do
  begin
    if (StoredArray[0] = $53) then
    begin
      Write('Found something = ');
      for i := Low(StoredArray) to High(StoredArray) do
        // only write out valid characters
        if (StoredArray[i] in [$41..$5A, $61..$7A])
        then Write(Char(StoredArray[i]))
        else break;
      Writeln;
    end
    else
    begin
      Write('skipped something = ');
      for i := Low(StoredArray) to High(StoredArray) do
        // only write out valid characters
        if (StoredArray[i] in [$41..$5A, $61..$7A])
        then Write(Char(StoredArray[i]))
        else break;
      Writeln;
    end;
  end;
end;

begin
  Writeln('simulating : ');
  SimulateRandomBytes;

  Writeln('create : ');
  MyByteArrayList := TMyByteArrayList.Create;

  Writeln('scanning : ');
  ScanA(RandomBytes, Sizeof(RandomBytes));

  Writeln('processing : ');
  ProcessData;

  Writeln('free : ');
  MyByteArrayList.Free;
  SetLength(RandomBytes,0);

  Writeln('done : ');
end.


[Gizmo]

Hey Molly

As you're being good enough to help in such detail, I'll give more detail that may help.

The program I am making reads large (several Gb) raw binary files into buffered segments (boundary issues accepted).

One of the things I need it to detect are IPv6 addresses. To do this, I have to pass over the data in its entirity 3 seperate times.

The first pass looks for 0xFE80000000000000 which is called the Local Unicast Prefix. Wherever it finds that, it then reads the next 8 bytes that follow it, which is the unique Interface ID. That is pass one, and the Unique Interface ID is what I store in the array, called here MyArray. That is why the second for loop reads

k := 7 to 15 do

because it's reading ahead 8 bytes from the first 8 bytes.

It is that populated byte array that then needs to be passed to PassB. The Local Uni Prefix is not actually needed anymore but for now I record its existance.

So, PassB then has to read the data again from start to end but this time, in order for it to be a valid ID number, it MUST NOT be prefixed by 0xFE80.... (it's just stuff to do with the IPv6 structure...valid addresses will not be have the LUP immediately before the InterfaceID). So the second pass is just to look for the ID (so searching for the byte sequence held in MyArray) that was found in the first pass in any other places in the file where it is not prefixed by FE8000....Thats why, for a moment, I took the stance of using the offsets of where the InterfaceIDs were found, but that won't work because we know that the offsets found from Pass1 relate to where THEY ARE prefixed by FE80, not where they are NOT prefixed by it. So, ultimately, I need the InterfaceID Byte array in my list and I need to search for where it is and only pull out the instances where it isn't prefixed by FE80... 00

Then, once those offsets are found, another pass is needed but that's another story! At some stage, I expect the values from all 3 passes to be stored in in6_addr records (http://www.freepascal.org/docs-html/rtl/sockets/in6_addr.html) where they can be displayed as strings.

So hopefully you see now why I need that byte array, and I need to be able to store their existance detected in PassA for use by PassB and PassC

That

[jack616]

Just an option as you seem to be having trouble - I don't know what speed you need or system resources
but have you considered just converting to strings - using a stringlist and
string functions? Maybe that would be easier to work with?

[Gizmo]

I have indeed considered that, yes, and, as you say, it would be easier, but it just feels wrong. I'd rather battle my way through the hardships to get an efficient result than accept an inefficient result just for simplicity. 

[jack616]

actually - it's pleasing to hear that. 

[User137]

If you are dealing with binary files directly, where do those hex values come from? If it's 0xFF i'd just call it 255, 0xFE = 254 and 0x80 = 128 byte values. I wouldn't think this with "string" mindset, which most of the solutions in the thread aren't. Also while reading through the binary files, if you know for certain that some sections are not going to have the data you need, seek past them. It would make the program slower if it actually puts all the data in the file in temporary memory.

[molly]

As you're being good enough to help in such detail, I'll give more detail that may help.

I'm honoured 

Alas, you haven't revealed too much details. As a matter of fact you told one additional detail (that of the LUP not being allowed to match as prefix on the second run) 

But, it did indeed made things a bit more interesting to work with ...

....To do this, I have to pass over the data in its entirity 3 seperate times.

Are you absolutely positively 100 percent sure about that ?

I ask, because proper data analysis and interpretation of the data before-hand (doing it right) usually reveals a few things that allow you to take/choose a different route. Not saying that you are doing it wrongly but afaict there would be a lot going on in those data-files you are scanning. Alas, i do not have the full specs of your data in front of me so can only take a uneducated guess here. see also (1)

That is why the second for loop reads

k := 7 to 15 do

because it's reading ahead 8 bytes from the first 8 bytes.

Yups, i fully understand that it needs the 8..15 for the index to the Buffer, but not for MyArray. Hence my remark.

MyArray (at least in the examples shown sofar) is an array indexed from zero to 7, not from 8 to 15 (assuming you made a typo there with 7).

E.g. MyArray[0..7] <> MyArray[k] (which runs from 8 to 15)

But perhaps i am wrong and in your code MyArray[] is actually indexed from 0..15 ?

So the second pass is just to look for the ID (so searching for the byte sequence held in MyArray) that was found in the first pass in any other places in the file where it is not prefixed by FE8000....Thats why, for a moment, I took the stance of using the offsets of where the InterfaceIDs were found, but that won't work because we know that the offsets found from Pass1 relate to where THEY ARE prefixed by FE80, not where they are NOT prefixed by it.

Yups, i fully understand. But, at first sight that was not clear to/for me (if it had been clear i would not have suggested it). See also (1)

So, ultimately, I need the InterfaceID Byte array in my list and I need to search for where it is and only pull out the instances where it isn't prefixed by FE80... 00

Yups, and it is not that different from my shown example   

Then, once those offsets are found, another pass is needed but that's another story! At some stage, I expect the values from all 3 passes to be stored in in6_addr records

Well, it shouldn't be _that_ hard to do 

The example as shown already implements pass one (ScanA) and needs some accommodation for pass two (ProcessData). Point is, the basics are the same.

So hopefully you see now why I need that byte array, and I need to be able to store their existance detected in PassA for use by PassB and PassC

Well, not exactly 

You have the ID's stored inside the int64list. This list is populated until eof() on the first pass (see also the example i posted)

The second pass, should basically do the same scan as the first, except that you _don't want to have the LUP in the prefixbuffer. Ergo every prefix buffer that matches LUP means skip. In case not matching LUP, you walk through the int64list to see if any of them listed matches the bytes inside your ID buffer.

So, i do not really see why you would have the need to pass along MyArray directly to the second pass, unless (and of course) i have interpreted something wrongly from your descriptions.

(1)
What are you actually trying to do ? analyzing raw wireshark dumps or something ?

That is what i meant to say earlier with proper analyzing the data first. Are you aware that in case you are scanning data-packets that they occur in a particular order (depending on the protocol) ? That data-packets have a particular header and length, etc. ? In which case, the more information you can extract before-hand the quicker you are able to scan through the data. Even if that means you do not need or are interested in all that data/information.

I have indeed considered that, yes, and, as you say, it would be easier, but it just feels wrong. I'd rather battle my way through the hardships to get an efficient result than accept an inefficient result just for simplicity. 

As a matter of fact, in practice it seems much easier doing so without using strings (even though i started out to suggest using a generic string list based on the initial description).

As long as you are able to 'abuse' any-which-type that is able to 'store' your data and is the smallest (= most efficient) in size (and easiest to work with) has preferences over storing things inside strings. I am aware that this is not true for some other languages out there but, we are currently not dealing with those. We use Pascal here 

So, if you have 8 bytes, the first possible candidate to 'abuse' is qword and/or int64. It doesn't even require records or any of that. Together with the absolute keyword you can manipulate things your way as that can do exactly those things that you seem to want (e.g. treating an array of 8 bytes as int64 and, the other way around of course).

The positive thing about using strings is that their size/length isn't set in stone (so can easily be adapted using 16 byte arrays, or even larger) and that there already a lot of routines available for manipulating strings. But a few move's and copy's here and there will do the job just as good 

BTW: the most efficient way is that of actually having working code