Lazarus

Programming => Operating Systems => macOS / Mac OS X => Topic started by: Josh on June 19, 2019, 11:24:33 pm

Title: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Josh on June 19, 2019, 11:24:33 pm
Hi
I have been battling with getting midi working with Cocoa.
Code that works fine on Carbon; give incorrect data error on Cocoa.

I have tracked down a problem; that I think is causing it; but do not know the best way to correct what I see as a BUG in the Cocoa Implementation of CoreMidi

A midi Data Packet is a record
timestamp:uInt64;
length:uInt16;
data:array[0..255] of byte

The record structure has a size of 268bytes on carbon; where it shows the correct information; but on cocoa the exact same structure give a size of 272bytes.
A similar issue exists with MidiPacketList which has 272(carbon) and 280 (cocoa).

Could this be a memory allignment issue?

Anyone have an idea of how to fix; as core midi and other mac frameworks exists; it is possible that the same could exists within these aswell.

Hope the above makes sense.

Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: engkin on June 19, 2019, 11:41:57 pm
use packed record.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Josh on June 20, 2019, 12:21:06 am
Hi

I have tried using packed
Code: [Select]
MIDIPacket = packed record
timeStamp: MIDITimeStamp;
length: UInt16;
data: array [0..255] of Byte;
end;                               

But the size is still 272 bytes.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: engkin on June 20, 2019, 12:25:42 am
Your first post shows:
timestamp:uInt64;

Your last post shows:
timeStamp: MIDITimeStamp;

Is MIDITimeStamp of uInt64 type? if not, replace it with uInt64.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: engkin on June 20, 2019, 12:27:19 am
Wait!!
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: engkin on June 20, 2019, 12:32:25 am
That was not packed, sorry. Try this:
Code: Pascal  [Select][+][-]
  1. {$push}
  2. {$PackRecords 4}
  3. type
  4.   TMIDIPacket = record
  5.     timestamp:uInt64;
  6.     length:uInt16;
  7.     data:array[0..255] of byte;
  8.   end;
  9. {$pop}

This should be 268 bytes.
Packed would be 266 bytes.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Josh on June 20, 2019, 01:03:49 am
hi

Code: [Select]
{$push}
{$PackRecords 4}
MIDIPacket =  record
timeStamp: UInt64;
length: UInt16;
data: array [0..255] of Byte;
end;
{$pop}       

The above code has no effect on cocoa; it is still 272bytes...
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: engkin on June 20, 2019, 03:18:07 am
Probably your target is 64bit. Back to packed records:
Code: Pascal  [Select][+][-]
  1. type
  2.   MIDIPacket = packed record
  3.     timestamp:uInt64;
  4.     length:uInt16;
  5.     padding:array[0..1] of byte;
  6.     data:array[0..255] of byte;
  7.   end;
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Thaddy on June 20, 2019, 07:28:51 am
The last is correct indeed. Midi streams need packed records and {$packrecords 1} not 4! There can not be or should not be any slack spacing.
This has nothing to do with alignment, but with reading/writing the stream correctly. Stored data should always be packed and Midi is no exception.

IOW: byte alignment of the members. The in-memory record start itself will always be aligned to current alignment so there is no speed disadvantage in any way.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Josh on June 20, 2019, 11:23:26 am
Hi

Something odd is going on, on my end.

If I create a midipacket type I my application unit; with pack record 4, then the structure size is correct at 268.

If I use unit MACOSALL; which uses MIDISERVICES definition, then the size is incorrect.
If I modify the MIDIServies definition it still is not correct.

Now if I make a syntax error in the MIDIservices definition; expecting a compile error; it does not and still compiles. So I then done a clean all build; and still it compiles .

So question now is how can I modify midiservices.

I have attached a project that show the issue, you can comment out the define statement to use either macosall definition or its own.

Hopefully someone can replicate.

Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Thaddy on June 20, 2019, 12:16:32 pm
Josh, read my explanation above: $packrecords 4 is wrong. You MUST use 1, which is the default. Think!!!!!!
(note I am an avid audio and midi programmer)
Record packing does not influence alignment of the record itself. <sigh>
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Josh on June 20, 2019, 12:40:14 pm
Hi Thaddy,

With carbon, the midipacket in macosall;midiservices is a size of 268.
So for it to work with the same in cocoa and carbon the midipacket size must be the same ie 268.

packrecords 1 create a size of 266  which is wrong
packrecords 4 creates a size of 268
packrecords 8 creates a size of 272
without packrecords a size of 272

If I use Align 1; i still get 266

I see similar problem with midipacketlist type as well..

If the size is not the same in carbon and cocoa should it be placed in bugtracker; as the issue is with coremidi implementation.


Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: rvk on June 20, 2019, 01:03:33 pm
According to this the MIDIPacket and MIDIPacketList should have the 4 byte alignment.

Quote
Setting {$Align 1} for the whole file was not the correct solution. Instead only the two records (MIDIPacket and MIDIPacketList) should have 4 byte alignments instead of 8 byte alignment.
https://stackoverflow.com/a/37748973/1037511

Quote
Note the #pragma pack(push, 4) above the definition of MIDIPacket in the CoreMIDI.h header file.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Thaddy on June 20, 2019, 03:36:03 pm
I am sorry but that is the wrong answer. The alignment is 1, not 4. I am writing music software for the past 20 years. There's no padding.
In Freepascal the record in-memory will be properly aligned. The stored records are packed to 1. The reason that 4 works too is because that is the minimum size of the record header members.
See also: https://en.wikipedia.org/wiki/MIDI#Technical_specifications

Just to be sure: the midi format is completely platform independent.

That "approved"  stackoverflow entry is simply wrong! You can test that by simply opening a midi file as a stream.
Which, btw, is its intended use.

(some people even stole my code - the reverb - without proper representation of its origins: Jezar's Freeverb 3, translated and optimized by me and stolen by Christian!, that's not midi, but... he still did not correct that even after asking for it politely. That's OK TobyBear and me know who wrote it...Stays freeware anyway... But I still hold a grudge because of misrepresentation, so Christian if you read this, put it right after all those years)
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: skalogryz on June 20, 2019, 04:29:57 pm
Quote from: josh on June 20, 2019, 11:23:26 am
Something odd is going on, on my end.

If I create a midipacket type I my application unit; with pack record 4, then the structure size is correct at 268.

If I use unit MACOSALL; which uses MIDISERVICES definition, then the size is incorrect.
If I modify the MIDIServies definition it still is not correct.
...
So question now is how can I modify midiservices.
It's not wrong or odd, but expected. If you change the sources of FPC packages (and in this case you do)
you need to recompile AND reinstall the modified packages.
(if I remember correctly, it should as easy as entering fpcsource/packages/unvint and running a command:
make clean all install
BUT depending on where your sources are (need of root access?) it might be a bit trickier.
"install" is likely to require you the root access)

Whenever you run a compiler it doesn't check the package source, instead it goes straight to recompiled binaries.
(Btw, Lazarus can recompile a "Lazarus package" automatically, but it cannot recompile "FPC packages" automatically. And both MacOSAll and MIDIServices are FPC packages)
...
and here's the project management problem.
Even if you provide the patch (and I think you should) to correct MIDIServices declaration, you will not get the fixed version until the next official FPC release (which will come with the correctly compiled header).
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Josh on June 22, 2019, 08:51:21 am
Hi

Thanks everyone for their input.

I managed to rebuild the modification and everythiong is now working correctly.
I have created a bug tracker entry

https://bugs.freepascal.org/view.php?id=35750
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Thaddy on June 22, 2019, 10:42:02 am
I have repeated my comments on the bug tracker. (And I checked the record sizes on my mini Mac)
The pack size is wrong: that is programmer error and not a bug.
Again: midi is a streaming protocol and does not allow slack space. Record packing should be byte aligned. Always.
< RTFM  >:D >:D>

What you seem to forget is that natural alignment between 32 and 64 bitness is not equal.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: BeRo on January 02, 2021, 06:49:45 am
Quote from: Thaddy on June 22, 2019, 10:42:02 am
I have repeated my comments on the bug tracker. (And I checked the record sizes on my mini Mac)
The pack size is wrong: that is programmer error and not a bug.
Again: midi is a streaming protocol and does not allow slack space. Record packing should be byte aligned. Always.
< RTFM  >:D >:D>

What you seem to forget is that natural alignment between 32 and 64 bitness is not equal.

Sorry, but {$packrecords 4} is right here (or a packed record together with a AlignmentDummy UInt16 after the length UInt16). I've tested with my own pascal-native cross-platform Mini-DAW (Digital Audio Workstation), which I'm developing in the moment. You are probably confusing API data structures (here CoreMIDI) with the MIDI specification of the MIDI bytestream. 
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Thaddy on January 02, 2021, 07:29:10 am
No it is not.  It is 1. not 4. Midi is a streaming protocol.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: BeRo on January 02, 2021, 04:58:38 pm
Quote from: Thaddy on January 02, 2021, 07:29:10 am
No it is not.  It is 1. not 4. Midi is a streaming protocol.

Sorry, but MidiPacket is a CoreMIDI data struct, only MidiPacket.data contains the raw MIDI

Even in the original CoreMIDI header sources from Apple itself one can see "// MIDIPacket must be 4-byte aligned" as code comment.

https://github.com/phracker/MacOSX-SDKs/blob/master/MacOSX10.9.sdk/System/Library/Frameworks/CoreMIDI.framework/Versions/A/Headers/MIDIServices.h#L1984

and see "#pragma pack(push, 4)" at

https://github.com/phracker/MacOSX-SDKs/blob/master/MacOSX10.9.sdk/System/Library/Frameworks/CoreMIDI.framework/Versions/A/Headers/MIDIServices.h#L338

Josh is saying, Apple is saying it, I'm saying it. You are pretty much alone with your opinion, which is probably due to the misunderstanding on your part that the MidiPacket Struct contains completely only raw MIDI data, which it defacto does not. It is a container structure including header data for raw MIDI data.
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: BeRo on January 02, 2021, 05:01:46 pm
(In addition, I work with it productively as a professional developer in the audio field as well.)
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: BeRo on January 02, 2021, 05:11:38 pm

Code: C  [Select][+][-]
  1. #include <stdlib.h>
  2. #include <stdio.h>
  3. #include <string.h>
  4.  
  5. #include <CoreServices/CoreServices.h>
  6. #include <CoreMIDI/MIDIServices.h>
  7. #include <CoreAudio/HostTime.h>
  8. #include <unistd.h>
  9.  
  10. int main(){
  11.   int i = sizeof(MIDIPacket);
  12.   printf("%i\n", i);
  13.   return 0;
  14. }      
  15.  

Prints also 268 on my 2015 MacBook Pro 13 inch under a 64-bit MacOS Big Sur. 
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: freq on September 07, 2021, 04:45:10 pm
Quote from: josh on June 22, 2019, 08:51:21 am
Hi

Thanks everyone for their input.

I managed to rebuild the modification and everythiong is now working correctly.
I have created a bug tracker entry

https://bugs.freepascal.org/view.php?id=35750

Hi, can confirm that your solution with {$packrecords 4} is working also here with Intel Mac, 64bit under BigSur!
Thanks for providing the solution!
Title: Re: Midi Packet Size is different between 32carbon and 64 bit cocoa
Post by: Key-Real on May 23, 2023, 08:51:15 am
I have FPC Trunk 3.3.1 [2023/05/22]

This Issue is still not solved :(

I'm on an AArch64 Mac (M1)

this works (around line 573):


Code: Pascal  [Select][+][-]
  1. {$push}
  2. {$packrecords 4}
  3.   MIDIPacket = record
  4.                 timeStamp: MIDITimeStamp;
  5.                 length: UInt16;
  6.                 data: array [0..255] of Byte;
  7.                end;
  8. {$pop}


and (around line 612):


Code: Pascal  [Select][+][-]
  1. {$push}
  2. {$packrecords 4}
  3.   MIDIPacketList = record
  4.                     numPackets: UInt32;
  5.                     packet: array [0..0] of MIDIPacket;
  6.                    end;
  7. {$pop}
  8.  



pls fix
TinyPortal © 2005-2018