Recent

Author Topic: Really big array on Raspberry Pi  (Read 830 times)

emjtech

  • New member
  • *
  • Posts: 25
    • Global R&D ltd
Really big array on Raspberry Pi
« on: February 13, 2019, 07:01:15 pm »
Hi
I need to work with a very large array  in order to grab in it hardware samples.
Since the rate is very fast I must predefine it and I have no tame to save to a file.
I read the Raspberry Pi 3B+ I am using has 1GBy of RAM
I defined a 2D array:
MyArray: Array [0..10000000,0..17] of Word;
That didn't work
Somebody knows what is the largest possible array?
Thanks
Thanks
Eli Jacob

engkin

  • Hero Member
  • *****
  • Posts: 2241
Re: Really big array on Raspberry Pi
« Reply #1 on: February 13, 2019, 07:05:55 pm »
Use dynamic arrays, the one you showed is reserved on the stack. The stack is usually small, while your array is around 340-MB.

emjtech

  • New member
  • *
  • Posts: 25
    • Global R&D ltd
Re: Really big array on Raspberry Pi
« Reply #2 on: February 13, 2019, 07:29:10 pm »
Thank you for the fast replay!
I now defined:
Var
  MyArray : Array of Array of Word;

Begin
  SetLength(MyArray,10000001,18);

It works OK, but when I used about 1000000 the Raspberry Pi 3B+ hanged very badly (only a power cycle got it out)

Somebody know what is the limit I can use?
Thanks
Eli Jacob

af0815

  • Full Member
  • ***
  • Posts: 219
Re: Really big array on Raspberry Pi
« Reply #3 on: February 13, 2019, 09:33:59 pm »
Start with a lower value and check with top how much memory is consumed.
regards
Andreas

emjtech

  • New member
  • *
  • Posts: 25
    • Global R&D ltd
Re: Really big array on Raspberry Pi
« Reply #4 on: February 14, 2019, 09:30:18 am »
Thanks
Sorry about the silly question but: what is Top ?
Thanks
Eli Jacob

lucamar

  • Hero Member
  • *****
  • Posts: 872
Re: Really big array on Raspberry Pi
« Reply #5 on: February 14, 2019, 09:47:16 am »
Thanks
Sorry about the silly question but: what is Top ?

No "Top", just "top". It's a standard unix command to dispay runnin tasks and system load. Open a console and type
Code: [Select]
man topfor more info.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus 1.8.4/FPC 3.0.4 on:
(K)Ubuntu 12..16, Windows XP SP3 (Home/Prof.) and various DOS incarnations.

dbannon

  • Sr. Member
  • ****
  • Posts: 461
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Really big array on Raspberry Pi
« Reply #6 on: February 14, 2019, 09:56:30 am »
... and when you have top running, press shift-m to see a memory using sorted list.

q to quit.

Davo
Lazarus 1.8, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

Thaddy

  • Hero Member
  • *****
  • Posts: 7436
Re: Really big array on Raspberry Pi
« Reply #7 on: February 14, 2019, 12:49:09 pm »
Hi
I need to work with a very large array  in order to grab in it hardware samples.
Since the rate is very fast I must predefine it and I have no tame to save to a file.
I read the Raspberry Pi 3B+ I am using has 1GBy of RAM
I defined a 2D array:
MyArray: Array [0..10000000,0..17] of Word;
That didn't work
Somebody knows what is the largest possible array?
Thanks
The largest possible array is determined by available memory or  NativeInt per  array dimension(which is unrealistically huge!). Make sure the array is global, though. Allocating it inside a procedure or function will give you stack problems.
I tested your array as a global array on a RPi3 and it simply works. If declared as a local it won't work because of the stack limitation. In such case you need to declare a local pointer-to-myarray instead of the array itself and use getmem or allocmem to allocate the memory for it on the heap. (Also note your array is probably 1 larger than you meant? 0..10000000?). Or indeed use a dynamic array, which also allocates from the heap and not the stack.
Code: Pascal  [Select]
  1. {$mode objfpc}
  2. var
  3.  MyArray: Array [0..10000000,0..17] of Word;
  4.  i,j:integer;
  5. begin
  6.   for i := 0 to 10000000 do
  7.     for j := 0 to 17 do
  8.       MyArray[i,j] := 1;
  9.   writeln(MyArray[10000000,17]);
  10. end.
« Last Edit: February 14, 2019, 01:10:12 pm by Thaddy »
Ad Brexinitum (can't help it)

MarkMLl

  • New member
  • *
  • Posts: 15
Re: Really big array on Raspberry Pi
« Reply #8 on: February 15, 2019, 10:12:17 am »
I would note however that the RPis can be spectacularly bad when they start thrashing, irrespective of whether they're running from an SD-Card, SSD or rotating media.

> Ad Brexitum

Oh /don't/. I come from near Wrexham and that was a bad place, and I fear we're heading to worse.

Thaddy

  • Hero Member
  • *****
  • Posts: 7436
Re: Really big array on Raspberry Pi
« Reply #9 on: February 15, 2019, 11:10:28 am »
I would note however that the RPis can be spectacularly bad when they start thrashing, irrespective of whether they're running from an SD-Card, SSD or rotating media.
Thrashing hardly happens when:
- Using a power supply > 2.5A combined with:
- Using a premium brand SD card (class 10)

BTW I boot some RPi's from a Sandisk 64GB USB drive and that is rock solid.
A good power supply is a *must*, otherwise you take the risk of trashing yourself.
What also helps is connect a powered USB extension hub when you run rotating media: hard disks can be power hungry.
Ad Brexinitum (can't help it)

MarkMLl

  • New member
  • *
  • Posts: 15
Re: Really big array on Raspberry Pi
« Reply #10 on: February 15, 2019, 12:05:29 pm »
I've worked with big banks of RPIs, with custom power and various storage devices (not limited to SD-Cards). Thrashing, when it happens, is particularly destructive on that platform.

MarkMLl

Thaddy

  • Hero Member
  • *****
  • Posts: 7436
Re: Really big array on Raspberry Pi
« Reply #11 on: February 15, 2019, 12:59:58 pm »
Then you should still check the power supply: it needs to provide 2.5 A continuous, not peak. E.g. it is not a good idea to use one of the fast chargers for mobile phones: these are often rated at 2.5A or higher, but can not provide the amperage continuously. I also have an array (of 4+1 controller) and have no real trouble until now. Although I trashed many a card on my first RPi 1B.
Measure the PS with a scope - or a simple multi-meter - and see what it really provides under load.
The power supply of the foundation has those characteristics and as a bonus nearly ripple free even on on bad mains.
« Last Edit: February 15, 2019, 01:03:46 pm by Thaddy »
Ad Brexinitum (can't help it)

MarkMLl

  • New member
  • *
  • Posts: 15
Re: Really big array on Raspberry Pi
« Reply #12 on: February 15, 2019, 03:40:40 pm »
I can assure you that this was not a power problem, and I /do/ know how to use the appropriate instrumentation.

MarkMLl

emjtech

  • New member
  • *
  • Posts: 25
    • Global R&D ltd
Re: Really big array on Raspberry Pi
« Reply #13 on: February 17, 2019, 08:53:25 am »
Thanks to all
There where 2 problems :
1 the Array must be declared Global, not into the procedure
2 at that size of the array the amount of time required to save it make it looked like the program hanged but it works

Still Raspberry Pi is exceptionally versatile, but maybe not fast enough for my application (almost enough)

This is for the forum, I was surprised to got so many replies while on the titles is written only 4 replies...
Thanks
Eli Jacob

avra

  • Hero Member
  • *****
  • Posts: 1442
    • Additional info
Re: Really big array on Raspberry Pi
« Reply #14 on: February 17, 2019, 12:53:19 pm »
at that size of the array the amount of time required to save it make it looked like the program hanged but it works
If this was the only reason to abandon RPi then you should know that it can be solved.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib