Recent

Author Topic: Parallel programming model  (Read 4555 times)

pascal111

  • Sr. Member
  • ****
  • Posts: 423
  • Un trabajo en equipo para programas serias.
Parallel programming model
« on: July 03, 2022, 10:04:15 pm »
Does free Pascal support "Parallel programming model (paradigm)", and how it looks like?

I was reading this link and I think I understood 10% of it:

https://en.wikipedia.org/wiki/Parallel_programming_model
La chose par la chose est rappelé.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: Parallel programming model
« Reply #1 on: July 03, 2022, 10:22:44 pm »
It supports threads, hardware such as MMX/SSE, but not (automatic) parallelisation of arbitrary code.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

pascal111

  • Sr. Member
  • ****
  • Posts: 423
  • Un trabajo en equipo para programas serias.
Re: Parallel programming model
« Reply #2 on: July 03, 2022, 10:28:14 pm »
It supports threads, hardware such as MMX/SSE, but not (automatic) parallelisation of arbitrary code.

MarkMLl

Humanly speaking, I can say this is beyond my current knowledge and experience, I won't ask about it again except if once I'll study threads.

thanks @MarkMLl
La chose par la chose est rappelé.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: Parallel programming model
« Reply #3 on: July 03, 2022, 10:44:40 pm »
Humanly speaking, I can say this is beyond my current knowledge and experience, I won't ask about it again except if once I'll study threads.

I'll try to break that down a bit, but the risk here is that if I simplify too much one of the "usual suspects" will jump in and muddy the water :-)

A thread is effectively a procedure that runs in the background. An example would be something that read input from a networking socket, serial port or physical device and placed it on a queue.

MMX/SSE etc. are "vector extensions" to the x86 architecture, other architectures often have something comparable. Using this you can, as an example, perform a sequence of mathematical operations on a 4-element array of numbers in parallel.

Automatic parallelisation is when you e.g. have a for i := 0 to 15 do loop, and the compiler works out how to distribute that over multiple ALUs ** . Cray had a compiler that did that, written in Pascal, but reputedly when SGI bought Cray they had a beach-party bonfire of most of their erstwhile competitor's engineering material.

MarkMLl

** I'm being careful with my terminology here. There's lots of ways parallelisation can work, including distributing calculations over a vector processing unit (MMX etc.) and- in traditional terms- distributing it over multiple CPUs. However these days PC-derived terminology tends to equate "CPU" with "chip", and to call a processing unit a core within a CPU.
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

pascal111

  • Sr. Member
  • ****
  • Posts: 423
  • Un trabajo en equipo para programas serias.
Re: Parallel programming model
« Reply #4 on: July 03, 2022, 10:58:44 pm »
Thank you @MarkMLl and forgive my aggressive responses sometimes, you know, you have something like what I can call heavy or dark fingerprint or impression, it makes the other side like me thinks you're evil person.

I know this topic is so rich and has some complex parts, but I get an idea about it from your explaining.
La chose par la chose est rappelé.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: Parallel programming model
« Reply #5 on: July 03, 2022, 11:17:11 pm »
Oh, I'm thoroughly evil: it's the Slav in me :-)

Quote
The worst kind of soul is the great Slav soul. People who suffer from it are usually very deep thinkers. They may say things like this: "Sometimes I am so merry and sometimes I am so sad. Can you explain why?" (You cannot, do not try.) Or they may say: "I am so mysterious ... I sometimes wish I were somewhere else than where I am." (Do not say: "I wish you were.")

http://f2.org/humour/howalien.html

You started it :-)

MarkMLl

Oh, and the Ll stands for Llwyd. Your problem :-)
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

pascal111

  • Sr. Member
  • ****
  • Posts: 423
  • Un trabajo en equipo para programas serias.
Re: Parallel programming model
« Reply #6 on: July 03, 2022, 11:37:54 pm »
Quote
Oh, I'm thoroughly evil: it's the Slav in me :-)

http://f2.org/humour/howalien.html

It seems so beautiful piece of English literature.

Llwyd:"(ˈhluːɪd, English lɔid) (in Welsh legend) a magician who avenged his friend Gwawl upon Pryderi, the son of Pwyll, by casting various spells upon Pryderi and his estate."

Yes, you have an evil name  :)

Voltaire talked about some of Pascal thoughts in one of his books and called him according to the Arabic translation I read "The most wonderful enemy of humanity". That because a secret that Pascal isn't human or what Pascal will be in last days. Pascal thought it's not good idea to make the beast free, so he put so hard difficult restrictions against him that the so freedom for the beast isn't good thing maybe because he has no inner conscience like humans so maybe he will do bad things because nothing internally directs him to do right things like other humans. Restrictions cause so horrible life and can cause death.
La chose par la chose est rappelé.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: Parallel programming model
« Reply #7 on: July 03, 2022, 11:46:48 pm »
It seems so beautiful piece of English literature.

Albeit written by a Magyar.

Quote
Llwyd:"(ˈhluːɪd, English lɔid) (in Welsh legend) a magician who avenged his friend Gwawl upon Pryderi, the son of Pwyll, by casting various spells upon Pryderi and his estate."

Congratulations on your research :-)

Although /Morgan/ Llwyd was a renowned Christian preacher.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

alpine

  • Hero Member
  • *****
  • Posts: 1032
Re: Parallel programming model
« Reply #8 on: July 04, 2022, 12:09:40 am »
Oh, I'm thoroughly evil: it's the Slav in me :-)

Quote
The worst kind of soul is the great Slav soul. People who suffer from it are usually very deep thinkers. They may say things like this: "Sometimes I am so merry and sometimes I am so sad. Can you explain why?" (You cannot, do not try.) Or they may say: "I am so mysterious ... I sometimes wish I were somewhere else than where I am." (Do not say: "I wish you were.")

http://f2.org/humour/howalien.html

You started it :-)

MarkMLl

Oh, and the Ll stands for Llwyd. Your problem :-)
Is there something allegorical in the "great Slav soul" or it somehow connected to your personal ancestry?
Because I know Lloyd is a Welsh surname meaning gray? Welsh and Slav? It should have some hidden context  :-\
"I'm sorry Dave, I'm afraid I can't do that."
—HAL 9000

pascal111

  • Sr. Member
  • ****
  • Posts: 423
  • Un trabajo en equipo para programas serias.
Re: Parallel programming model
« Reply #9 on: July 04, 2022, 12:17:26 am »
Quote
Although /Morgan/ Llwyd was a renowned Christian preacher.

By mentioning "Christian preacher" Christianity is the preferred religion to Pascal and it what he chose for beast, while Voltaire - as I thought - tried to free the beast from that choice and suggested many other religions, and we don't know what Pascal was thinking about when he chose Christianity. He wants something we don't know. Some sources said that Christianity is for removing the terrorist mind nature that can be cause because some other religions.

Reading history is so useful. I think you are so accurate reader and thinker like that piece said "...very deep thinkers.", you have so detailed information in any topic of computer you are been asked for, I noticed that.
La chose par la chose est rappelé.

dbannon

  • Hero Member
  • *****
  • Posts: 2778
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Parallel programming model
« Reply #10 on: July 04, 2022, 04:19:32 am »
I might (very boringly) return to parallel processing.  Having spent a good part of my working life telling various researchers the 'sort' of parallel they have put all their effort into was a waste of time, its fun to re-live it.

There are two broad 'sorts' of parallel (although some dispute even the word) -

* Message Passing/b] - using an approach like MPI or MPICH (yes, works with FPC I believe)

* MultiProcessing,  chmem - really just multithreaded but todays OSs know how to move a thread process onto another core of the same 'box', either the same CPU or another CPU in the same memory space.

MPI programming is the basis of almost all the Supercomputers around these days, even Cray and SGI sell vastly more such systems. They consist of lots of individual 'boxes', each with there own memory space and a fast interconnect. Generally harder to programme but some jobs can and do scale to thousands of processes.

MP on the other hand is always limited to the one box and while people like SGI mastered the art of cramming lots of CPUs in one box, it was always a more complicated, expensive and fragile solution.

Intel championed OpenMP with some nice 'self paralleling compilers' but for many jobs, the parallel efficiency was generally limited to 4 to 16 cpus. chmem is a library that shares memory between boxes but for many applications it still has quite a lot of overhead.

Oh, and then people started running parallel code on Video Cards.....

Davo

Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

Zvoni

  • Hero Member
  • *****
  • Posts: 2300
Re: Parallel programming model
« Reply #11 on: July 04, 2022, 10:01:52 am »
In my experience, to understand something like Threads/Parallel computing in this case, i always come from "Real life".
Like this morning:
I entered my car, and started it.
After getting my morning coffee at the local gas-station i continued driving on towards my workplace.
Once driving again (one Process/Thread), i turned on the music in my car (launching a second Thread/Process).
Some minutes later while waiting at a red light, i lit a cigarette (Yes, i smoke in the car) (launching a third Thread/Process)

Bottom line: As described above, there are three distinct "Processes" happening at the same time

Or let's take driving only.
When you drive a car, and you want to switch lanes, you hit your direction indicator, because your Thread "driving" received a message "You have to get to the other lane, because you want to use the next exit".
This is happening WHILE you're still driving. It's not like you stop driving, hit the indicator, and then start driving again.

The human brain itself is a marvel of "parallel computing", though many women will argue that males are not capable of "multi-Tasking" :-)

EDIT: Or to use another analogy:
1) Breaking up a big problem into smaller parts
Imagine a big Open-Air-Show moving from one venue to the next. What do they do?
They load up the whole Production (="One Big Problem" --> Stage, Lighting, Speakers, Instruments) into separate, multiple Trucks, wherein each Truck has its own "Load" ("smaller Problems")
They drive in "parallel" (think "Convoy") from Point A to Point B. The "Big Problem" is "solved" when each Truck arrives at their destination (Note: Not neccessarily at the same time)
2) Parallel Processes passing messages
Take a Football-Team. You have 11 Threads/Processes running at the same time.
The left defender "passes" the ball to the right Midfielder, and sending this Midfielder the Message "Pass the ball back to me because i will be running forwards along the sideline" --> synchronous! It doesn't make sense for the right Midfielder to pass the Ball back, if the left Defender has'nt arrived at his intended Destination.
Now the left defender has gotten the ball back, and shoots a pass blindly into the penalty box in front of the goal --> asynchronous! The left defender passed the Message: "I'm done. Do with it whatever you want"
« Last Edit: July 04, 2022, 10:20:51 am by Zvoni »
One System to rule them all, One Code to find them,
One IDE to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
Code is like a joke: If you have to explain it, it's bad

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: Parallel programming model
« Reply #12 on: July 04, 2022, 10:06:41 am »
* Message Passing/b] - using an approach like MPI or MPICH (yes, works with FPC I believe)

That is of course the dominant technology these days: while supercomputers in the 80s and 90s had some combination of vectorisation and multiple processors, the emphasis has switched to using a large number of commodity CPUs/GPUs and distributing the workload using one of the message-passing systems.

However this is entirely library-based, and I don't know what if any language implementations automate it.

Apropos threads, with the exception of some very early mainframes I'm unaware of any OS which was unable to migrate work between processors on closely-coupled systems i.e. within the context of a single system image.

Loosely-couple systems (where each node has its own OS) are a far more tricky matter, since the POSIX programming model allows threads to have state which can clash if they're moved. In part, that accounts for the rise of virtualisation and containerisation: in many cases a running VM can be fairly easily suspended and migrated.

I think I can fairly throw https://en.wikipedia.org/wiki/Vector_Pascal into the ring as one of very few free compilers which make at least some attempt to vectorise/parallelise. However it's some while since I last looked at it, and I can't remember whether it still builds on modern systems.

MarkMLl
« Last Edit: July 04, 2022, 11:07:16 am by MarkMLl »
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

dbannon

  • Hero Member
  • *****
  • Posts: 2778
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Parallel programming model
« Reply #13 on: July 05, 2022, 02:52:38 am »

I did not see any Pascal applications running on my machines although it may have happened. There is definitely an interface to MPI - https://wiki.lazarus.freepascal.org/MPICH - does not appear to have had any love for some time. Fortran and C/C++ dominated.

In terms of migrating processes on one box, yes, relatively easy but not always as efficient as you would expect. Speaking only of Linux here, twenty years ago, on (Ultrix, Compaq Unix) Unix systems, we were encouraging end users to use MPI even when they could request single image. Good MPI libraries were smart enough to skip the interconnect if they could and 8 cpus in a box was a sweetspot for cost effectiveness. Memory bandwidth was always an issue, even with chips like the Alpha.

> virtualisation and containerisation ....

Is, sort of, the opposite to parallel ? 

https://en.wikipedia.org/wiki/Vector_Pascal

While Intel do now have some vector support but nothing like that offered by, eg, Cray. I wonder if the speed up delivered would be less than the penalty for not using an optimized compiler like FPC ?  Cray could do a complete step through a matrix with one op. But such machines are incredibly expensive, all purchased by US homeland defense and military. Apparently very efficient at listening in on trans Atlantic communications ....

I see absolutely no reason why FPC would not be just as useful in an MPI environment as C++. It would be far more reliable code (important on long running, compute intensive jobs) and, hitting the same MPI libraries, would be just as efficient through the interconnect.

I have seen, for example both python and java running MPI. But we would groan when we saw those users logging on ....

Davo

Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

MarkMLl

  • Hero Member
  • *****
  • Posts: 6647
Re: Parallel programming model
« Reply #14 on: July 05, 2022, 09:13:01 am »
> virtualisation and containerisation ....

Is, sort of, the opposite to parallel ? 

Yes, but I mentioned those since they're the dominant technology for moving work between systems larger than a single system image (noting that that term gets very indistinct considering e.g. large IBM mainframes, which have had distributed operation for decades).

Quote
https://en.wikipedia.org/wiki/Vector_Pascal

With the caveat- which I should possibly have made clear earlier- that there is a strong APL (or at least J) influence there. Nothing wrong with that: Wirth was involved with APL long before he designed Pascal.

In actual fact, I feel that an understanding of APL is no bad thing. Apart from anything else, it highlights two curious facts:

* A sizeable fraction of the CS fraternity is prepared to ignore memory requirement in the search for an improved O() applied to speed.

* A sizeable fraction of the CS and mathematical fraternities is prepared to throw transcendentals into an algorithm if doing so allows all elements of a vector to be handled in parallel.

Quote
While Intel do now have some vector support but nothing like that offered by, eg, Cray. I wonder if the speed up delivered would be less than the penalty for not using an optimized compiler like FPC ?  Cray could do a complete step through a matrix with one op. But such machines are incredibly expensive, all purchased by US homeland defense and military. Apparently very efficient at listening in on trans Atlantic communications ....

I see absolutely no reason why FPC would not be just as useful in an MPI environment as C++. It would be far more reliable code (important on long running, compute intensive jobs) and, hitting the same MPI libraries, would be just as efficient through the interconnect.

I have seen, for example both python and java running MPI. But we would groan when we saw those users logging on ....

Noting obviously Chris Fenton's summary of the Cray https://www.chrisfenton.com/homebrew-cray-1a/

I agree that FPC support for this could benefit from some TLC. Unfortunately most of the workload distribution research over the last 20 years has focussed on better ways to download cat videos.

Apropos Python, Java etc.: however much we may loathe the syntax or architecture, it's possible to JIT-compile them to something at least half-way efficient. But my comment apropos APL applies: the resources required to do so might be so obscenely large that it would have been better to have an efficient toolchain in the first place.

But that, of course, is where the "Itanic" failed: the difficulty of an efficient toolchain.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

 

TinyPortal © 2005-2018