This post isn't about anything very specific to FreePascal, it's rather a general computer science question which I would like some opinions on. I wasn't entirely sure where to put it, as although I will be discussing a potential server application written in FreePascal, the question is not related to the networking or web programming of FPC, so I thought this was the best place to put the post.
I recently had the idea to write a server application which allows various client applications to share a heap of memory through a subscription model. Similar to a message queue, but rather than sharing structured messages, a page of memory is shared between various applications over the network. This page of memory in the application is then treated like a regular page of memory, but it's updated and synced with all the other client applications subscribed to the same page on the server. This allows programs to easily share RAW data structures in memory with extremely low overhead, think of uses in embedded devices for example, where processing power might not be enough to parse other data types. With this new server, the embedded device grabs the current state of the heap from the server, and subscribes to new updates. Once a new update is ready, the device locks the heap pointer on it's side while it's transferred from the server replacing the existing memory in the embedded application. While most of the world has moved towards everything being encapsulated in the HTTP protocol these days with REST, this server I am writing uses an extremely lightweight persistent TCP protocol stream. For embedded programming this is much more optimal, but it can also work on servers as well, such as in a microservices environment where data moves around very fast between multiple applications. And of course, I am currently developing both the server software, and the client library in FreePascal.
I am currently debating on if this project/idea will actually be useful to a wider audience or not. This is why I am posting this today. I'd like to know the community here's opinion on such a project, and if anybody else might find it useful. If so, I have plans on publishing it on GitHub, still have yet to decide on a license for it. The initial prototype server and pascal client unit are almost ready, not production ready, but ready to be shown and tested by a wider audience if anybody is indeed interested on this concept for their own projects.
Currently my focuses with this project are:
- Authentication, and limiting access to specific pages based on this authentication. So say, this client can both read/write to these pages, but can only read from that one.
- Over the wire encryption, this is where I'd like a bit more community support to ensure it's secure enough.
- Subscribing to pages, and having the clients notified over their persistent TCP connection of any updates.
- Developing a management front-end for the server in Lazarus to manage pages, connections, and authentication. Perhaps the ability to even view a page in a hex editor/viewer.
- Currently the server supports what I call Application identifiers, and Type identifiers. This can be used to search for a specific page from the client without knowing the page number. So, if you mark a page with an AppId of $34, it can be searched for via a look-up table on the server.
- Pages can also be given a title currently as well, this cannot be searched, but is useful for identifying additional metadata about that specific page in a management front-end tool that can list all the pages from the server.
Which leads me to my last paragraph. How did I come up with such an absurd idea? Well, I've always had a weird fascination with RAW memory, and the memory cards used in the older video game consoles. I originally created my own version of a memory card system in FreePascal, where I can easily create a file on my disk separated in distinct blocks. The block size is entirely adjustable starting at 512 bytes as the smallest size. After playing with this code in a local non-network program, I wondered how this could scale in a network, and so this new project was born. I dubbed the project "Memory Card Server" and started work rather quickly on making these memory cards network accessible. The server supports multiple distinct "memory card" files with different block sizes depending on the client application requirements. Each memory card with it's own block-size is then divided even further into blocks of memory based on the formatted block-size, with Block 0 being reserved for the custom header and I guess a FAT table which hold the titles, AppId, and TypeId data.