File sync system approach

[aradeonas]

Hi,

For one of my projects I want to send and receive files between server and client and for years I saw demos in Indy,Synapse ... but I wonder if any one done it in real world and have experience or suggestion that want to share with so prevent me to go wrong or reinvent wheel or engine if they are out.

I want to know what is the best approach for making a sync system,what is the powerfullest library and live library,tips to know about security or even file changes recognition and upload just changed part like DropBox.
I don't want all,just if you can help me in some part please do.

Regards

[aradeonas]

Ok.Now I want to work on file sync part and for that I need to choose a library.Indy,Synapse or ... .
What you guys prefer in this case?Synapse is easy to use and I like it but Indy seems larger and more powerful but I didn't work with it.
For this case that I need to send and receive files in threads with multi connections it seems I need TCP and there are demos but I want the project stable and no need to rewrite it and maybe there will be need to some p2p connections so I probably I will need to write many things myself and this is not a problem but I want to ask you what library is better in your experience and is there any demos for them in these case or not?

[jc99]

Ok.Now I want to work on file sync part and for that I need to choose a library.Indy,Synapse or ... .
What you guys prefer in this case?Synapse is easy to use and I like it but Indy seems larger and more powerful but I didn't work with it.
For this case that I need to send and receive files in threads with multi connections it seems I need TCP and there are demos but I want the project stable and no need to rewrite it and maybe there will be need to some p2p connections so I probably I will need to write many things myself and this is not a problem but I want to ask you what library is better in your experience and is there any demos for them in these case or not?

Generally use what you know and like.
If you like Synapse, then use it, normally what you know best, is the best, unless you have issues with it.
I used to use Indy because it came with delphi(long ago ...), so to me it would be Indy, but they now have some issues with FPC ...
On the File-Sync part look at SVN or GIT how they do it, they already have a running file-syncing.
Another approach is the P2P-Filetransfer (e.G: lime, torrent, donkey) (this is a file-sync of some sort. but only one-way)   
 
 

[aradeonas]

Im agree with you but I wanted to check before and ask others opinions not jumping in the code

[bylaardt]

These files do you want synchronize comes from external pages (http or ftp)? if yes, then use synapse or indy.
If the server under your responsibility and these files are not huge files, what you think about use a DBMS with external access? like mysql ou postgres without bind address? security issues can be administered by de DBMS and the files can be saved on blob fields.

[aradeonas]

These files do you want synchronize comes from external pages (http or ftp)?

Files may be huge and comes from server that may be in local network or not.

[nomorelogic]

as I understand, you have to realize something like rsync
have you discarded the idea of using this (c) library?

[bylaardt]

Files may be huge and comes from server that may be in local network or not.

"local or not" doesn't matter, but files over 2GB (2^31 -1) is can be a limit for direct access.
i prefer put my files in blob fields because just a few number of users can overwrite the existing files (the file owner or authorized users), and you can create a structure of field to maintain a better control of these files, like specify the owner, version, description, search in contents, personal lists, share with information, key words, etc...

with rsync is a good way to do that if system date is the only overwrite argument you need to sync.

[aradeonas]

For a system like archive DB cant be good because many performance problem but in many times storing files in DB is a easy and very good way.
About rsync yes it similar to it but first I saw limits in running it Windows and more important I want to have control on the system also couldn't find a good way to control it powerfully,what you think?

[bylaardt]

make a pros and cons list.
you need a better control? DB is my choice.
you need performace? create a ftp server and access with synapse. Only if control over files is secundary.
files with high Information sensitivity? my choice is Encrypt data with a server application in a protected local DB (bind address): worst option to performace.

[aradeonas]

I dont know about FTP but I think a TCP client server will be a goo approach for this situation

[jc99]

I dont know about FTP but I think a TCP client server will be a goo approach for this situation

BTW, is only 1 client involved ?
So the syncing should act as some sort of backup-mechanism ?
Or do you plan to have multible clients and each can change data ?
Next Question: central or decentral,
central approach: more control over the whole thing, but on server-fail- no syncing possible, maybe loss of data.
decentral approach: less control, but your system cannot be shut down by switching of the server. maybe slower syncing but that depends.
And since it is a sync-process, only differences, files and parts of files must be synced, it can also work on a slow connection, when the initial sync-state is reached.
The initiation, of cause will transfer all files, maybe compressed.
(Sometimes a Harddrive shipped by (Snail-)Mail is faster than any ISP-internet-connection e.G: 6T/12h)

[gafe]

I made a little command line utility as part of my "learning Pascal" experience, maybe you'll find it useful.

It's another "i'm learning" file copy utility, with two options that make it different:

- Copy only "changed" parts of a file. It's done reading the file in blocks (of the size you want) and storing a hash of that block in a file. Later, when copying again the file, simply checks if the block hash is the same, if not, writes that "block" in the destination file.
- Read to memory a configurable (in MB) amount of files before writing them to disk. I suppose this is faster if you copy files inside the same disk.

It doesn't have support for wildcards, and surely needs some love, a "overwrite" option,etc...but you can copy a c:\users folder and doesn't stop. In the end prints the files that could not be copied (if any).

https://github.com/miguel-cv/dcopy/blob/master/dcopy.pas

Once you read the file and have the block, you can compress it, send it over the network...what you want.

[aradeonas]

@jc99 : multi client and yes like a backup and they will store in a central center.
@gafe: very interesting code,Im looking at it,its a bit hard for me because of the subject and non English comment.
I searched a lot but didnt find yours,You are kind with helping me and sharing it but for future please add a English description so other people can find your works easier.
I will wok on it and ask you

[gafe]

Sorry...i did it for me only with the purpose of learn the language, never thought of sharing it. Maybe before weekend if i have time i'll try to make/translate some comments (in English, of course).