Recent

Author Topic: Hiring... Maybe..  (Read 18751 times)

aidv

  • Full Member
  • ***
  • Posts: 173
Re: Hiring... Maybe..
« Reply #15 on: September 15, 2015, 02:12:00 am »
You mean like Google Drive? Then no, that's not what I'm looking for. I just want to programatically send files to the server, I'm just unsure of how to achieve the best results.

Ok, so you are saying you arn't supporting delayed sends.

Exactly. When the client sends a file it needs to be sent as soon as possible and as quickly as possible.

rvk

  • Hero Member
  • *****
  • Posts: 6716
Re: Hiring... Maybe..
« Reply #16 on: September 15, 2015, 04:46:56 am »
You tell me, I've tried developing my own protocol, doesn't work as expected even though I've done all I can to make it work properly.
You did (all you can)? Did you come here with some example code for us to try and ask why it didn't work?

I still feel that if you couldn't make it work you either made a mistake, the components is buggy or you have a communication issue like defective cable/connection or something. If you made a mistake we should be able to  help you with your code and fix it. If the components is buggy or you have defective cabling/connection, then going for HTTP would also fail.

aidv

  • Full Member
  • ***
  • Posts: 173
Re: Hiring... Maybe..
« Reply #17 on: September 15, 2015, 05:14:38 am »
You tell me, I've tried developing my own protocol, doesn't work as expected even though I've done all I can to make it work properly.
You did (all you can)? Did you come here with some example code for us to try and ask why it didn't work?

I still feel that if you couldn't make it work you either made a mistake, the components is buggy or you have a communication issue like defective cable/connection or something. If you made a mistake we should be able to  help you with your code and fix it. If the components is buggy or you have defective cabling/connection, then going for HTTP would also fail.

Yes well I believe I've done all I can, I haven't been programming on it since March for no reason I guess.

And why does it matter? I'm offering money and whoever would be up to take on the project would be in full control of making sure it works, so my say has no value.

I have a question now though. I've been playing with the example called HTTPServer in fcl-Web, and I am wondering how exactly does it handle files?

Lets say the server receives a file, and apparently I access it  with the following code:

Code: [Select]
ARequest.Files.Files[0].Stream;
When I access it, has it already been received and stored into memory?
Or is data requested as the file is being read from?

In other words: If I send a 2GB file to the server and use the code above in the server to read the 2GB file, has the file already been downloaded in to memory or is the file somehow dynamically received?

I hope I make sense.

derek.john.evans

  • Guest
Re: Hiring... Maybe..
« Reply #18 on: September 15, 2015, 06:50:43 am »
Here is some code for you to read. This uses Indy, since I found fp-web a little too buggy.

I tested it on a 1.8 Gig movie, sent in chunks of 16M. Each chunk is MD5 checked and is retryed up to 5 times, and the final file is MD5 checked.

The problem is, MD5 seems somewhat slow, so, you may prefer to try another checksum calculation. ie: The MD5File() command takes a long time to return for the 1.8G file.

Other issues include security. But, thats up to you. Personally, I wouldn't do it this way, but its your choice. I would use folder syncing, since sending files this way can only cope with basic comunication errors. ie: If the connection goes down for a minute, what is the app ment todo? Just wait? Until when?

HTTP Server with 2 extra commands for MD5 checks, and file joining.
Code: Pascal  [Select][+][-]
  1. program server;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   Classes,
  7.   FileUtil,
  8.   IdContext,
  9.   IdCustomHTTPServer,
  10.   IdHTTPServer,
  11.   IdURI,
  12.   Interfaces,
  13.   MD5,
  14.   pl_indy,
  15.   SysUtils;
  16.  
  17. type
  18.  
  19.   TFileServer = class(TIdHTTPServer)
  20.     function ResolveUrl(const AUrl: String): String;
  21.     procedure DoCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo;
  22.       AResponseInfo: TIdHTTPResponseInfo); override;
  23.     procedure DoCommandOther(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo;
  24.       AResponseInfo: TIdHTTPResponseInfo); override;
  25.   end;
  26.  
  27.   function TFileServer.ResolveUrl(const AUrl: String): String;
  28.   begin
  29.     Result := ProgramDirectory + 'http' + IncludeLeadingPathDelimiter(SetDirSeparators(AUrl));
  30.   end;
  31.  
  32.   procedure TFileServer.DoCommandGet(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo;
  33.     AResponseInfo: TIdHTTPResponseInfo);
  34.   var
  35.     LIndex: Integer;
  36.     LFileName: TFileName;
  37.     LOutput, LInput: TStream;
  38.   begin
  39.     inherited;
  40.     try
  41.       LFileName := ResolveUrl(ARequestInfo.URI);
  42.       case ARequestInfo.CommandType of
  43.         hcGET: begin
  44.           case ARequestInfo.Params.Values['cmd'] of
  45.             'md5': begin
  46.               if FileExists(LFileName) then begin
  47.                 AResponseInfo.ContentText := MD5Print(MD5File(LFileName));
  48.               end else begin
  49.                 AResponseInfo.ContentText := 'ERROR';
  50.               end;
  51.             end;
  52.             'join': begin
  53.               with TStringList.Create do begin
  54.                 try
  55.                   CommaText := ARequestInfo.Params.Values['files'];
  56.                   LOutput := TFileStream.Create(LFileName, fmCreate);
  57.                   try
  58.                     for LIndex := 0 to Count - 1 do begin
  59.                       LInput := TFileStream.Create(ResolveUrl(Strings[LIndex]), fmOpenRead);
  60.                       try
  61.                         LOutput.CopyFrom(LInput, 0);
  62.                       finally
  63.                         FreeAndNil(LInput);
  64.                       end;
  65.                     end;
  66.                   finally
  67.                     FreeAndNil(LOutput);
  68.                   end;
  69.                 finally
  70.                   Free;
  71.                 end;
  72.               end;
  73.             end else begin
  74.               AResponseInfo.ServeFile(AContext, LFileName);
  75.             end;
  76.           end;
  77.         end;
  78.         hcDELETE: begin
  79.           DeleteFile(LFileName);
  80.         end;
  81.         hcPUT: begin
  82.           LOutput := TFileStream.Create(LFileName, fmCreate);
  83.           try
  84.             LOutput.CopyFrom(ARequestInfo.PostStream, 0);
  85.           finally
  86.             FreeAndNil(LOutput);
  87.           end;
  88.         end;
  89.       end;
  90.     except
  91.       on E: Exception do begin
  92.         AResponseInfo.ContentText := E.Message;
  93.       end;
  94.     end;
  95.   end;
  96.  
  97.   procedure TFileServer.DoCommandOther(AContext: TIdContext; ARequestInfo: TIdHTTPRequestInfo;
  98.     AResponseInfo: TIdHTTPResponseInfo);
  99.   begin
  100.     DoCommandGet(AContext, ARequestInfo, AResponseInfo);
  101.   end;
  102.  
  103. begin
  104.   with TFileServer.Create(nil) do begin
  105.     try
  106.       Active := True;
  107.       WriteLn('Server started');
  108.       ReadLn;
  109.     finally
  110.       Free;
  111.     end;
  112.   end;
  113. end.
  114.  

HTTP Client code:
Code: Pascal  [Select][+][-]
  1. program client;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   Classes,
  7.   IdHTTP,
  8.   IdURI,
  9.   Interfaces,
  10.   Math,
  11.   MD5,
  12.   pl_indy,
  13.   SysUtils;
  14.  
  15.   function HttpGet(const AUrl: String): String;
  16.   begin
  17.     with TIdHTTP.Create(nil) do begin
  18.       try
  19.         Result := Get(AUrl);
  20.       finally
  21.         Free;
  22.       end;
  23.     end;
  24.   end;
  25.  
  26.   function HttpGetMD5(const AUrl: String): String;
  27.   begin
  28.     Result := HttpGet(AUrl + '?cmd=md5');
  29.   end;
  30.  
  31.   function HttpJoinFiles(const AUrl: String; const AFileNames: String): String;
  32.   begin
  33.     Result := HttpGet(AUrl + '?cmd=join&files=' + AFileNames);
  34.   end;
  35.  
  36.   function HttpDelete(const AUrl: String): String;
  37.   begin
  38.     with TIdHTTP.Create(nil) do begin
  39.       try
  40.         Result := Delete(AUrl);
  41.       finally
  42.         Free;
  43.       end;
  44.     end;
  45.   end;
  46.  
  47.   function StreamCopyChunk(const ADst: TMemoryStream; const ASrc: TStream; const AChunkSize: Integer): Integer;
  48.   begin
  49.     ADst.Clear;
  50.     Result := Min(AChunkSize, ASrc.Size - ASrc.Position);
  51.     if Result > 0 then begin
  52.       ADst.CopyFrom(ASrc, Result);
  53.       ADst.Position := 0;
  54.     end;
  55.   end;
  56.  
  57.   function MemoryStreamGetMD5(const AStream: TMemoryStream): String;
  58.   var
  59.     LPointer: Pointer;
  60.   begin
  61.     LPointer := AStream.Memory;
  62.     Result := MD5Print(MD5Buffer(LPointer^, AStream.Size));
  63.   end;
  64.  
  65.   procedure StreamHttpPut(const AStream: TStream; const AUrl: String);
  66.   begin
  67.     with TIdHTTP.Create(nil) do begin
  68.       try
  69.         AStream.Position := 0;
  70.         Put(AUrl, AStream);
  71.       finally
  72.         Free;
  73.       end;
  74.     end;
  75.   end;
  76.  
  77.   procedure StreamHttpPutWithErrorCheck(const AStream: TMemoryStream; const AUrl: String; ATryCount: Integer);
  78.   begin
  79.     while ATryCount > 0 do begin
  80.       StreamHttpPut(AStream, AUrl);
  81.       if HttpGetMD5(AUrl) = MemoryStreamGetMD5(AStream) then begin
  82.         Exit;
  83.       end;
  84.       Dec(ATryCount);
  85.     end;
  86.     raise Exception.Create('Chunk MD5 Check Failed');
  87.   end;
  88.  
  89.   procedure SendFile(const AURI: TIdURI; const AStream: TStream; const AChunkSize: Integer);
  90.   var
  91.     LIndex: Integer;
  92.     LRemotePath: String;
  93.     LMemoryStream: TMemoryStream;
  94.   begin
  95.     LRemotePath := AURI.Protocol + '://' + AURI.Host + AURI.Path;
  96.     LMemoryStream := TMemoryStream.Create;
  97.     try
  98.       with TStringList.Create do begin
  99.         try
  100.           repeat
  101.             if StreamCopyChunk(LMemoryStream, AStream, AChunkSize) > 0 then begin
  102.               Add(Format('__temp%d', [Count]));
  103.               StreamHttpPutWithErrorCheck(LMemoryStream, LRemotePath + Strings[Count - 1], 5);
  104.             end;
  105.           until AStream.Position = AStream.Size;
  106.           HttpJoinFiles(LRemotePath + AURI.Document, CommaText);
  107.           for LIndex := 0 to Count - 1 do begin
  108.             HttpDelete(LRemotePath + Strings[LIndex]);
  109.           end;
  110.         finally
  111.           Free;
  112.         end;
  113.       end;
  114.     finally
  115.       FreeAndNil(LMemoryStream);
  116.     end;
  117.   end;
  118.  
  119.   procedure SendFile(const AURI: TIdURI; const AFileName: TFileName; const AChunkSize: Integer);
  120.   var
  121.     LStream: TStream;
  122.   begin
  123.     LStream := TFileStream.Create(AFileName, fmOpenRead);
  124.     try
  125.       SendFile(AURI, LStream, AChunkSize);
  126.     finally
  127.       FreeAndNil(LStream);
  128.     end;
  129.     if HttpGetMD5(AURI.URI) <> MD5Print(MD5File(AFileName)) then begin
  130.       raise Exception.Create('Final MD5 Check Failed');
  131.     end;
  132.   end;
  133.  
  134.   procedure SendFile(const ADst: String; const AFileName: TFileName; const AChunkSize: Integer);
  135.   var
  136.     LURI: TIdURI;
  137.   begin
  138.     LURI := TIdURI.Create(ADst);
  139.     try
  140.       SendFile(LURI, AFileName, AChunkSize);
  141.     finally
  142.       FreeAndNil(LURI);
  143.     end;
  144.   end;
  145.  
  146. begin
  147.   try
  148.     SendFile('http://127.0.0.1/result.mp4', 'E:\New Movies\Inland Empire (2006).mp4', $1000000);
  149.   except
  150.     on E: Exception do WriteLn(E.Message);
  151.   end;
  152.   WriteLn('DONE');
  153.   ReadLn;
  154. end.
  155.  
« Last Edit: October 02, 2015, 03:09:58 am by Geepster »

derek.john.evans

  • Guest
Re: Hiring... Maybe..
« Reply #19 on: September 15, 2015, 07:12:04 am »
BTW: With Indy, you should be able to PUT large files by overriding TIdCustomHTTPServer.CreatePostStream(). The default uses a TMemoryStream, but I have been able to change that to a TFileStream (temp file).

But, again, you need to test that on your systems.

rvk

  • Hero Member
  • *****
  • Posts: 6716
Re: Hiring... Maybe..
« Reply #20 on: September 15, 2015, 11:10:13 am »
And why does it matter? I'm offering money and whoever would be up to take on the project would be in full control of making sure it works, so my say has no value.
Ok, maybe that would have been more appropriate in the sub-forum Jobs: About pascal related job offers or requests then.

(And you wrote "Maybe" in the title... so it sounded like you could also, maybe, need help with the problem you got with LNet. But directly outsourcing the whole problem is also a way to go.)

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12202
  • FPC developer.
Re: Hiring... Maybe..
« Reply #21 on: September 15, 2015, 11:16:46 am »
Indy comes with an own md5 class. (from idhash iirc). Maybe that one is faster.

Thaddy

  • Hero Member
  • *****
  • Posts: 16945
  • Ceterum censeo Trump esse delendam
Re: Hiring... Maybe..
« Reply #22 on: September 15, 2015, 12:15:05 pm »
Indy comes with an own md5 class. (from idhash iirc). Maybe that one is faster.

No. (as usual with Indy code it is even primitive.) Determine this by actually looking at the code..

(There's a better and memory safe one on my website)

MD5 is not a very good hash to use in the context of assuring delivery in the context of http since it is broken and very prone to midm attacks. Combining http without s and on top of that md5 is asking for trouble. md5 is in principle still ok to verify *afterwards* to a certain degree, but not on a packet basis which is actually implied here. It all depends on the amount of reliability that is required. A more secure hash doesn't necessarily mean a slower hash. Hashing blocks is already taken care of on a lower level. If streaming data is an issue, md5 is not a very good or informed choice. It needs to disappear quickly, otherwise people may use it.
« Last Edit: September 15, 2015, 12:24:09 pm by Thaddy »
Due to censorship, I changed this to "Nelly the Elephant". Keeps the message clear.

derek.john.evans

  • Guest
Re: Hiring... Maybe..
« Reply #23 on: September 15, 2015, 01:13:58 pm »
In this situation, MD5 is only used as an extra sanity check. Nothing else. It shouldn't even be required, but its good to have to make sure everything is working.

Considering the code I posted breaks up a 1.8G file into ~115 chunks, all of which get MD5 checked, that's a little overkill. But, overkill is a good place to start from, and then work back.

There is crc, sha1, md2, md4. I'd choose the fastest.

I'd also look into the Indy FTP client/server. After all, FTP was designed for file transfers  ;D

Im sure the original poster will pick and choose what suits their needs.

Thaddy

  • Hero Member
  • *****
  • Posts: 16945
  • Ceterum censeo Trump esse delendam
Re: Hiring... Maybe..
« Reply #24 on: September 15, 2015, 01:26:00 pm »
If it is all about guaranteed delivery, look at the program flow first. It is hard to guarantee delivery anyway. I would go for highest probability. In that case an md5 hash is not an option unless used on a private network with very specific legacy needs. Md5 should be considered an indicator of correctness, not a proof of correctness.

Your assessment is correct insofar your correct interpretation of time vs compute. There are better options.
Due to censorship, I changed this to "Nelly the Elephant". Keeps the message clear.

snorkel

  • Hero Member
  • *****
  • Posts: 817
Re: Hiring... Maybe..
« Reply #25 on: September 15, 2015, 06:48:14 pm »
Could you use SSH and SFTP?
Devart makes a really nice product called Securebridge which implements a SSH/SFTP server in native FPC. (and client)
***Snorkel***
If I forget, I always use the latest stable 32bit version of Lazarus and FPC. At the time of this signature that is Laz 3.0RC2 and FPC 3.2.2
OS: Windows 10 64 bit

aidv

  • Full Member
  • ***
  • Posts: 173
Re: Hiring... Maybe..
« Reply #26 on: September 15, 2015, 08:10:45 pm »
Damn. Thanks everyone for your input, and thanks Geepster for your sample code.

And Snorkel: sure, why not, as I've stated before: I don't care how it works, it can be black-magic for all I care, this is for a commercial product so it is very important that it works properly.

RVK: yes that's true actually, but I see a lot of activity here in General and also I wasn't sure if I could get away with doing it myself with some help from you guys.

This is what I'll do, I'll experiment with Geepsters code for a couple of days and see if I figure it out.

If I don't, I'll come back with a full blown paid offer.

Thanks again guys!

aidv

  • Full Member
  • ***
  • Posts: 173
Re: Hiring... Maybe..
« Reply #27 on: September 16, 2015, 02:00:18 am »
Geepster, I'm trying your code but I get the following error:


Code: [Select]
unit1.pas(68,44) Error: identifier idents no member "URI"
At "URI" is supposed to be part of the class "TIdHTTPRequestInfo" inside the unit "IdCustomHTTPServer".

What's going on here?

Also, where is the unit "pl_indy" located? Because the project can't find that unit.

derek.john.evans

  • Guest
Re: Hiring... Maybe..
« Reply #28 on: September 16, 2015, 02:32:03 am »
What version of Indy are you trying to use?

You can either grab Indy 10 from here: (Currently 10 r5303)
https://indy.fulgan.com/ZIP/

Or, 07-07-2015 ver 10.6.2.10  SVN Rev 5284 from the CodeTyphon libraries:
http://www.pilotlogic.com/codetyphon/current/src/typhon_src.7z

With the Typhon package, you need to remove the requirement "bs_ideintf" and replace with "ideintf". It will then compile and install into Lazarus.

If you go with the latest Indy from fulgan, you need to add these 3 folders to your project path:
indy\Lib\Protocols;
indy\Lib\System;
indy\Lib\Core;

I prefer the Typhon option. Note: pl_indy is from the Typhon package. It can be removed if you go with the original. But, I think the original doesn't always have a working Lazarus package.

I dont like using design time Indy components anyway. I prefer tight reusable functions and classes.
« Last Edit: September 16, 2015, 02:39:54 am by Geepster »

aidv

  • Full Member
  • ***
  • Posts: 173
Re: Hiring... Maybe..
« Reply #29 on: September 16, 2015, 02:44:11 am »
I actually just found the Fulgan version haha

Ok so I'll try the CT one instead.

Just another question: When I try to install a package I get an error saying "Unable to find lazarus.pp."

Why is this occuring?

 

TinyPortal © 2005-2018