Announcements => Third party => Topic started by: schuler on April 27, 2021, 03:33:13 am

Title: FPC Neural Networks + Places2: 1.8 million training images - TESTING HAS STARTED
Post by: schuler on April 27, 2021, 03:33:13 am
 :) Hello everyone :)
I think that the importance of the moment deserves a new topic.

Previous methods of the API load the full set of training images into memory. This was ok or even stretched with:

For a dataset with 1.8 million images, keeping everything into RAM wouldn't work. Places2 standard has 1.8 million training images:

For my own tests, I'm downloading "Small images (256 * 256) with easy directory structure with 21GB".

A solution for this is currently in testing. The folder structure is loaded into RAM with:
Code: Pascal  [Select][+][-]
  2.     FTrainingFileNames, FValidationFileNames, FTestFileNames: TFileNameList;
  3. ...
  4.     ProportionToLoad := 1;
  5.     CreateFileNameListsFromImagesFromFolder(
  6.       FTrainingFileNames, FValidationFileNames, FTestFileNames,
  7.       {FolderName=}'places_folder/train', {pImageSubFolder=}'',
  8.       {TrainingProp=}0.9*ProportionToLoad,
  9.       {ValidationProp=}0.05*ProportionToLoad,
  10.       {TestProp=}0.05*ProportionToLoad
  11.     );

There is a new fitting class capable of working with the above:
Code: Pascal  [Select][+][-]
  1.     NeuralFit := TNeuralImageLoadingFit.Create;
  2.     ...
  3.     NeuralFit.FitLoading({NeuralNetworkModel}NN, {ImageSizeX}256, {ImageSizeY}256, FTrainingFileNames, FValidationFileNames, FTestFileNames, {BatchSize}256, {Epochs}100);

Is it bug free? I don't know. It's currently in testing. My current test case is modifying a source code intended for Plant Village dataset by just switching the folder name:

When creating the model (the neural network), the number of classes comes straight from the dataset:
Code: Pascal  [Select][+][-]
  1. NN.AddLayer([
  2.       TNNetInput.Create(FSizeX, FSizeY, 3),
  3.       ...
  4.       TNNetFullConnectLinear.Create(FTrainingFileNames.ClassCount),
  5.       TNNetSoftMax.Create()
  6.     ]);

:) Wish everyone happy pascal coding. May the source be with you. :)
TinyPortal © 2005-2018