How much visuals will there be in the largest projects? TCanvas based drawing has its limitations when it comes to performance, but you can definitely start with TPaintBox or TImage. Is the canvas smaller than 2000x2000? If so, you may get off with problem of scrolling by using TScrollBox as a container for your canvas object. You can make a toolbar using TPanel and placing buttons, dropdowns and other things in it.
But if you plan on having possibility for tens of thousands of objects and lines with vector accuracy for scaling, i'd use OpenGL based approach. I may be biased as developer for suggesting nxPascal, but you can definitely find or use many other choices too. You will need to define what "a line" is. Are you just going to use GL_LINE or a thin rectangle. You would have more control with the rectangle, and make it look like arrow links, animated etc.
Whichever option you choose, the selecting and moving part should come naturally by using the mouse events generated for the graphics context (onMouseDown, onMouseMove...). You know the position of cursor, and the position of each object. Find function that matches a point hit with rectangle, or write a new one.