I guess this is not working with the standard tools because panning affects the extent of the entire chart, not that of a series. The PanDrag tool (like the other extent tools) does not know which series you clicked to drag, you even can begin dragging when you click on an empty space on the chart.
What you need is a combination of an extent tool with a datapoint tool. You either must write your own chart tool, or you use a TUserDefinedTool and take advantage of the mouse events available here for any purpose. Your code must detect the series clicked (study TDataPointTool.FindNearestPoint in TATools.pas), and it must follow the mouse motion such as in TBasicPanTool.PanBy (same unit), but discard the change of the y direction in the new extent (variable "ext").