Two gestures do two different jobs, and they are kept apart on purpose. Sliding a row sideways sets how far along the task is. Dragging the handle at the left of the row moves the task. A short, accidental touch on the handle does not pick the task up, because the drag only starts once you have moved a few pixels.
That separation matters on a phone. You can scroll the list with your thumb, slide a task to update it, and drag a task to move it, without one of those turning into another.