It takes two things in order to twirl a pencil between your fingers.
The pencil and enjoyment of the activity enough to make you want to do so. Take away the enjoyment, and you'll wish to drop the pencil. All that's left now is to convince yourself into letting go.