Part of the problem is that Cu is generally unpleasant to machine; it tends to pull the tool in. That same pulling can lift small pads/traces off the board if the drill is constrained. Center punching subjects the board to mechanical abuse that weakens the bond between the Cu & the board.
You're probably going to trash a few bits & some boards while finding the tools & techniques that fit your budget.
Roy