That toggle thing really is a fail.
Just replace it with a socket head bolt (yes it will be metric), or at least cut the rod off and take it out.
Yes the rubber band will sort it out, but I would just hate looking at it all the time.
If you need to mount your drill without a tool just use a star shape or tri-lobe knob.
I also really like those clamp bolts which can be locked in any orientation (like, up for heaven's sake!) with a spring loaded clutch:
<
http://1.2.3.11/bmi/pregler-kg.de/images/categories/kipp/KI06460.jpg>
ST