Homebrew PCBs

I was looking to get some drill bits but I only want to get just a few sizes, especially if I spring for the more expensive carbide types.  Just out of curiosity, to get an idea, I measured the lead diameters of a wide variety of components.  Surprisingly the diameters are all over the place - everywhere from 0.016" up to 0.032" in increments of 0.001" for the most common sized components.  I would like to stick to maybe just three sizes of drill bits to (hopefully) cover most parts I'll come across.  I'm thinking maybe 0.035" (#65 bit), 0.025" (#72 bit) and 0.020" (#76 bit) would cover most eventualities.

Are these three sizes workable or am I limiting myself too much?  I don't do a lot of PC boards so if I can keep down the number of sizes on hand this would be a plus.

(I'm in Canada so I'm limited to what I can get here, usually from machine tool suppliers like KBC.)

Cheers,
John