[sdiy] large numbers
rburnett at richieburnett.co.uk
Fri Jul 13 20:30:34 CEST 2018
Out of interest, what application requires table lookup of values to a resolution of better than 1 part in 10 to the 19 !?!?!?
Sent from my Xperia SP on O2
---- Tim Ressel wrote ----
>Verified. I don't relish (or other condiment...) doing that calc
>manually 4096 times. But it does tell us that a windoze machine can do
>the deed. I wonder if its a matter of floats versus doubles?
>On 7/13/2018 9:59 AM, Ben Bradley wrote:
>> Windows Calculator does about twice that ("In Scientific mode,
>> Calculator is precise to 32 significant digits."), though of course
>> using it is tedious. If it helps, you can copy-paste operations into
>> it. Copy and paste this:
>> (two to the power of 2 reciprocal - this calculates the square root of
>> two) gives:
>> There's a list of keyboard shortcuts somewhere.
>> If you're using integers, I recall that Python, rather than
>> overflowing, will handle arbitrarily large integer values.
>> On Fri, Jul 13, 2018 at 12:42 PM, Tim Ressel <timr at circuitabbey.com> wrote:
>>> I need to generate a lookup table with 64 bit values. Usually I use Excel
>>> but there is a problem: It only does 14 decimal places. I don't think this
>>> is enough. Any suggestions?
>>> --Tim Ressel
>>> Circuit Abbey
>>> timr at circuitabbey.com
>>> Synth-diy mailing list
>>> Synth-diy at synth-diy.org
>> Synth-diy mailing list
>> Synth-diy at synth-diy.org
>timr at circuitabbey.com
>Synth-diy mailing list
>Synth-diy at synth-diy.org
More information about the Synth-diy