[sdiy] Re: Information Content of Signals

[Grant Richter]

I thought of a couple other ways to state the problem.

1. I have a perfect white noise source connected to a power meter and a 600
ohm resistor. I adjust the amplitude to 1 milliwatt. How many theoretical
bits are available in this channel per second?

We know it can not be infinite, because if there is any correspondence
between information and heat, then an infinite information source would
generate infinite heat. So it must be finite, but how do we quantify it?

2. I connect an analog to digital converter across the 600 ohm resistor.
Assuming I do not care about converter speed, what is the maximum number of
bits the converter could have? Does converter speed affect the maximum
number of bits?

That is, if the source is 2^64 bits per second, then can you get 1 reading
of 2^64 bits or 2^32 readings of 2^32 bits, but not 2^32 readings of 2^64
bits?

Once again, thank you for any help clarifying these difficult ideas.

> From: Grant Richter <grichter at asapnet.net>
> Date: Sun, 18 May 2003 15:44:19 -0500
> To: Magnus Danielson <cfmd at swipnet.se>
> Cc: <synth-diy at dropmix.xs4all.nl>
> Subject: [sdiy] Re: Information Content of Signals
> 
> My question actually comes from page 47 of that paper. Channel capacity is
> expressed as
> 
> Capacity = Bandwidth * log (Signal + Noise / Noise)
> 
> But is not defined as Noise approaches the limit of zero. Assuming log
> (infinity) = infinity.
> 
> Philosophically, if we are interested in the ABSOLUTE channel capacity, the
> absolute signal power is the total of all signal sources natural and
> manmade. There is no Noise term because it is included in the Signal term.
> 
> So the idea that channel capacity approaches infinity as noise power
> approaches zero, is not intuitive. Quantum mechanics should indicate there
> is a limiting factor, and that is what I am trying to figure out.
> 
> Boltzman's constant would seem to suggest that there is minimum
> thermodynamic bit at any temperature. This could possibly be used to compute
> an absolute thermodynamic channel capacity from temperature and power?
> 
> (An erg is the same as a watt-second, according to my reference)
> 
> I am way out of my depth, but thrashing around in deep mathematical waters
> can be very enlightening. And I'm counting on Magnus to throw me a life
> preserver.
> 
>> From: Magnus Danielson <cfmd at swipnet.se>
>> Date: Sat, 17 May 2003 15:47:30 +0200 (CEST)
>> To: grichter at asapnet.net
>> Cc: synth-diy at dropmix.xs4all.nl
>> Subject: Re: Information Content of Signals
>> 
>> From: Grant Richter <grichter at asapnet.net>
>> Subject: Information Content of Signals
>> Date: Sat, 17 May 2003 02:36:44 -0500
>> 
>> Grant,
>> 
>>> As a point of interest, I have been trying to understand the theoretical
>>> information content of signals.
>> 
>> 55 years ago Claude Elwood Shannon published a lovely article on the subject
>> which I really hope you have read. It's the "A Mathematical Theory of
>> Communication" which you can get from:
>> 
>> http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html
>> 
>> This is admittedly not the popular edition of the theory, but if is a very
>> fresh and crisp reading which still makes very much sense half a century
>> later
>> like any good text should do.
>> 
>> He works out the formulas for both time-discrete and time-continous signals.
>> The information content with and without presence of noise.
>> 
>> In relation to the other stuff you are talking about, this text should give
>> the
>> reference-floor for any further discussions.
>> 
>> As for the noise-floor of sound, this is difficult. It is highly temperature-
>> dependent since the energy distribution of the atoms/molecules bumping around
>> in the air is temperature-dependent. Just to make the whole matter even more
>> interesting sound waves is really the deviations in pressure and particle
>> velocity (which is zero in equilibrium). There are several good books on
>> sound
>> and I tend to drop into the works of Olsen, Beranek and Morse for a good
>> reference.
>> 
>>> Contemplation of this has led me to a
>>> paradox, and I wonder if anyone can shed light on this.
>>> 
>>> Information and thermodynamics are linked together by the expression of an
>>> information "bit" in a thermal sense. The best definition I have found uses
>>> Boltzman's constant (minimum energy required to create a new thermal state
>>> at temperature).
>>> 
>>> Using room temperature and the reference of 0 dB = 1 milliwatt, the
>>> theoretical information content of a 1 milliwatt-second signal would be
>>> (approx) 2^64 bits per second. So for sample rate of 192 kHz (~2^18)
>>> theoretical bit depth would not exceed 2^46 (64-18).
>> 
>> 0 dBm <=> 1 mW
>> 
>> You need to properly define things energy of a sound-wave etc. For instance,
>> the Sound Pressure Level (SPL) is really the excess pressure relative a
>> reference pressure.
>> 
>> The amount of bits per second (bitrate or information rate per time-unit, but
>> NOT bandwidth even if it is in popular use within the clueless network
>> community) comes from relationsships of noise and bandwidth. This is a more
>> complex analysis than the bits * samples per second that we usually do.
>> 
>>> The paradox comes from the idea of over-sampling a thermally maxed out
>>> information channel. What information would then be gathered? Would it be
>>> merely redundant information, or would the waste heat of the conversion
>>> apparatus become a new signal generator?
>> 
>> Your encoder generates heat, which contribute to the entropy in the
>> acoustical
>> media, which effectively means more noise. Noise is best understood as an
>> unwanted information source. The charactersitics of a noise source may be
>> known, but you may not have a perfect estimator for compensation, so then
>> statistics have to work for you instead.
>> 
>>> Note that from an information theory standpoint, thermal noise is a
>>> naturally occurring signal (reverb tail of the Big Bang?) mixed with the
>>> human generated signal.
>> 
>> There are many non-human sources... I can still hear birds sing forinstance.
>> 
>> Anyway, as for your paradox, I wonder if you just haven't gone astray in the
>> theory.
>> 
>> Cheers,
>> Magnus
>> 
> 
>