[sdiy] Re: Information Content of Signals

[Grant Richter]

My question actually comes from page 47 of that paper. Channel capacity is
expressed as

Capacity = Bandwidth * log (Signal + Noise / Noise)

But is not defined as Noise approaches the limit of zero. Assuming log
(infinity) = infinity.

Philosophically, if we are interested in the ABSOLUTE channel capacity, the
absolute signal power is the total of all signal sources natural and
manmade. There is no Noise term because it is included in the Signal term.

So the idea that channel capacity approaches infinity as noise power
approaches zero, is not intuitive. Quantum mechanics should indicate there
is a limiting factor, and that is what I am trying to figure out.

Boltzman's constant would seem to suggest that there is minimum
thermodynamic bit at any temperature. This could possibly be used to compute
an absolute thermodynamic channel capacity from temperature and power?

(An erg is the same as a watt-second, according to my reference)

I am way out of my depth, but thrashing around in deep mathematical waters
can be very enlightening. And I'm counting on Magnus to throw me a life
preserver.

> From: Magnus Danielson <cfmd at swipnet.se>
> Date: Sat, 17 May 2003 15:47:30 +0200 (CEST)
> To: grichter at asapnet.net
> Cc: synth-diy at dropmix.xs4all.nl
> Subject: Re: Information Content of Signals
> 
> From: Grant Richter <grichter at asapnet.net>
> Subject: Information Content of Signals
> Date: Sat, 17 May 2003 02:36:44 -0500
> 
> Grant,
> 
>> As a point of interest, I have been trying to understand the theoretical
>> information content of signals.
> 
> 55 years ago Claude Elwood Shannon published a lovely article on the subject
> which I really hope you have read. It's the "A Mathematical Theory of
> Communication" which you can get from:
> 
> http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html
> 
> This is admittedly not the popular edition of the theory, but if is a very
> fresh and crisp reading which still makes very much sense half a century later
> like any good text should do.
> 
> He works out the formulas for both time-discrete and time-continous signals.
> The information content with and without presence of noise.
> 
> In relation to the other stuff you are talking about, this text should give
> the
> reference-floor for any further discussions.
> 
> As for the noise-floor of sound, this is difficult. It is highly temperature-
> dependent since the energy distribution of the atoms/molecules bumping around
> in the air is temperature-dependent. Just to make the whole matter even more
> interesting sound waves is really the deviations in pressure and particle
> velocity (which is zero in equilibrium). There are several good books on sound
> and I tend to drop into the works of Olsen, Beranek and Morse for a good
> reference.
> 
>> Contemplation of this has led me to a
>> paradox, and I wonder if anyone can shed light on this.
>> 
>> Information and thermodynamics are linked together by the expression of an
>> information "bit" in a thermal sense. The best definition I have found uses
>> Boltzman's constant (minimum energy required to create a new thermal state
>> at temperature).
>> 
>> Using room temperature and the reference of 0 dB = 1 milliwatt, the
>> theoretical information content of a 1 milliwatt-second signal would be
>> (approx) 2^64 bits per second. So for sample rate of 192 kHz (~2^18)
>> theoretical bit depth would not exceed 2^46 (64-18).
> 
> 0 dBm <=> 1 mW
> 
> You need to properly define things energy of a sound-wave etc. For instance,
> the Sound Pressure Level (SPL) is really the excess pressure relative a
> reference pressure.
> 
> The amount of bits per second (bitrate or information rate per time-unit, but
> NOT bandwidth even if it is in popular use within the clueless network
> community) comes from relationsships of noise and bandwidth. This is a more
> complex analysis than the bits * samples per second that we usually do.
> 
>> The paradox comes from the idea of over-sampling a thermally maxed out
>> information channel. What information would then be gathered? Would it be
>> merely redundant information, or would the waste heat of the conversion
>> apparatus become a new signal generator?
> 
> Your encoder generates heat, which contribute to the entropy in the acoustical
> media, which effectively means more noise. Noise is best understood as an
> unwanted information source. The charactersitics of a noise source may be
> known, but you may not have a perfect estimator for compensation, so then
> statistics have to work for you instead.
> 
>> Note that from an information theory standpoint, thermal noise is a
>> naturally occurring signal (reverb tail of the Big Bang?) mixed with the
>> human generated signal.
> 
> There are many non-human sources... I can still hear birds sing forinstance.
> 
> Anyway, as for your paradox, I wonder if you just haven't gone astray in the
> theory.
> 
> Cheers,
> Magnus
>