[sdiy] Re: Information Content of Signals

[Magnus Danielson]

From: Grant Richter <grichter at asapnet.net>
Subject: Re: [sdiy] Re: Information Content of Signals
Date: Sun, 18 May 2003 18:29:20 -0500

Hi Grant,

> I thought of a couple other ways to state the problem.
> 
> 1. I have a perfect white noise source connected to a power meter and a 600
> ohm resistor. I adjust the amplitude to 1 milliwatt. How many theoretical
> bits are available in this channel per second?
> 
> We know it can not be infinite, because if there is any correspondence
> between information and heat, then an infinite information source would
> generate infinite heat. So it must be finite, but how do we quantify it?

None. There is no "real" information being transfered since P is 0, so

          P + N         N
C = W log ----- = W log - = W log 1 = 0
            N           N

If you try to interprent this signal as some form of signal, you have an
endless source of useless information. I.e. the excess information the noise
represents is meaningless.

Only if you appended a signal with some bandwidth W and some power P we would
be able to judge how high capacity a perfect receiver would be able to extract
and thus the maximum information capacity C of that channel. The pure noise is
however meaningless and this is really equalent to say it actually doesn't
carry any information.

> 2. I connect an analog to digital converter across the 600 ohm resistor.
> Assuming I do not care about converter speed, what is the maximum number of
> bits the converter could have? Does converter speed affect the maximum
> number of bits?
> 
> That is, if the source is 2^64 bits per second, then can you get 1 reading
> of 2^64 bits or 2^32 readings of 2^32 bits, but not 2^32 readings of 2^64
> bits?

Again it is meaningless. A 0 bit AD gives as much information as a 64 bit AD,
namely 0 bits of information per sample.

This is really what takes some understanding to "get" namely that all changes
in some variable does not constitute "usefull" information. We should
understand noise as "not usefull" information, and this may be both cause by
physics and by interference of other systems. Which source of useless
information you have is not very interesting, just how much they all add up to.

It is worth it to read carefully through the original paper and slowly let
things "melt in". Terms like ergodic processes might at first not be very
natural, and some of the math may first be confusing (and that comes from me!),
but there are wisdome there and in related articles.

> Once again, thank you for any help clarifying these difficult ideas.

My pleasure! ;O)

Cheers,
Magnus