[sdiy] Information Content of Signals

[Magnus Danielson]

From: "John L Marshall" <john.l.marshall at gte.net>
Subject: Re: [sdiy] Information Content of Signals
Date: Mon, 19 May 2003 18:02:21 -0700

> <snip>
> > Cheers,
> > Magnus - not sure how many bits of information I am really transmitting...
> </snip>
> 
> Isn't it the received bits of information that really counts?
> But then, I'm not sure that I am decoding all of the bits correctly.

But this is the point really... if I succseeds to transmitt X bits of
information, then would an optimal receiver be able to retreive all those X
bits (recall, we talk amount of information, not amount of encoding bits, which
is _MORE_ ) but if we assume that you are not an optimum receiver (which some
consider a fair assumption in your case for the topic at hand) you will
receive Y bits of information where Y is equal or less than X. It won't be more
since I only sent you X bits of information and if my communication channel
maximally allowed X bits and I used an optimum transmission for that channel,
then you should be able to retreive all of them.

Now, what IS information and what IS information transfer?

Well, this is a bit hard, but lets say I have "a state of mind" (let's assume
for a minute and the sake of the argument that all rumours of me having lost my
mind is false), "know" something. This then would be some form of information.
As I express them through a gazillion of bit-multiplication through speach,
text, pictures or whatever, but hopefull the information in my mind is being
conveyed through that "modulation". I better shape it in a way that I suspect
that the reader would be able to understand. For instance, if I would write
this text in Swedish I know for a fact that most of the readers of this email
would not be able to grasp even the basic flow of thought. Why this care in my
modulation? Well, if I want the same information to appear at some other point
in space, I need to transfer it there, and then I have to pass through
communication channels of various characteristics, and my ability to get the
message across will depend on how quickly I try to sent it and how I formulate
it. Then again, I could use feedback to learn if there where an error in how
the message where received and attempt to transmitt corrections in forms of
clarifications, further transmission in other forms (in hope they better
convey it through the channel) etc. etc.

To some degree information feels much like porn: "You can't define it but you
know it when you see it". The thing is, you can actually define it. If we
decode a message, for each phase of decoding we toss transmitted bits of
redundancy over the shoulder (this is actually an expensive thing to do - yes,
it consumes ENERGY) when we are *completely* done we even passed the test of
"making sense" and the few bits of information we do have in our hands (minds
actually) is really bindboggeling supprising to us, since they are the truely
new part of the message, since they are the information of the message.
The information is thus what actually modifies the state the most, being the
most random and unpredictable you may come up with. Thing is, when we encounter
noise we shut that out since it only wastes energy for the higher level parts
of our brain, so when we hear noise that doesn't bring much information, but as
soon as a just a slight sine is tossed in there, we heard it comming, since
that was the news - "oh, they tossed me a sine in that noise!!!", but then
nothing new (this could naturally also be seen as information, but it bores us
in the long run).

How can thermal noise be so full of entropy? It is also full of lack of order,
and we can make sense of that - we call it noise! We can't much else sense of
it however. We might classify it as thermal noise maybe, or pink noise if we
get a toss of that, but that's not a whole lot of information anyway.

Information, entropy etc. is all so confusing, probably since it is such a
high entropy field and we don't know if it is due to the noise level, the high
content of information or just that we are not shapped to comprehend (being
able to understand, i.e. to decode) the message.

Cheers,
Magnus - the noise-maker, ensuring the high entropy to keep people awake/asleep