[sdiy] Information Content of Signals

[Magnus Danielson]

From: "Czech Martin" <Martin.Czech at micronas.com>
Subject: RE: [sdiy] Information Content of Signals
Date: Mon, 19 May 2003 09:43:41 +0200

> In contrast to what SciFi movies tell us, (androids
> make chirping or other small bandwidth noise) efficient computer
> "language" sounds like noise, i.e. a broad band signal
> that fills the complete avaiable bandwith.

Exactly - by maximising the entropy!

This is actually one of the confusing matters, since a noise source has high
entropy.

A low entropy source is quite predictable. An entropy of 0 means a
predictability of 1, so any of the possible patterns of the source is
retransmitted continously. The maximum entropy means that for each symbol (in
discrete time) or for each time unit is the maximum information transmitted.

Interestingly enought this view on information sources proove very usefull in
cryptology.

So, what about noise (as in thermically noise) - which is not predictable, so
entropy is high, so isn't it transmitting information then? Well, in a sense it
is, it is transmitting nonsense information, it doesn't mean anything to anyone
except being just noise.

> I is easy to understand that more information can be send
> in this case.
> Older schemes like FSK work with narrow bandwidth, i.e.
> also slow modulation. But you could think of many FSK
> machines in parallel to fill the bandwidth. This
> would increase signal throughput.

Right. The good old SSB takes up half the space of old AM since it was realized
that only one side where needed. FM and PM could also be used. FSK is really a
form of FM modulation.

> To achieve a good, noisy signal
> phase and amplitude modulation is used at the same time, 
> the complex signal pointer get's longer and shorter
> and the angle is rotated at the same time.
> Usually the pointer has a set of NxN fixed points, called symbols,
> where it will settle until the next symbol is expressed.
> I have heard about 16x16 symbols. They sit on a rectangulkar grid
> in the complex plane arround the origin, a square so to say.

There exist both square-grid patterns as well as others, but when you have
64, 256 or 1024 QAM (Quadrature Amplitude Modulation) position systems the
square-grid dominates. This is combined with Trellis-coding. I must admitt that
Trellis-coding isn't my biggest stregth, but I should pick up on it as well as
Turbo-codes.

> This modulation method seems to be very robust against noise.
> Communication to very far away space probes seems also to be done
> this way.

Indeed. It is also how those "38,4 kBaud" modems work.

> And: modern modems analyse the incomming stream and try to compress
> this with classical methods, before modulation.

Right.

> You can notice that if you try to transmit a compressed archive.
> This can not be sqeezed more, so it takes longer then expected.

It is already very random, i.e. has already a high entropy, so compression is
hard. I never trust such compression but only counts on raw capacity. You can
even loose by using a simlpe compression scheme on high entropy data.
A compression scheme acheives compression by raising the entropy, i.e. by
spending less bits to communicate the same amount of information, i.e. by
spending less excess bits over the amount of bits needed for the information
content.

Cheers,
Magnus - not sure how many bits of information I am really transmitting...