[sdiy] Granular synthesis/graintable synthesis

[Scott Nordlund]

> Not sure if it falls or not in the scope of granular stuff, but I've
> worked on some code that "wavetablize" a (monophonic) sample. Here's
> how it worked:
> 1/ The f0 of the sample was detected using YIN, and the sample was
> chopped into 1 period chunks
> 2/ The MFCCs of each chunk were computed
> 3/ VQ was used to cluster the chunks into 16 classes, with mahalanobis
> distance over the MFCC as a distance metric. The output of VQ is a
> codebook of 16 single-cycle waveforms. Their period was resampled to
> 256 samples.
> 4/ Going back to the original sample, assign to each chunk its
> codeword index. This yields a kind of "compressed" representation of
> the original sample as a concatenation of the single-cycle waveforms
> extracted at the previous step. You can further remove duplicate in
> this list to get a Waldorf-style wavetable.

Robert Bristow-Johnson wrote a good paper on this, and wrote somerelated stuff on the music-dsp mailing list.
I'm working on an FFT-based additive/wavetable type thing in PureData that does something similar. If you're writing the inverse FFToutput to a double buffer and crossfading to the next waveform, and then play this at some arbitrary pitch, it's essentially equivalent to wavetable synthesis (and probably the Kawai additive synths also).In principle I guess I could "resynthesize" some input, but I haven'ttried this.
Alignment of phase between adjacent waveforms is important. A lotof "classic" synths just use a summation of sines for everything (maybe manipulating polarities to reduce peak amplitude).  This gives an instantly recognizable sound (and makes some other thingsmore convenient), but IMO it's important to have arbitrary controlover phase. It makes a big difference in the character of detuned oscillators. Also, a constant phase shift can simulate inharmonic partials; it just has to be relatively small from waveform to waveformor you'll get an unpleasant tremolo.
I'm sort of wondering if it would be practical to split the wavetablesinto a "harmonic band" with linear crossfades and no phase shiftbetween waveforms, and an "inharmonic band" with constant powersin/cos crossfades and a +/- 90 degree phase shift between waveforms.That's probably overdoing it, but some variation of that could benearly as flexible as brute force "sine bank" additive synthesis. At least it could make a large variety of novel detuning or "ensemble"effects...
Anyway, as far as granular synthesis goes, the specifics depend on what you're trying to accomplish.  It all basically amounts to playingsmall windowed portions of a source waveform, but there's a greatdeal of flexibility: you can play back the "grains" at the originalpitch, new pitch, or random pitch, play them at some periodic rateor randomly, control the grain lengths, density, etc.  It's a matter of whether you want to do formant stretching, pitch shifting, timestretching, or stochastic atonal soundscape type stuff. There doesn'thave to be a formula to follow, just be lucid about what options areavailable and how they affect the sound.