Spectral resynthesis and effects in The Grid

[andrei_olenev]

A spectral resynthesis Grid module, with a suite of spectral effects to modify the partials of the resynthesized audio.

The general idea of this is that audio would be analyzed (Fast Fourier Transform, FFT) and converted into a series of partials that represent that sound. Those partials could then be modified to achieve different types of effects.

Common spectral effects are: freeze, blur, stretch, and shift. Many other types of effects are possible as well. Like additive synthesis, common effects, like filters, compressors, and reverbs are all possible by modifying the partials of the audio.

Spectral effects have a unique sound. For example, time spectral time stretching is different sounding than granular stretching, and can achieve very wild timbres.

What problem(s) would this feature resolve?

Bitwig users looking for the characteristic sounds and effects of spectral resynthesis have to rely on third party products.

If Additive synthesis in The Grid is implemented, it should be taken into account that many additive synths (Loom, Harmor, Razor) allow for resynthesis to be the sound source. Even without additive synthesis, spectral effects are very powerful sound design tools.

How does this feature fit in Bitwig as a product?

Bitwig is known for being a powerful sound design platform. The addition of spectral effects would open up more possibilities in The Grid.

Is there already an alternative way to achieve this on Bitwig?

The Vocoder works similary.

Basic resynthesizers can be developed with The Grid. See for instance Bitwig Preset: Audio Resynthesizer 1.11.

Could it be implemented using Bitwig components or APIs, without Bitwig team’s support?

No

Could it be provided by a VST or something else reasonably integrated with Bitwig?

There are a handful of spectral effect plugins. Unfiltered Audio SpecOps is probably the best known currently. GRM tools has been doing it for a long time. AudioThing - Frostbite is another one. Melda has some spectral effects. There is a bundle of spectral fx blocks in Reaktor Blocks (by Salamanderanagram). There are Spectral effects in other modular environments, like Max4live and Kyma. There are also a few Eurorack modules, such as the Panharmonum, by Rossum Electro.

Are there other products that offer this feature?

All the programs mentioned above, and more

Relevant links (optional)

Additive synthesis in The Grid

A sine bank oscillator in the Grid together with partial shaping modules to give usable synthesis features, like waveshape, filters, effects etc… This would enable basic additive synthesis in Bitwig. It could also become a foundation for future related features. Additive synthesis is a sound synthesis technique that creates timbre by adding sine waves together (more). By adding and modulating hundreds of sine waves, the possibilities for new sounds and effects are endless. What problem(s) woul…

[andrei_olenev]

To my understanding, spectral resynthesis is the the process of audio being analyzed and then converted into a series of partials. It’s a similar principle to additive synthesis, because basically any sound can be made of sine waves. So, once a sound is converted to partials, those partials can be manipulated in various ways, to get different types of sounds. Common types of spectral effects are: Freeze, Blur, Stretch, and pitch shift. These have a unique sound. For example spectral stretching is very different sounding from granular stretching, because they are done differently. And, like additive synthesis, many other types of common effects, like filters, compressors, and reverbs can be mimicked in spectral effects by manipulating partials to sound like those effects.

Since spectral effects work by first converting the audio to partials, I thought it would be cool to just have a spectral resynthesis module, and a collection of modules that can be connected to that module to manipulate the partials to get different types of effects. I think that would be a cool approach, because it fits into the modular way of doing things in The Grid. There is a collection of spectral fx for Reaktor Blocks that work in this way (by Salamanderanagram).

The reason I think it would be cool to have these in The Grid, as opposed to a general Spectral effect device is that I just want to see The Grid be a more complete sound design platform. I really like using the Grid, and am excited for it to develop further. There are already great plugins for Spectral effects ( like GRM Tools, Unfiltered Audio - Spec Ops, Audio Thing - Frostbite etc…) So, for me personally, I’m much more interested in having Spectral Effects in The Grid.

Also, this would compliment the idea of having additive synthesis in The Grid, because they both work with Partials. So, hopefully, if we got partial modifying modules, they could be connected to a sine bank oscillator, or a spectral resynthesis module. I’m not sure how all that works, but since most additive synths have resysnthesis options for the sound source (Like: Razor, Harmor, and Loom), then I think that a suite of effects that work with modifying partials should work for both. I’m not sure about that though. As far as I know, the way that resynthesis is done in additve synths is different than how it is done with fft/spectral effects. But, the general idea of a modular approach to both in The Grid would be cool, and hopefully, they could integrate them both together.

Another way they could do spectral effects also, would be to just have a suite of spectral effects that stand alone, and don’t need a resynthesis module. But, again, I’m interested in seeing a general suite of partial based modules in the Grid. I think it would be very cool for example, if we get additive synthesis, to simply be able to swap out a sine bank oscillator in an additive patch, with a resynthesis module, like you find in additive synth plugins. Anyway, I’m probably rambling on enough. Hopefully all this makes sense.

[Fooltopia]

The problem I see with spectral FX is that they need to work on blocks of audio meaning they’ll introduce latency. How do other modular environments deal with this?

[andrei_olenev]

I’m not sure of that, but there are several examples of spectral effects in modular environments that can be looked at.

One of them of them is “Spectral Blocks” by Salamanderanagram. Since it is in Reaktor, you can look at the code, to see how it is done. He also has tutorials on making spectral effects. You need to sign up to his site though, to download the Spectral Blocks. It is a great resource though.
https://www.reaktortutorials.com/search/videos/spectral

Here is an example of spectral effects in VCV rack:

Tonecarver Blur

I don’t use it, but Plogue Bidule is supposed to have a bunch of spectral effects in a modular environment

Plogue | Bidule :: Modular Audio VST2/VST3/AU Host and Plugin, OSC

And, this is a eurorack example of spectral effects in the modular world:

Panharmonium

[Taika-Kim]

I have made resynthesizers in the Grid, IIRC that one was one of the very early ones.

There’s several problems. Even with serial BP filters for each band it’s hard to get a good resolution as the amount of banks is increased.

Also using oscillators is very CPU heavy. A typical. spectral effect has between 256 and 2048 bands. That’s two thousand oscillators max!

To my understanding, after the spectral analysis is done, the inverse FFT creates the emergent waveform without having to individually calculate the partials as oscillators.

And yes, the larger the window is (=more frequency and less time resolution), the longer the latency. This is the uncertainty principle in action, and AFAIK can’t really be avoided.

I’ve been requesting this for ages, let’s see if it eventually happens. They obviously have the code already at least for the analysis part, because they have a spectrum analyser. About their vocoder I can’t say if it just uses very optimised oscillators or an inverse FFT procedure.

[Taika-Kim]

For an example on optimising the latency by breaking down the FFT block to smaller parts
https://www.researchgate.net/publication/289209738_A_time_distributed_FFT_for_efficient_low_latency_convolution

Still, there would be some latency.

This is also the reason why I suspect Bitwig didn’t add any FFT based effects: they seem to avoid any processing which introduces latency.

[Jumpstar]

I’ve made spectral effects in the grid but the only way I could do it was using banks of high-Q band-pass filters to isolate the partials. Is that what you are talking about.

Yes, I’ve been asking for it for a long time as well, it would be really great.

[Jumpstar]

I’ve thought about it quite a lot and although I haven’t written that much audio synthesis software for a living I have done a bit on occasion.

I don’t really see a problem with the implementation side.

The way I see it you would have a grid module that converts to frequency domain and another one that converts back to time domain, and you’d have a bunch of modules that only work in the frequency domain. Shifters, filters, math and convolution operators, modulators, that kind of thing.

Internally it could just pass a fixed allocated block of frequency bands from one operator to the next, so it could be quite efficient.

I don’t know how they implement filters like EQ-5 right now, but my guess would be it would be cleaner to do it in the frequency domain, so maybe they have a transport for frequency domain internally already.

As far as block sizes, yeah, difficult to know how they handle audio blocks internally and how latency is managed. I know they must already be doing a lot of magic, as the way things are synchronized appears to be at a much higher level than the block sizes you specify at the audio card level. But again, who knows.

Anyway I vote for it for sure, it would be very cool.

[Taika-Kim]

Yes I’ve used sharp filters for isolation. That’s a good point about having separate modules for only processing stuff in the frequency domain.

[z_z_z_z]

It would be amazing if a FFT module and iFFT module could be added, then you could operate on blocks in a mathematical way of your choice in between the two in the FFT domain.

This has been done in max/bidule etc. before so it is not unreasonable to expect this in bitwig at some point.