add GranularPitchShift class

[FoamyGuy]

This adds a new class audiodelays.GranularPitchShift. It uses granular synthesis for pitch shifting, producing a an effect that sounds different from the existing PitchShift implementation, and offers the density and spread levers to further change the sound.

I have tested mostly on Fruit Jam using variations of the synthio based example code included in the docstrings, as well as a more complex sampler project code that plays back recording samples at different pitches.

[FoamyGuy]

I disabled bitmaptools on the archi device to make room. That device has a built-in microphone, but no build-in display. I figured it would be better to keep CIRCUITPY_AUDIOEFFECTS enabled which will include audiodelays and the new GranularPitchShift and instead disable the bitmap helpers.

[dhalbert]

The raspberrypi port defaults to OPTIMIZATION_FLAGS ?= -O3, which can make for large code. Several boards reduce that to -O2 so things will fit. Try turning CIRCUITPY_AUDIOEFFECTS back on, on the archi board, and adding

OPTIMIZATION_FLAGS = -O2

or even -Os if necessary, to mpconfigboard.mk.

[FoamyGuy]

Ooh, yeah using -O2 allows both bitmaptools and audioeffects to fit with a decent chunk of headroom to spare. Latest commit changes archi to use that and re-enables the module.

[dhalbert]

@relic-se Would you be willing to take a look at this?

One question I have is whether having both PitchShift and GranularPitchShift are sufficiently different that it's worth having both, or whether we should settle on Granular if it's more flexible. I have not listened to either.

[relic-se]

Only noticed one odd thing on my first visual review of the code. I'll have to run some audio tests for a subjective opinion and to make sure that everything functions properly.

[relic-se]

I'll have to look around, but isn't there some form of "clamp" macro that would clean this up a bit? If not, maybe something like MIN(MAX(spread, MICROPY_FLOAT_CONST(0.0)), MICROPY_FLOAT_CONST(1.0)). Not necessary, but something to think about.

[FoamyGuy]

There are helpers that validate a value is within a range and raise ValueError if it isn't. I'm not sure if there is one that clamps to the ends of the range instead of raising though.

Arguably this clamping logic is really argument validation and might be more appropriate on the shared-bindings side. I may refactor it over to there, if so I'll check for a relevant helper function, or use the MIN/MAX syntax that you suggested. I do think it's a bit nicer than the longer form if blocks.

[dhalbert]

There are a bunch of validators that could be used, and if you don't see the exact one, you could add one in argcheck.c. I merged already, but another PR is fine.

[relic-se]

@dhalbert In order to do a proper subjective assessment, I'll have to generate some dry and wet audio samples using both algorithms (AudioFileWriter should be perfect for this). I'll try to get around to that soon (unless you get to it first, @FoamyGuy).

The two algorithms do appear similar in their functionality, but with multiple read points (grains) instead of the single circular read which the windowed algorithm uses in PitchShift. If GranularPitchShift is a clear winner and has a similar memory/CPU footprint, then there'd be an argument to replace the original.

[relic-se]

Here are the results of my testing.

Code

import time
from audiocore import WaveFile
from audiodelays import PitchShift, GranularPitchShift
from audiofilewriter import AudioFileWriter
from synthio import LFO

import adafruit_wave

INPUT = "/StreetChicken.wav"

# read wave duration
info = adafruit_wave.open(INPUT)
duration = info.getnframes() / info.getframerate()
info.close()

# open wave file as sample
wav = WaveFile(INPUT)

def write(effect_class, semitones: int, lfo: bool = False) -> None:
    filename = "/sd/{:s}{:+d}{:s}.wav".format(
        effect_class.__name__,
        semitones,
        "_lfo" if lfo else ""
    )
    
    effect = effect_class(
        semitones=LFO(scale=semitones) if lfo else semitones,
        mix=1.0,
        channel_count=wav.channel_count,
        sample_rate=wav.sample_rate,
    )
    effect.play(wav)

    print("Writing to {} for {}s...".format(filename, duration))
    start = time.monotonic()
    with open(filename, "wb") as f:
        writer = AudioFileWriter(f)
        writer.play(effect)
        time.sleep(duration)
        writer.stop()
    print("Complete!")
    
    effect.stop()
    effect.deinit()
    
write(PitchShift, -5)
write(PitchShift, 5)
write(PitchShift, 3, True)

write(GranularPitchShift, -5)
write(GranularPitchShift, 5)
write(GranularPitchShift, 3, True)

Output

All files have been converted from WAV to MP3 at 192kbps for compatibility with GitHub.

Original.mp3

PitchShift-5.mp3
PitchShift+5.mp3
PitchShift+3_lfo.mp3

GranularPitchShift-5.mp3
GranularPitchShift+5.mp3
GranularPitchShift+3_lfo.mp3

Comments

I am surprised by the outcome here, and I don't think that the result is as obvious as I would have anticipated. In fact, I think the new algorithm does better in some areas and the old algorithm does better in others.

Issues with Testing

• The source audio features a wide spectrum of frequencies and is transient heavy. Definitely a difficult task for any pitch shifter. Using a simpler source (ie: single instrument) may produce different results.
• I am using the default arguments of each effect. It may be possible to tune them further for different results.

LFO (BlockInput)

When reviewing the GranularPitchShift code, this was something I was worried about. The grains store the current read_rate whenever generated. This makes sense to avoid artifacts caused by sudden changes to the semitones value either by the user or with a BlockInput object. The downside is that it doesn't react immediately to changes and ends up causing a "stepped" effect to the pitch shift rate which is audible.

Output Frequency

I think I need to do some FFT analysis here to determine the "winner". Ideally, there would be a perfectly shifted frequency spectrum from the input to the output proportional to semitones. But since we're not using more advanced pitch shifting techniques (which are likely not possible on MCUs), there's definitely a difference in the "EQ" of the output.

From my ear, I'd say that PitchShift has better reproduction of higher frequencies but boosts lower frequencies somewhat unpleasantly (at least when pitching up), and GranularPitchShift can reproduce bass frequencies more accurately but rolls off the higher frequencies, reducing the "fidelity" of the sound.

Transients

Both effects have a tendency to either cause delay to some transients (ie: kick and snare hits). I don't think this is something that can be easily overcome without utilizing more advanced algorithms.

Conclusion

I don't think there is a clear winner. I would love some comment from @FoamyGuy on the matter as how best to utilize the new algorithm and demonstrate its advances over audiodelays.PitchShift.

[dhalbert]

@relic-se So it sounds like it's not worthwhile to choose just one. That is OK with me. We can have both. The extra code size is not so large.

[dhalbert]

@relic-se Thanks very much for the detailed review. Based on what you said, let's have both pitch shifters for now, and people can experiment. If there is a way of combining them or refactoring that can be done later.

[FoamyGuy]

@relic-se I am not really familiar with the audio theory behind the different algorithms so I could not say with much confidence which types of audio each are best suited for. The concepts were new to me, I didn't know how it would sound compared to the existing implementation before I started. I was given the quest to implement the granular synthesis shift algorithm by ladyada.

In experimenting with it while developing, some of the biggest differences I noticed were with larger values for density and spread which starts to add more texture, or layers to the sound. Even for small/default values the source audio has an impact also. You're right about simpler sources, I did some testing with basic synthio tones and found the different filters creating very noticeably different results.

I do think it's worth having both options as well since they can create different sounding effects.