How to remove nasty brown noise from segment of an audio

SpectraLayers may not be the right tool for that particular task. Sounds like you might need to use a different technique altogether. Perhaps something like RX7 Dialogue Isolate, for example.