First, load the narration. Render it as a wav. Use that new wav file only as the base for your production. Just a guess, but it might be that even though you delete the video track itself, it is still a sound track that's cued to a video, even though you are ignoring it on the timeline. It may be that timecodes and sample rate differences are causing problems and unexpected fx. If that's the case, the above experiment might work.