How to stop downloading an audio stream in an audio tag?

Question

I'm trying to set up a live stream on a web page. I already have prelaod=none, which works until you click play, but then when I pause the player with

stream.pause()

it continues downloading. This is something everyone who goes down this path finds out. However, I expected that if I remove and replace the <audio> tag (and hence its enclosed <source> tag) it would reset the player to be as though it hadn't started yet.

stream.remove()
sibling.parent.appendChild(stream)

However, even if I just remove the stream and don't add it back in, server-side logs show the download continue until the page is reloaded.