I can not really say much about the audio quality as I am using my internal TV speakers. It sounds good to me though. I think a lot of audio from the N64 is low bitrare anyways due to the limited storage space on cartridges.
Do you think the audio is noticeably better when you use RCA? In that case you might have a point. I believe the hard part of the audio processing to send it over HDMI is that the N64 uses weird sample rates. Therefore, it needs to be resampled by a DSP to something common for HDMI such as 48 KHz. You will introduce some noise when doing that and it depends on your algorithm how much. As far as I know it is fairly hard to do this kind of stuff on an FPGA. I can almost only find papers when I search for stuff on it.
I haven't done a direct comparison of any kind on the topic, just FYI. Everything I'm saying is based on my memory of using RCA. The most obvious difference that I've seen is that there's a tiny bit of distortion present at some points with UltraHDMI that I don't remember being there when I was using RCA audio before. Not low bitrate distortion (like watery cymbals and things like that), but clipping type distortion. This could just be my implementation, as I have the UltraHDMI connected to a Yamaha Audio Receiver which outputs the audio. You may see no difference with internal speakers.
I think the biggest example I could give would be the introduction to Paper Mario. I'm sure it's low bitrate audio, but it doesn't have the characteristic sound of really low-end compression. However, with UltraHDMI, there's an added crackle/scratching sound at some high points in the intro.
Now, onto implementation. I have no clue how difficult it would be to implement the necessary audio conversions. IIRC, HDMI has a few different audio implementations you can use, but I highly doubt that the digital audio stream from the N64 fits any of those protocols. So we can only imagine the hassle of taking that signal and converting it to something that HDMI can actually use. I feel, though, that the clipping issue is due to the code used to implement the conversion from N64 digital audio to whatever HDMI audio standard Marshall chose to use, and as such can be fixed in a firmware update. It may just be that the resulting signal needs to be attenuated before it's sent to the end device. Because of this, I wouldn't be too concerned with the issue. Just test for yourself the difference between HDMI and RCA audio, and if HDMI is worse, be sure to follow firmware updates and retest once an update is released that seeks to improve fidelity.