All of the Roku HD audio and video outputs are active at the same time, which is a good thing because you can have sound simultaneously from a TV and a receiver (and even through a DVD recorder for non-DRM channels like YouTube) originating from different Roku cable connections at the same time. The sound outputs whether through HDMI, analog audio, or optical digital audio are at fixed levels coming from the Roku. It doesn't have a volume control on its audio outputs; but of course volume levels vary between channels and programs, sometimes significantly.
The most likely cause is that your mom was hearing sound through the TV's HDMI connection. If the receiver's master volume was low or muted, you wouldn't hear sound coming from the Roku connection to it, whether through analog or optical digital audio connections, no matter which set of input jacks you connected the Roku audio output cables to at the receiver end.
I have two Roku HD boxes, one connected via analog audio cables to an older receiver, and one connected via optical digital cable to a newer home theater receiver. Both types of Roku audio connections respond normally to both receivers' master volume and muting controls. Netflix doesn't offer 5.1 encoding yet, but the attachment is simpler if you get an optical digital cable if the receiver supports that. And if using analog audio cables to a receiver, there's no reason to select 5.1 in the Roku sound options anyways.
You wouldn't have to disable the TV's sound through its HDMI connection to the Roku. Just leave it in place and turn the TV volume setting down to '0'. Sound through any receiver/speaker combo beats most tiny amplifiers and speakers in almost all flat panel HDTVs today anyways. I leave both HDTV's volume at '0' when using my Roku boxes since both are connected to home theater setups.
It sounds like part of the problem is that you are not in the location of the equipment if you're communicating with your mom via Skype? Hopefully it's just a simple connection or operator error problem.