You don't say what model of Roku you're running. Assuming this a soundbridge, unless you're running an M1000 or M2000 you will need to install Firefly or a similar server software on your Mac in order to play your music files. Firefly would be the advised option.
Does the SB show in Bonjour on your Mac? What model does it specify when you boot the Soundbridge? Does internet radio play?
G5, OS X 10.5.8, 3 x M1000, iTunes 10.4, slimserver 6.2.1/alienbbc (now redundant), firefly svn 1586, uNSLUng 6.8