OK, some clues; looks like it may be some sort of setting issue.
Back when I had last tested SMB transfer rates, I was running 10.4.2 on my mac and getting rates of 17-20 Mbps (which I guess is a little low but approximately normal). Before I upgraded to the 2.0.36 firmware, I ran the tests again (I now have 10.4.3 on my Mac) and got rates of around 10 Mbps. Not sure if the difference has to do with the update to OS X, but 10.4.3 definitely changes something with the NFS server on my mac; now I can't even mount a shared NFS directory on my local machine (but I digress).
After the updgrade to 2.0.36, as I mentioned before, I couldn't even measure the transfer rates. I was about to reinstall 2.0.35 this morning, but decided to take a look at my /etc/smb.conf file on my mac, and I noticed the following line which I had commented out:
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=2048 SO_RCVBUF=2048
This came from some post a few months ago here on the forums, and was supposed to speed up SMB transfer rates. In my hands, it made things considerably worse, so I commented it out. I decided to try, just for yucks, enabling it again, and my SMB transfer rates went back to 7 Mbps. Not great, I know, but at least it was working.
I've tried setting the two buffer options at 16384, which I gather is the new segment size in 2.0.36, but it hosed things again. I'm currently running with these two buffers at 4096 and getting transfers at 17.6 Mbps. I can play around with the segment size settings on both the mac and the Roku, but I don't really know what I'm doing and I'm a little nervous.
Any other mac users out there have any suggestions on what to do here? Or any of the Unix users have any suggestions on how I might change the smb.conf file so that things work better?