You need either MPEG-1 or MPEG-2 video in order for the ROKu to play it.
Search the forum here for more details on particulars for VLC configuration to make working files for the Roku.
http://www.videolan.org/doc/streaming-h ... to-en.html
You can play with setting up VLC to stream the QT to your Roku. I just haven't done so as I prefer transcoding them offline and then sit back in front of the TV and watch a bunch of transcoded videos in one sitting.
If you don't want to mess with transcoding, you can try setting up a UPnP server on your Mac and point it to where your QT media is located, then connect to that UPnP server using the MPlay app on the Roku. Maybe someone who has done this can offer some tips, I don't have a Mac so not familiar with UPnP stuff for it.