I don't think the "select() returned -1" error is fatal or directly related to the small number of files being found by Firefly. I get that message too on startup of Firefly and on my server it shows all my music files (5000 or so) to my Soundbridge.
What format are most of your music files in? If they are mainly WMA (and only a few in MP3 format), that might explain it, as out-of-the-box Firefly doesn't index or display WMA files. You have to go to the Advanced Configuration item in the Start menu (which opens Notepad on the configuration file for Firefly) and add WMA to the list of supported extensions (by adding ,.wma to the end of the list you'll already see there).
If you have already done that and/or the majority of your music is not in WMA format, then I don't know what else to suggest - if the above doesn't help can you provide an example of a music file (filename, path relative to the root of your music collection, format, how well it is tagged, etc) that isn't being shown in Firefly, and hopefully someone will be able to tell what the missing ingredient is to get it working properly for you.