It's a bit of a confusing question, since a dedicated server is technically also a home computer (assuming it is located in your home). There really is nothing that definitively distinguishes a dedicated server from a home computer. It is all really on the configuration,software & hardware. And w/o providing any configuration, software, nor hardware details, it is difficult to answer the question.
That said, I don't know much about Roksbox. But Playon can run on any Windows computer. So if you run windows on your dedicated server, then you can run Playon. But playon configurartion is all gui based, so configuring it will require you to have a monitor, keyboard, & mouse connected to the dedicated server. Note that transcode files with Playon you will need a somewhat powerful cpu on the dedicated server.
My current favorite local streaming solution is Plex. Plex Media Server is available for OSX, Windows, and Linux so it should run on pretty much anything. Also, while it transcodes, it can also do what it calls "direct streaming". Direct streaming is when it detects that the video codec is supported, but no the container, it simply remuxes the file to a supported container. This is less cpu intensive than transcoding so it doesn't require as much power as Playon (which always transcodes). But if you run into a file that requires transcoding, the cpu load will be higher.