Removing the app from your iOS device and re-adding it is the only way I've found to clear out devices.
As a bit of information, the device keeps not only the devices tied to your account, but any device you connect to. For example, if you're at a friend's house, and you connect to his Roku, it will continue to show in your app (though grayed out) when you return home.
When you install the app on your iOS device, it will show the Roku devices on your current network. When you log in, it will show any device on your account that's not on your network (say, a device you own that's at another location).
There appears to be no press-hold or swipe left option to remove grayed out devices.
Roku Ultra (4660)
Apple TV (5th gen), TiVo
Roku boxes from every generation.
Apple TV (2nd, 3rd, 4th gen)