It's always helpful to cite the title(s) that you're playing when you see a problem like this. I see some of what you're talking about when watching Love and Friendship but not when I tried 3 or 4 other things. I see the same delay when playing that title in the Amazon app on this smart TV and the app on TiVo Bolt, so I suspect that it's an encoding problem.
As a matter of fact, I experienced this when watching Love and Friendship. What a coincidence! Apparently it's pretty widespread across all of the platforms, so you're probably right that it's a problem with Amazon's encoding or server or something. Irritating!