You can try diagnosing the issue using the following methods:
1) Scene objects that aren't animated can be converted to static mode and 'Optimize Material' can be applied for optimal performance.
2) Apply 'Lite' mode to faraway MD characters.
3) If you already recorded a pass on an MD character, disable 'Activated' to convert it to a Lite character. This will free up resources reserved for MD.
4) If none of the steps above have not made a drastic improvement on performance, you can experiment with lowering the real-time render quality and testing its impact on the scene.