1. Before rendering with Iray, check to see how much VRAM is consumed by the scene in iClone/Character Creator.

If the VRAM consumption of the scene is already taking up half of your system resources, then we recommend switching to the Minimal Mode under Preference > Real-time Render Options > Quality.

Minimal Mode will help you activate and deactivate certain options for real-time rendering and reduce the max texture size for the entire scene to 1 pixel. This helps to free up precious VRAM that Iray can use.
2. Iray provides the Max Texture Size and Texture Compression settings to adjust the load on the graphics memory.