QuickTime-based video preview has been removed from After Effects. It relied on a QuickTime API that Apple has deprecated, and calling this API was instigating After Effects CC (12.x) to crash under common circumstances. These were motivating factors for why QuickTime-based video preview was replaced with Mercury Transmit, and we are pleased that Mercury Transmit also provides additional functionality and performance improvements.
I want to repeat and clarify that Mercury Transmit is designed to handle non-standard frame formats, and that this works well with other Mercury Transmit devices. We talk with Blackmagic regularly and they are aware that this limitation occurs in how they choose to implement the Mercury Transmit API. We will continue to discuss this issue with them.
The best workaround I can advise for now is for you to nest your comp into a comp that uses a supported frame format. I realize that this is neither ideal or convenient, but it does put your frames onto the video preview monitor.