It almost looks as if you connected your interface’s spdif to Nuendo’s mixer under the output tab, instead of doing that in the “studio” tab. When using control room you’ll set to your destination there, not using “outputs”. You can almost think of the mixer as one software and control room as another. The input/output tabs belong to the mixer, and the studio tab control room. So control room allows you to “tap” the signals on the mixer, which means that you don’t want to select your monitor output in the outputs tab (mixer) but instead the studio tab.
The way I have it set up is I have a bunch of channels in the “output” tab, and none of them actually go to a physical output. Instead, I treat them as the “don’t touch” last-stage in my signal chain. All my stems and mixes end up on these outputs, and they don’t go anywhere. Then I use Control Room to monitor those outputs, and in the studio tab I select as my destination my surround speakers, my Grace interface for headphones or ‘crap’ speakers, or just a stereo set of speakers. So those destinations are all set in the studio tab, not in the output tab.
So, maybe you can post a picture of your routing, because it sounds as if something is off.