I was about to like your post until I read your last sentence.
You definitely can't see nuclear transmutations optically. There is no way to optically distinguish if heat is coming from chemical or nuclear reactions.
You will have to observe higher energy photons (x-ray or gammas) for this.
The transmutation of elements as a source of energy might be detected by the observation of emission lines that the elements produce in an analysis of the light given off by the reactor.
For example, the emission lines in a spectroscopic analysis of emitted light would decrease in intensity while the emission lines of helium would appear and strengthen over time.