The purpose of those lights was to give a glimpse of the status of the machine - which cores were idling and which were not. The upper 16 LEDs of each board were doubled. I assume the extra ones were used for diagnostics.
If your computer lives in a datacenter, far from view, there is little purpose on them. If, however, it's a smaller unit that lives on a desk or in an office, being able to quickly tell its state is interesting. Good visualization is an art.
Only slightly related, but imagine, if you take a huge server rack- and you insert hidden behind the front - a glas jar with a silicon-brain, into which tons of glowing wires run.
I want to hear the maintenance calls for that one.