The first things that springs to my mind is that the spectrum of emitted light from an active display is different from that of reflected sunlight/artificial light off of paper made from pulp/linen/cotton.

Active displays can be fatiguing for a variety of reasons, including brightness as you mention, but also the "unnatural" light spectrum.

Also, of course, blue light is believed to affect your circadian rhythm, so can cause disturbances after sundown. Most ebook readers have adjustments for this though.