Emacs dev here. The primary cause is something else.
Emacs's inefficiency in handling files with long lines is due to two factors: (a) The primitive unit of work for the display engine (the code that determines how to combine text, font metrics, syntax highlighting, inline image display, etc.) is a line (a newline-delimited span of characters). (b) The redisplay routine is called very frequently—not just when the screen is repainted, but pretty much any time the screen location of a buffer element need to be calculated, and so, e.g., during navigation. So Emacs is constantly, under the hood, going back to the previous newline and re-calculating how the buffer contents from that point forward should be rendered.
I’m not an Emacs dev so I’ll concede your expertise here - but if it is fundamental to the core navigation and redisplay, why does “stepping down” modes help so much? Fundamental and text mode certainly aren’t great with long lines, but are usually orders of magnitude better (eg a few seconds per command instead of 10s of seconds).
(I have not tried so-long-mode and am mostly on Emacs 26 with some 25.)
What you're observing is caused by what I'm describing. The features of these other modes are expensive because they do things that implicitly invoke redisplay.
Consider an ostensibly simple operation like determining the column in which a particular character appears. The answer to that isn't the number of characters since the last occurrence of `\n`, because whatever modes are active can inject arbitrary spacing, or display particular strings as something shorter or longer (a trivial example is the mode that causes `lambda` to be displayed as `λ`), or cause some characters to be displayed in a non-roman font where a single glyph is more than one column wide, and so on. To determine the character's column accurately you need to take into account everything that could affect how you'd paint the screen at that position, i.e., run the whole redisplay loop for some portion of a buffer. The richer the set of active modes, the more expensive it is to do that and the more often it is done.
Emacs's inefficiency in handling files with long lines is due to two factors: (a) The primitive unit of work for the display engine (the code that determines how to combine text, font metrics, syntax highlighting, inline image display, etc.) is a line (a newline-delimited span of characters). (b) The redisplay routine is called very frequently—not just when the screen is repainted, but pretty much any time the screen location of a buffer element need to be calculated, and so, e.g., during navigation. So Emacs is constantly, under the hood, going back to the previous newline and re-calculating how the buffer contents from that point forward should be rendered.