The way the language model is trained is by rewarding it for correctly predicting the next word in a sequence.

The output of the model is a predicted probability distribution of the next word and a “state” — the next iteration takes the state output of the previous interation and generates another word and state (and this process repeats many times).

Since there’s a probabilistic dimension, what may have happened in this case is that it happened to repeat once by chance and the model had learned that if something repeats 2x it’s likely that it will repeat a third, fourth, and fifth time.

Basically it’s just trying to game the loss function which rewarded it for predicting the next word in the sequence correctly.

Thanks for the explanation. Your description superficially reminds me of a Markov chain (https://en.wikipedia.org/wiki/Markov_chain). Is this related or is it totally different?

I haven't read the paper the work is based on, but if the RNN outputs a probability distribution for the next letter/word then they form Markov Chains (since then they only depend on the current state and not the previous state)!

RNNs are just fancy parametric functions that take a (state, input)-pair and return a new (state', output)-pair.

From what I understand, repeated output is a common failure state with neural net stuff in particular, though I don't know why.

Because it needs to determine if a string is a '????? '''''?????, of course.