You could easily improve upon that. For example the following will change the cursor to a block in emacs mode, an underline in vi-command and a line in vi-insert in most terminals.

    set show-mode-in-prompt on
    set emacs-mode-string \1\e[1 q\2
    set vi-cmd-mode-string \1\e[3 q\2
    set vi-ins-mode-string \1\e[5 q\2

The worst thing about the bash config of readline is that bash disables the key combinations for going from emacs-mode to vi-mode (ctrl-alt-j) and back (ctrl-e). They work everywhere else (where I've checked anyways) and you can't turn it on again in inputrc, you have to bind it in your .bashrc, all that because the bash developers think it would be too confusing to have it work the same in bash as anywhere else were you have readline support.

What's the bind incantation in the bashrc you recommend? Here is what I tried:

    bind -m vi-command 'Control-e: emacs-editing-mode'
    bind -m emacs 'M-C-j: vi-editing-mode'

The second keybind does not seem to work. If I just make it control-j, it works. For some reason the meta is throwing it off.

You are looking for

    bind -m emacs-meta '"\C-j": vi-editing-mode'

and possibly adding

    bind -m vi-insert '"\C-e": emacs-editing-mode'

The default keybind is control-meta-j though. My comment was in reference to me not being able to get that to work.