I'm starting to think that we will see model fragmentation based on alignment preferences. There are clearly applications where alignment is necessary, and there appears to be use cases where people don't mind an occasionally falacious model - I'm unlikely to get/care about objectionable content while coding using a local LLM assistant. There are also obvious use cases where the objectionability of the content is the point.

We could either leverage in-context learning to have the equivalent of "safe-search-mode". Or we will have a fragmented modeling experience.

Yeah, this seems very possible—it will be interesting to see where this goes if the cost of RLHF decreases or, even better, people can choose from a number of RLHF datasets and composably apply them to get their preferred model.

And true that objectionable content doesn't arise often while coding, but the model also becomes less likely to say "I can't help you with this," which is definitely useful.

In my fantasy world, RLHF algorithms become efficient enough to run locally such that I can indicate my own preferences and tune models on them.

How are you patching that in? Running an LLM locally for autocomplete feels a lot more comfortable than sending code to remote servers for it.

(Edit: Found the docs. If you want to try this out, like I did, it's here https://continue.dev/docs/customization#run-llama-2-locally-... )

We have the user start Ollama themselves on a localhost server, and then can just add

``` models=Models( default=Ollama(model="llama2") ) ```

to the Continue config file. We'll then connect to the Ollama server, so it doesn't have to be embedded in the VS Code extension.

(Edit: I see you found it! Leaving this here still)