Adding the position vector is basic sure, but it's naive to think the model does... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		adroniser 24 days ago \| parent \| context \| favorite \| on: Starting from scratch: Training a 30M Topological ... Adding the position vector is basic sure, but it's naive to think the model doesn't develop its own positional system bootstrapping on top of the barebones one.

thesz 23 days ago [–]

For some reason people are still adding position encodings into embeddings.

As if they are not relying on the model's ability to develop its own "positional system bootstrapping on top of the barebones one."

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact