Transformers like Llama use rotary embeddings which are applied in every single ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		topwalktown 10 months ago \| parent \| context \| favorite \| on: The FFT Strikes Back: An Efficient Alternative to ... Transformers like Llama use rotary embeddings which are applied in every single attention layer https://github.com/huggingface/transformers/blob/222505c7e4d...

markisus 10 months ago [–]

Very interesting! Do you know if there were any studies about whether this improves performance?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact