Reading the paper, for even the best books it only guesses a name in a passage right more then seventy percent of the time on 5 of the over 500 tested books. On “The fellowship of the ring”, it got hardly over 50%, and that’s hardly a little known book. These LLM’s are definitely familiar with the content, I would hardly call that memorizing verbatim. (Humans are also reasonably good at this after reading a book)
Reading the paper, for even the best books it only guesses a name in a passage right more then seventy percent of the time on 5 of the over 500 tested books. On “The fellowship of the ring”, it got hardly over 50%, and that’s hardly a little known book. These LLM’s are definitely familiar with the content, I would hardly call that memorizing verbatim. (Humans are also reasonably good at this after reading a book)