LLMは進歩していないのか？
SWE-benchの結果から、LLMのコード生成能力が向上していない可能性が示唆されています。この記事では、ベンチマークの限界や評価方法の問題点を探り、実際の進歩を正しく測定するための課題について考察します。
関連記事
Gemini can identify public figures in images, while ChatGPT and Claude currently do not offer this capability. This represents a functional difference between major AI models regarding image recognition of people.
The article examines whether modern large language models can accurately count the number of 'b's in the word "blueberry," testing their ability to handle this specific adversarial question.
The article discusses using large language models to predict coffee preferences and suggests benchmarking with physical experiments. It explores the potential of AI models to understand and forecast individual coffee taste patterns.