Skip to content
TopicTracker
From HackerNewsView original
TranslationTranslation

Benchmark unlimited Claude.md files against eachother

Clawmark is a tool that benchmarks unlimited Claude.md files against each other, allowing users to compare performance across different configurations or versions.

Background

- **Clawmark** is an open-source tool for benchmarking Claude.ai project knowledge (`.md`) files — letting users test which version of a prompt, instruction set, or context document performs better at a given task. - It works by running multiple `.md` files against the same test queries and scoring the results, so users can systematically compare different versions of their Claude project instructions. - This matters because Claude's "Projects" feature lets you upload custom knowledge files, but there's no built-in way to A/B test or measure which version of those files yields better outputs — Clawmark fills that gap for power users and prompt engineers. - The repo is by Emiliano Lugo and is aimed at developers and advanced Claude users who want to iterate on project instructions in a data-driven way.

Related stories

  • Newer Claude models sometimes invent extra keys in tool call arguments, breaking validation in Pi's edit tool. The author suspects post-training for Claude Code's forgiving harness makes alternative schemas fail. This suggests closed RL training can degrade general tool-use reliability.

  • Simon Willison released llm-coding-agent 0.1a0, an experimental coding agent built on his LLM library. The agent provides tools for reading, editing, searching files and executing commands, and ships with a Python API and CLI. It was developed using Claude Code (Fable 5) via TDD with a spec-first approach, and is available as a slop-alpha on PyPI.

  • Truth Social remains an outlet primarily for Trump's own posts, while other administration officials continue using X. The platform functions as a blog-like channel for Trump's messages rather than a genuine social network competitor.