Learning to Replicate Expert Judgment in Financial Tasks
Thinking Machines Data Science team developed a machine learning model to automate expert judgment in finance by fine-tuning Llama 3 8B, achieving 80% agreement with senior analysts. The model uses probability-based reasoning instead of direct predictions, and passes critical industry disqualifiers, making it ready for initial deployment alongside a human-in-the-loop system.
Background
- The article discusses using machine learning to replicate expert human judgment in financial tasks—essentially training AI models to make decisions that typically require experienced finance professionals.
- "Expert judgment" refers to the subjective, experience-based decisions that human experts make when evaluating financial data (e.g., assessing credit risk, valuing assets, or detecting fraud), which are hard to encode as simple rules.
- The key challenge: expert decisions are often inconsistent or biased, but they are also the gold standard in many financial contexts. Teaching an AI to replicate them requires high-quality labeled data from human experts and careful handling of their inconsistencies.
- This is a practical application of "supervised learning" (training AI on human-labeled examples) applied to finance—a field where regulatory scrutiny and the cost of mistakes are very high.
- Why it matters: If successful, such models could make financial decisions faster, cheaper, and more consistent than humans, while also capturing the nuanced judgment that purely rules-based or purely data-driven systems miss.