As soon as I gained access to OpenAI o1 model by API, I ran the latest o1, gpt-4o and claude sonnet LLMs through my private CFA level 1 exams.
Now o1 has officially scored in the 99th percentile with a correct rate of 90.7%, without any access to CFA textbooks or mock exams, just a calculator and good prompts.
| CFA Level 1 Chapter | Total Questions | Correct (gpt-4o) | Correct (sonnet) | Correct (o1) |
| Ethical and Professional Standards | 126 | 74.60% | 73.81% | 82.54% |
| Quantitative Methods | 91 | 75.82% | 75.82% | 94.51% |
| Economics | 84 | 88.10% | 90.48% | 94.05% |
| Financial Reporting and Analysis | 160 | 78.13% | 78.75% | 91.88% |
| Corporate Finance | 67 | 79.10% | 71.64% | 89.55% |
| Equity Investments | 85 | 74.12% | 69.41% | 94.12% |
| Derivatives | 43 | 74.42% | 83.72% | 95.35% |
| Fixed Income | 89 | 73.03% | 71.91% | 88.76% |
| Alternative Investments | 37 | 83.78% | 81.08% | 97.30% |
| Portfolio Management | 57 | 77.19% | 75.44% | 85.96% |
| Total | 839 | 77.47% | 76.76% | 90.70% |
It's worth noting that o1 scored particularly well in investment-related chapters which are the most relevant to the target users of SigTech.
I expect the o1 performance in "portfolio management" and "fixed-income" to be significantly improved if it's given access to our quant framework as tools.
I also expect the incoming o3 model performance to be >95% correct rate, i.e. 99.9th percentile.
I define AGI as a LLM that's able to do almost all knowledge work in the 99.9th percentile. I fully believe that it will be here by the end of 2025.
It will be expensive to start with but the cost is on an exponential curve of decreasing 90% every 6-12 months.
In 2025, our entire business including our product MAGIC and organisation itself must be built for the arrival of AGI. There is no playbook so we have to think creatively and from the first principles.
I hope everybody has a good holiday to rest and reflect. See you in 2025.