Why My Machine Learning Trading System Couldn't Justify a Startup

This is the story of how I spent 21 months building a trading system, wrote down what would be needed to turn it into a company, and then had two AI models tell me that I should not do it.

If you’ve read my last three posts, you know about TopSet. It’s the system I built, mostly with AI coding agents, to use machine-learning models to choose a group of stocks and trade them with real money. The first post was about the automatic tests that made it safe to run code written by AI agents. The second was about how information from the future accidentally got into my models. The third was about random noise in the results. It ended with a promise to explain what I decided to do after learning that my model’s results could not be clearly separated from luck. This is that post, and you don’t need to read the others to understand it. There are no charts showing returns, and that is intentional.

Why Did I Give Myself a Deadline?

I started TopSet because I saw promise in using machine learning to find patterns in the stock market. My early results looked good enough to make me think I had found something useful. But I also knew that a project like this could take years without ever answering the main question: could it become a business? So I set a rule for when I would stop. I would spend about two years working on it full time to find out whether there was a real business opportunity. If there wasn’t, I would return to leading engineering teams.

The first live trading bot started running on December 2, 2024, and I began working on the project full time in January 2025. By September 2026, the system had been running continuously with real money for almost two years. During that time, I had worked through about 1,600 pull requests, each one a reviewed change to the code. My deadline was close, and I needed an answer I could trust.

To be honest, I already had a strong feeling about the answer. Around the start of the summer of 2026, I began to think that my results were not strong enough, over the long term, to justify starting a company. But when you want a different answer, it is easy to talk yourself out of what you suspect. I wanted a process that would either confirm my doubts or convince me that they were wrong.

How Do You Ask an AI Whether to Start a Company?

Most of the work happened before I asked the question. The obvious approach would have been to open a chat and type, “Here’s my trading system. Should I start a company?” I would probably have received an encouraging answer because I would have described the system the way founders usually describe their ideas. Instead, on September 17, 2026, I first wrote two documents with help from a Cursor agent.

The first document was a guide to everything in the project’s code repository. It explained what had been built, what was running live, and, just as importantly, what had already been tested and abandoned. It listed every completed experiment, its result, and a link to the place where the decision had been made. The goal was to make sure the evaluation started with the actual record, not with my sales pitch.

The second document explained how to evaluate the project using a series of gates. Each gate asked a question and included a written rule for what would count as failure. This stopped the evaluator from changing the definition of failure after seeing the results. There were five gates:

1. Difference from other products. What would a customer get from this that they could not get from a cheap index fund, which simply holds all the stocks in an index such as the S&P 500?
2. Evidence. Did the case for starting a company depend on anything that had not been proven, or on anything I had already tested and abandoned?
3. Following the rules. Could I honestly market the product while following the rules for investment advisers? In the US, people who get paid to manage or advise others about their investments generally must register with regulators. There are also strict rules about what they can claim about investment performance.
4. Business costs and income. Could the business work for one person without raising money?
5. My next step. Should I start a company, run a 90-day test to see whether anyone would pay, keep using the system only for my own money, or stop and look for a job?

For example, the first gate would fail if the most honest way to describe the product was “levered / concentrated market beta with model complexity attached.” Beta describes how much a portfolio moves when the overall market moves. In simple terms, if my product was just the stock market with extra risk and a complicated model on top, it would fail. This matters because taking on extra risk is cheap and easy. Anyone can make a low-cost index fund move more sharply with the market by using a little leverage. A product that only does this would charge customers for something they can already get cheaply.

I also wrote down what the evaluator was not allowed to do. At the top, the instructions said, “Do not treat years invested … as evidence of a company.” A list called “Anti-patterns (do not do these)” included the instruction “Soften a STOP / JOB SEARCH verdict to protect sunk cost.” Sunk cost means the time and money already spent, which cannot be recovered no matter what you decide next. I was the person most likely to want the verdict softened.

There is a small irony in those instructions. Under the section about how to find customers, I had written that I would attract them by blogging about building investment products powered by AI. And here I am, blogging about it.

What Did the Memos Say?

About half an hour after I saved those documents in the repository, Claude Fable 5.1, running in Claude Code, evaluated the project using the index, the issues and pull requests linked from it, and its own web research into competitors and regulations. The next morning, Cursor’s agent, Composer, carried out the same evaluation independently.

Both reached the same conclusion: “personal only.” I should keep running the system with my own money but not turn it into a company. The first memo made this clear: do not start a company now, and do not raise money. It described the most honest sales pitch as “concentrated, high-turnover S&P exposure with ML attached, viable only inside an IRA.” This was very close to the failure condition I had written for the first gate. The memo also recommended that I start looking for a job immediately instead of waiting until I reached the two-year mark.

The phrase “viable only inside an IRA” needs some explanation. My system changes its stock holdings about once a month. That means it sells some stocks it has owned for less than a year. In a regular brokerage account in the US, profits from these short-term trades are taxed at the same rate as ordinary income each year. These taxes eat away much of the benefit of frequent trading. In a retirement account such as an IRA, trades are not taxed as they happen, so this tax cost goes away. That meant any product based on my system would have to be sold for use in retirement accounts.

I was heartbroken. I had spent 21 months on the project and felt that I was close to something big. But the conclusion was that the evidence I had was not enough to justify launching a financial product.

Can Two AI Models Be Wrong in the Same Way?

Yes. That was the first thing I learned when I began challenging their conclusions.

On September 20, I asked Claude Opus to review both memos. It found the same two mistakes in each. First, both memos described the strategy as riskier than the version I actually run. They called it leveraged or high-beta, even though its movements are roughly in line with the overall market.

The second mistake involved retirement accounts. Both memos said these accounts were naturally small because the government limits how much people can add to an IRA each year. That is true for new contributions, but it leaves out rollover IRAs. When people leave a job, they often move their entire 401(k), or workplace retirement savings, into an IRA. There is no yearly contribution limit on this kind of rollover. So the accounts into which the product would be sold could be much larger than the memos assumed. The corrections added to both memos explained this clearly.

Neither mistake changed the final decision. But they changed how I understood the fact that both models agreed. Two models that read the same records and receive the same instructions are not two fully independent opinions. They can share the same blind spots in the information they were given. As the IRA mistake showed, they can also share a wrong idea about the outside world, even when that idea is not in the records. Their agreement means something, but much less than it seems to mean when you first read it.

So I reviewed the memos the same way I had started reviewing my research: I asked a different model to check them, then continued reviewing them until no major problems remained.

How Many Ways Can You Argue With a Verdict?

I did not accept the decision right away. I spent days trying to find flaws in it, and whenever I thought of a reason the memos might be wrong, I investigated it. I did not start with a list of arguments against the decision. Each idea came to me as I tried to accept that there would not be a company.

I also wrote down every argument in a section of the memo called “Arguments tried and retired.” At the top, I added this rule: “A new argument should say which of these it is not.” In other words, before spending another afternoon exploring an idea, I had to check that I had not already tested and rejected the same argument.

By October 10, there were 25 entries. Here are a few examples to show what this process looked like:

1. Sell to financial advisers instead of individual investors. This would reduce the cost of finding customers one by one. But professional buyers look for exactly the kind of long, independently checked performance record that I did not have. So this approach would make the requirements harder to meet, not easier.
2. Charge only a performance fee, meaning a share of the profits instead of a fixed fee. US rules generally allow performance fees only for clients who are already wealthy. That would rule out the very customers that all the other ideas were aimed at.
3. Put the strategy inside an actively managed ETF, or exchange-traded fund, which trades on the stock market like a single stock, to solve the tax problem. ETFs can avoid passing capital gains on to investors through a process called in-kind redemptions. When a large investor takes money out, the fund gives them shares of the stocks it owns instead of selling those stocks for cash. This means the gains are never realized inside the fund. The approach works, and established funds use it to avoid distributing capital gains. But running this kind of fund costs hundreds of thousands of dollars a year, so it only makes sense when the fund is already large.
4. Registering as an investment adviser turned out to be much cheaper than the memos had assumed if I completed the paperwork myself. That was correct, so I kept it as a correction. But cheaper registration does not provide any evidence that the strategy can beat an index fund.
5. Instead of waiting years for evidence, change the portfolio so that the evidence appears sooner. Holding more stocks would reduce how much the portfolio moves compared with the market. With less random variation, a real advantage might become visible in fewer years. The calculations made sense, but the data did not support the idea. The model’s best picks stood out only at the very top of its list. Adding more stocks weakened the advantage just as quickly as it reduced the random variation.

I kept a few arguments in part because they corrected mistakes in the memos. A few questions are still open, including two that only a securities lawyer can answer. None of these arguments changed the decision.

What Happened on the Day the Verdict Almost Flipped?

On September 22, I carried out a review of the evidence that supported my side four times in a row. Each review found something that needed correcting, and each correction made the project look better than the memos had suggested. One result confirmed that my trading costs, meaning the money lost between deciding to make a trade and actually completing it through the broker, were almost zero. I had spent months building the tools needed to measure this.

I opened a pull request that day to change the decision from “personal only” to “90-day discovery,” meaning that I would test whether real customers were willing to pay. Then I checked the most important of the four results against a second, much longer set of price records. The first set covered about 8 years, while the second covered about 24 years. The result did not hold up over the 24-year period. It was the second encouraging result in three days that failed when tested against a longer history. Near the top of the pull request that was eventually merged, I wrote, “This PR was originally drafted as a revision to 90-DAY DISCOVERY on the eight-year evidence. Checking the 24-year pool reversed it.”

If you have not spent much time studying stock-market data, this is an important pattern to understand because it happens far more often than people might expect. Over a few years, it is surprisingly easy to find a stock-picking method that beats the market. There are so many possible methods to try that some will appear to work just by chance. Over longer periods, most of these apparent advantages become smaller or disappear. A well-known 2016 study by David McLean and Jeffrey Pontiff, published in the Journal of Finance, examined 97 stock-market patterns that academic papers had identified as ways to predict returns. After those papers were published, the patterns earned an average of 58% less than they had in the original research.

It was deeply disappointing. As an entrepreneur, it is exciting to feel that you have discovered something important. When you learn that it may not be important after all, it is hard to come down from that excitement. Entrepreneurs experience both very high highs and very low lows.

So, What Actually Decides It?

Looking back at all 25 arguments, the pattern is clear. Changing the product, its price, its business partners, or the way it is sold can change the business case. But only time, real money, and results that cannot be changed after the fact can answer the question about whether the strategy truly works. Nothing I could build or buy could make that time pass faster.

The reason it takes so long comes down to the difference between a real signal and random noise. If a strategy’s average advantage is small compared with how much its results change from year to year, a few good years can look just like a run of good luck. Given the size of the possible advantage in my system and how much a concentrated portfolio, which holds a relatively small number of stocks, can move compared with the market, proving that skill rather than luck is responsible would take decades of live trading, not a few quarters. This is the same problem the momentum index in my last post faced, even after I rebuilt its history using 24 years of data.

My system has now run continuously with real money for almost two years. Based on the evidence so far, its results cannot be statistically distinguished from those of cheaper alternatives. In other words, the difference is too small compared with the random variation to rule out luck. Even famous stock pickers can spend long periods doing worse than the market index. The evidence I had was simply not strong enough to justify a non-professional launching a financial product.

While I was working through these arguments, after the first memo had been written, I read Burton Malkiel’s A Random Walk Down Wall Street. I had read many books about finance, but never this one. Malkiel’s argument, first published in 1973 and updated in many later editions, is that markets take in new information so quickly that consistently beating them is extremely difficult. Because of this, an investor who buys and holds a low-cost, diversified index fund is likely to do better than most portfolios carefully selected by professionals. I already knew that broad index funds are very hard to beat, but seeing the argument supported by so much research was extremely helpful. It felt as if I had spent almost two years proving the book’s main point.

Was It 21 Months Wasted?

Some people may read this story as 21 months spent proving that something does not work. I do not regret the time. I learned a great deal about the ideas and methods behind quantitative trading, including how to find and store reliable data, how to build machine-learning systems that are both fast and dependable, and how to make trades quickly and reliably when real money is at risk. Still, I wish the outcome had been different.

It is always better to test an idea early, so you do not spend more time than necessary chasing something that will not work. Setting a clear stopping rule and defining the evaluation gates were my attempts to do that. But I also do not think I could have been confident in either decision without doing the work. Sometimes, making mistakes along the way and understanding them afterward is the only way to know the truth.

What Happens Now?

I have stopped working toward turning the project into a company. The system continues to run in my own accounts, and I still do research on the side while doing other things. The question of whether the strategy really works remains open. If I ever get an answer, it will come from time and evidence, not from another memo.

Next, I am returning to leading engineering teams. I will take with me the habits that helped me make this decision: writing down the rules before looking at the results, holding my own ideas to the same standards of evidence I would expect from anyone else, and treating a stopping rule I set in advance as a decision to follow rather than something to renegotiate.

Thanks for reading.

---

This describes software I run in my own accounts. It is not investment advice, and I do not offer investment services.