If It Cannot Guarantee the Future, What Is Quant Research For?

The 2023 envelope is open.

Hoppy has added both periods to the summary and checked everything once more: same companies, same groups, same return calculation. Nothing was quietly swapped.

But he does not close the computer.

“First we proposed a hypothesis and fixed the rules. After studying the first two years, we even saved 2023 for a separate check.”

Hoppy counts the steps on his fingers.

“And we still cannot say that 8 works. We certainly cannot say it will make money in the future.”

He turns to Dr. Hop.

“So what did all that work actually give us?”

Dr. Hop smiles.

“You have finally reached the most important question in this study.”

First, we have not built a strategy

We have been studying whether a stock ID contains the character 8.

Strictly speaking, this is not an executable trading strategy.

We never defined:

• When to buy;
• When to sell;
• How long to hold;
• How much to allocate to each company;
• How to account for trading costs;
• What to do after a large loss.

We did something earlier in the process: split companies into two groups and ask whether one feature appeared alongside a return difference.

Even if the difference had been stable, it would first be a clue worth studying. Only after defining clear buying, selling, and holding rules could we begin to ask how a strategy performs.

So “does 8 provide a useful way to separate these groups?” and “does a strategy work?” were never the same question.

Three questions that sound alike but are not

Dr. Hop writes three questions on the board:

1. Why would it affect returns?
2. Can it help us predict?
3. Can it guarantee future profit?

“Aren’t they all asking whether it is useful?” says Hoppy.

“They sound similar,” says Dr. Hop, “but a study cannot treat them as one question.”

Question one: why did it happen?

This question looks for a cause.

Suppose we believe growing demand in an industry can improve company earnings, which may then change market expectations and share prices. A researcher would keep checking whether that proposed path makes sense.

For the digit 8, we already know the answer. The fictional IDs were generated randomly and independently of returns. The character 8 has no path through which it could push prices.

Question two: can it provide repeatable predictive information?

This question does not require us to explain every cause immediately. It first asks whether a pattern defined in advance continues to separate outcomes in data that played no part in the earlier research.

Sometimes we find a fairly stable pattern before we fully understand why it appears.

Dark clouds can help us judge whether rain is likely, but the clouds are not reaching down and pressing a button that releases each drop. A predictive clue and a complete causal explanation are related, but they are not the same thing.

When a convincing explanation is missing, we should be more cautious and ask for broader, more stable evidence. But an incomplete explanation does not make all predictive research worthless.

Question three: can it guarantee the next profit?

Quantitative research cannot do this.

Markets change. Participants learn. A pattern that once existed may weaken, and new policies or events can alter the environment. Even a pattern that worked many times can fail next time.

Doing quantitative research does not make all three questions resolve themselves together.

In practice, quantitative methods mainly help us build the second kind of evidence: can a pattern continue to provide useful information in data that was not used to shape it? Data may also offer clues for the first question, but it does not automatically prove a cause. The third question was never part of the promise.

Quantitative research does not stamp ideas “approved”

Consider a hypothesis that sounds more connected to the market: after a certain signal appears during a sustained decline, prices may be more likely to rebound over the following period.

Before seeing the results, the researcher defines the signal, observation window, and comparison method. The signal shows an advantage in the study period, and a similar direction and size appear in a true holdout period that played no part in the earlier work.

What should we say?

We should not say:

This pattern has been proven forever. It will always make money.

But we should not say this either:

The future can never be guaranteed, so these results tell us nothing.

A better statement is:

The pattern showed a reasonably consistent advantage in two historical periods separated in advance. The evidence is stronger than it was after the study period alone, so the idea is worth another round of checks.

That is what a holdout check can add.

If the pattern does not repeat, our confidence falls. If it repeats fairly consistently, our confidence rises. If it produces a mixed result like the digit 8, we preserve that disagreement instead of forcing a winner.

Quantitative research does not teleport us from “we do not know” to “absolutely true.”

It helps our confidence change with the evidence.

A clue can stand on different levels of evidence

Dr. Hop draws a staircase with no trophy at the top.

The first step is an intuition:

I feel that this phenomenon might be related to returns.

The second step turns the intuition into fixed rules and finds a difference during the study period.

The third keeps those rules unchanged and lets data that played no part in the earlier analysis check them again.

Farther up, a researcher might ask:

• Does the result remain across other periods?
• Does it survive in another reasonable sample, or depend on a few unusual observations?
• Is the advantage large enough to cover trading costs?
• How much risk sits behind the return?
• Does the pattern survive when the market environment changes?

As we climb, the evidence usually becomes richer. No step suddenly turns into a certificate of future victory.

Not every idea deserves the entire climb, either.

If strong opposing evidence appears early—or if we already know that the proposed relationship is random—stopping can be the sensible result.

What if both the study and holdout periods show an advantage?

This is where people often run toward one of two extremes.

The first says:

It won in both periods. It is proven.

The second says:

The future is still uncertain, so both results are meaningless.

Both claims go too far.

If the rules were fixed before the results were seen, the holdout data truly played no part in earlier adjustments, and the advantage remained reasonably stable in both periods, then the idea has gained stronger predictive evidence.

That is not “nothing.”

It means the idea may deserve more time: more environments, risk, costs, and the work needed to turn it into executable rules.

The word “works” also needs a home. We should ask: in which market, over which period, under which rules, and by which comparison did it work? If the idea later becomes a trading strategy, we must also ask whether it remains useful after costs and risk.

The clearer those conditions are, the closer “works” comes to a research claim we can actually check.

Even then, we are responsible only for the evidence we have seen. Nobody can sign tomorrow’s market result in advance.

Where does the digit 8 stand now?

Place our experiment on the staircase:

• We proposed a clear hypothesis;
• We fixed the groups, periods, and measures before seeing the results;
• Some descriptive differences appeared in the study period;
• The holdout period produced a mixed result—some differences repeated and one did not;
• We know the fictional IDs were generated randomly and independently of returns.

The sensible choice is not to promote 8 into a stock-selection rule or try a series of new definitions until it wins.

We can stop treating the character 8 as a return clue.

“So the study failed?” asks Hoppy.

“This hypothesis did not earn a reason to move forward,” says Dr. Hop. “But the research did not fail.”

It helped us discover early that this path did not deserve more time. That can prevent us from investing additional effort—or carrying a coincidence into a real decision.

Ruling out an unhelpful idea is itself a research result.

In one sentence:

The digit 8 can leave the classroom. The research process stays.

A study does not end with only “success” or “failure”

At the end of one research round, we usually have more than two choices:

• Stop: the evidence is clearly insufficient, or we already know the key relationship came from randomness;
• Keep observing: there is some support, but it is not stable enough to justify a broader claim;
• Form a new hypothesis: the original question was too vague or its logic needs to change, so the new version becomes a new study instead of a quiet rewrite of the old one;
• Move into a fuller backtest: a pattern remained reasonably stable in data that played no part in shaping it, so it may deserve checks of timing, risk, and execution rules.

There is no universal choice for every study.

AI can help organize the evidence and spot omissions. A human still has to confirm which path fits the research goal and the limits of the evidence.

You can give the current research materials to your AI and ask it to help with one final review:

Decide the research direction with Codex

Copy

Read the experiment agreement, original study summary, and holdout summary for the current fictional stock-ID research. Do not recalculate anything, change the rules, or add a new analysis.

In plain language, answer each question separately:

1. Which historical facts have the current data established?
2. Do the results provide stable predictive evidence?
3. Can they show that the character 8 caused the return differences?
4. Can they guarantee future performance?
5. Given the known fact that the fictional IDs were generated randomly and independently of returns, should this clue stop, remain under observation, become a newly stated hypothesis, or move into a fuller backtest?

Explain your reasoning. Clearly distinguish “stop studying the digit 8” from “reject quantitative research as a method.” Report back to me first; do not create a new program or modify any files.

The point is not to let the AI press the button for you.

The point is to check whether its reasoning matches the evidence already on the table.

Once you have checked that AI’s recommendation and reasoning stay within the evidence, make the final call yourself and ask it to save the decision:

Save the final decision

Copy

I confirm this research direction: stop treating the digit 8 as a return clue, while keeping the research process we used.

Append this final decision and its reasoning to the end of the existing research summary. Explain that the original and holdout periods produced a mixed result, that the fictional IDs were generated randomly and independently of returns, and that stopping this clue does not mean rejecting quantitative research. Do not recalculate anything, change the existing results, or create a new analysis. When finished, tell me which file you changed and where the new passage appears.

What did we actually take away?

After that step, Hoppy looks at the final line now saved in the research summary:

Stop treating the digit 8 as a return clue. Keep the research process.

Then he finally closes the computer.

“I think I get it now.”

“Quantitative research does not have to find a profitable answer every time.”

“It makes me state the question clearly, then lets the data tell me whether to believe the idea a little more or a little less.”

Dr. Hop nods.

“Sometimes the most valuable result is learning early that you should stop.”

Starting from one character that never changes over time, we have completed our first full research round:

Propose a hypothesis
→ Fix the rules
→ Test with data
→ Read the results
→ Question the results
→ Check again with data not used in the earlier analysis
→ Write a provisional conclusion
→ Decide what comes next

Our next question will take a step forward.

Instead of studying whether an ID contains a character, we will examine a market signal that can appear on a particular trading day and then disappear.

That creates new problems. On which day did the signal appear? How far ahead should we look? What would count as better performance?

We already have one tool we can reuse.

It is not a particular program. It is the habit of making evidence speak when a new idea appears—and knowing where that evidence must stop.