Testing instances and statistical error

[foxtrade]

I'm writing on this particular theme because i'm preparing an exam about statistic. It will be useful for me to clarify some concepts, and i hope it will be also useful for some of you as traders. Maybe, you can write your opinion and your experiences about it.
I'll do my best to write in correct english. I dont know some technical terms, i'm sorry for it.

What is a good number of tests we should make before saying that a trading strategy works?

When we say that a certain formation 'is working', we are going to predict the percentages that, in the reality, an event will go in a certain way. Until we cannot test an infinite number of cases, we will do a limitate number of tests in the early past. It's not so bad, for the simple reason that markets changes.

But, can we really expect, by testing for example 20 cases, that a particular formation has a 70% probability of winning?

The more cases we test, the better is. The statistic Error Margin defines the range of error that our tests can produce. For example: we test 25 cases of a specific formation and we have 70% winning trades? Well, the difference from the reality and our limitated number of tests may be consistent: in this specific example, with 25 tests the real probability of winning are from 70+19% to 70-19%. It means the real probability of success of our next trades are from 89% to 51%. Warning, it counts for all the next trades, the probability dont change ( in the same market conditions ) from one trade to another. I can find with experience and hundreds of trades that the studied formation works for the 83%, 65%, 55% of the times, or any percentage from 89% and 51%. So, if i see that 70% of times it works, on 25 cases, dont get surprised if you will obtain differents results.

The more cases, the more the Error Margin is little. There should be a little more to say, but it's complicated and it's not so important. If you want to know the error margin on a certain number of cases ( 10 cases, 100 cases, 1000 etc ) there is a specific statistical table for this ( but sorry, i dont know its name, just try to google error margin ).

example: 10 cases, error margin 30%
50 cases, error margin 14%
200 cases, error margin 7%
1000 cases, error margin less than 3%

Maybe, this will be useful; i hope for some interesting comments

[Joe Ross]

foxtrade, I don't know how many people will answer this to help you do your homework, but, I use 40 test cases to get what I consider a statistically valid number.

[foxtrade]

eheheh unluckily i have many doubts i'll talk about it at the exam but.. the more you know.. the better is... I was just thinking on it; there should be a relation between the theory and the testing phase, or maybe i'm missing something.. if it is, i'd like to know

[Soron]

Foxtrade:

Purely statisticly this can be correct. However you have to acknoledge that the statistical power of prediction aswell as error is OK until something changes in the system or in the inviroment. Meaning, you can have 1000 test results, but if for example something change in the markets the result is meaningless.
That goes also for other areas. Let say that I can predict bacteria growth under such and such conditions. I can have 1000 tests and I can be very accurate in prediction, however if the temperature change for few degrees the result would be quite different.

Also with statistics everything is OK while you are in the correct range. For example if I test how a plane fly at 10000m, 20000, 40000m and 50000m. Doesn't mean that I can extrapolate the data how it would fly to the moon

I am not trying to be smart here, just so that you are aware of those things. Statistisc can be a great and powerfull tool, just that you know its limitations. If something changes you have to rerun and recalc things. Or as mr. Joe say if the markets change you have to change aswell. Otherwise you have a problem.

Take care.
Soron