automated trading - How do you distinguish "significant" moves from noise?

How do you distinguish between losses that are within the normal range for day-to-day shifts and situations with a real potential for loss? The specific application I have in mind is pattern recognition-based algorithmic trading.

Answer

Measuring expected shortfall (also known as conditional value-at-risk) answers the simpler question of "what is my average expected loss at the i-th quantile?" given the empirical distribution of returns. A variation is value-at-risk which measures the loss at the i-th quantile.

Arguably you could leave at this this and you have your answer.

You probably want a more robust estimate of your risk.

In this case you can use the bootstrap methodology. When you compute confidence intervals from a random sample, the statistics are themselves random variables. Indeed, your sample of returns itself is one of many possible samples. Each possible sample gives a possible value of Value-at-Risk, mean returns, etc. Although we observe one set of statistics using all your data it was selected at random from many values so it is therefore a random variable.

Enough with the theory - the procedure is not very difficult.

1. 
   

   Define some statistic(s) of interest. Let's say it is value-at-risk.

   
   


2. 
   

   Create a re-sample. You do this by sampling from your distribution of returns WITH replacement. Sample 'n' times where n is the number of observations. (You can actually sample 2n, 3n... if you'd like)

   
   



3. 
   

   Calculate the statistic of interest on the re-sample.

   
   


4. 
   

   Repeat steps #2 and #3 a couple thousand times.

   
   


5. 
   

   Since the re-samples are independent of each other (b/c we re-sampled with replacement), the statistic you calculate in #4 is itself a random variable. You can now construct a confidence interval for the statistic of interest by measuring the standard error of the estimate and the t-statistic.

   
   

The bootstrap let's you answer your qualification "within a normal range". The bootstrap recognizes that the empirical distribution is itself a sample from an unknown population.

You can read more about the bootstrap here.