The Walk Forward Performance Metric Explorer (WFPME) reads all files generated by the PWFO and searches each PWFO file for a Top N-Metric-PF-LR filter that generates the best average out-of-sample performance. Here we define the “Top N” as the TOP 5 or Top 10 or Top user selectable number N. The Top N Metric-PF-LR filter first eliminates all cases in the PWFO test files that do not meet certain Profit Factor (PF), Losers in a row (LR) criteria. Then from the cases(rows) that are left meeting the PF-LR criteria, the WFPME chooses the rows that contain then Top N values of a PWFO Performance Metric. The PF, LR and Top N criteria ranges are user selectable generating many filter searches in one run. The WFPME is a stand alone exe program that is super fast and automatically displays it's extensive K-fold cross-validation statistical summary and results in Excel. In addition, using modern "Bootstrap" techniques, the WFPME calculates the probability of whether or not each filter's results were due to chance.

Click here to see the WFPME Input Form and a description of the inputs

Data Mining and Curve Fitting. Data mining or curve fitting a price series will always produce the best performance results. If you look hard enough using optimization you will always find patterns in the price data. But they are not real. Why use the PWFO walk forward technique? Why not just perform a TradeStation optimization on the whole price series and choose the input parameters that give the best total net profits or profit factor? Whenever you run a optimization (combinatorial search} over many different combinations of input parameters on noisy data on a fixed number of prices, no matter how many, the best performance parameters found are guaranteed to be due to "curve fitting" the noise and signal. When we run, say, 5000 different input parameter combinations , the best performance parameters will be from those system input variables that are able to produce profits from the price pattern and the random spurious price movements While the price patterns, if there, will repeat, the same spurious price movements will not. If the spurious movements that were captured by a certain set of input parameters were a large part of the total net profits, then choosing these input parameters will produce losses when traded on future data. These losses occur because the spurious movements will not be repeated in the same way. This is why system combinatorial searches with no out-of-sample testing cause loses when traded in real time from something that looked great in the test section.

In order to gain confidence that our system input parameter selection procedure on test data will produce profits "on average" in the future we must perform the walk forward out-of-sample analysis many times. Why not just do the analysis once? Well just as in poker, where there is considerable vagaries in hand to hand luck, walk forward out-of-sample analysis give considerable vagaries in week to week out-of-sample profit "luck". That is, by pure chance we may have chosen some input parameters that did well in the test section data and the out-of-sample section data. In order to minimize this type of "luck", statistically, we must repeat the walk forward out-of-sample (oos) analysis over many test/oos sections and take the average of our weekly results over all out-of-sample sections (we need at least 30 oos sections for statistically significant results). This average gives us an expected weekly return and a standard deviation of weekly returns which allows us to statistically estimate the expected equity and it's range for N weeks in the future.

Walk Forward Performance Metric Explorer (WFPME) Description.As stated above, curve fitting a price series will always produce optimization cases with the best test performance results. If we eliminate the optimization cases with the best performance results we are sure to eliminate many of the curve fitting system input parameters that fitted the past spurious noise movements of the price series. Here is an example of one of the filters that the WFPME can produce. Very few systems can sustain Profit Factors (PF) above 3 over time. Therefore if we eliminate from the test sample optimization results, all cases that have profit factors greater than 3, we will likely eliminate many of the cases that are due to curve fitting the noise. Also, we like to filter for only those cases in the test section had a positive net profit. In addition, because in real time it is difficult to sustain more than five losses in a row and still keep trading, we will eliminate all those test sample cases that have more than 5 losses in a row. After using a PF and LR filter as above there can still be 100’s of rows left in the PWFO file. The PWFO generates the metric R2 which is the Trade Equity Regression Trend Line Coefficient of Correlation in the test section. As an example, let us choose the rows that contained the Top 10 R2 values from the rows that are left from the PF-LR screen. That is we sort R2 from high to low, eliminate the rows that have PF>3 and LR>5 and then choose the Top 10 Rows of whatever is left. This particular filter will now leave 10 cases or rows in the PWFO file that satisfy these filter conditions. We call this filter t10R2|p<30|5 where t10R2 means the Top 10 R2 Rows left after the PF-LR filter. p<30 means PF<=3.0, "|5" means LR<=5. Suppose for this filter, within the 10 PWFO rows that are left, we want the row that has the maximum PWFO mtnp metric in the test section. This would produce a filter named t10R2|p<30|5-mtnp. This filter leaves only one row in the PWFO test section with it’s associated out-of-sample net profit in the out-of-sample section. This particular t10R2|p<30|5-mtnp is then calculated for each of the PWFO files that are run by the WFPME and the average out-of-sample performance, plus many other important statistics for this filter is summarized along with the hundreds of other filter combinations that are constructed in a similar manner. This filter summary is sorted by total out-of-sample net profits after costs for all the various filters and written to a comma delimited file by the WFPME. When The WFPME run is done just click on the Excel Icon next to the Run button and the WFPME output file will appear in Excel or your spreadsheet

Bootstrap Probability of Filter Results. Using modern "Bootstrap" techniques, the WFPME calculates the probability of obtaining each TopN-Metric-PF-LR-NT filter's total out-of-sample net profits by chance. Here is how the bootstrap technique is applied. Suppose, for this example, the WFPME calculates the total out-of-sample net profits(tONet) of 700 different TopN-Metric-PF-LR-NT filters. A mirror filter is created for each of the 700 WFPME filters. However, instead of picking an out-sample net profit(OSNP) from a filtered row, the mirror filter picks a random row's OSNP in each PWFO file. Each of the 700 mirror filters will choose a random row's OSNP of their own in each of the PWFO files. Thus if there are 50 PWFO files, each of the 700 mirror filters will pick a random row's OSNP in each of the 50 PWFO files. At the end, each mirror filter will have 50 random OSNP's picked from the rows of the 50 PWFO files. The sum of the 50 random OSNP picks for each mirror filter will generate a random total out-of-sample net profit(tONet) for each of the 700 mirror filters. The average and standard deviation of the 700 mirror filter's random tONets will allow us to calculate the chance probability of each WFPME filter's tONet. Thus given the mirror filter's bootstrap random tONet average and standard deviation, we can calculate the probability of obtaining the TopN-Metric-PF-LR-NT filter's tONet by pure chance alone.

Below is a snippet of the output from a WFPME run sorted by the total out-of-sample net profit statistic (tONet). This example shows the partial output file from a WFPME run on the PWFO files generated by the Nth Order Fixed Memory Polynomial Acceleration Strategy that was run on 1 contract of the Japanese Yen 1 minute bar futures for the 52 weeks of 04/01/05 to 03/24/06.

---

---

The WFPME Columns are defined as follows

• Row 1 is the PWFO Stub, File Start Date, File End Date and Number of weeks.


• Filter = The filter that was run. For example row 3, t10mwb|p<25|3 is: PF<=2.5, LR<=3, t10mwb=Top 10 mwb(Median Winning Bars) values in the rows that are left from the PF-LR filter. The row 7 filter, t10eq10|x|a is: any PF, any LR, t10mwb=Top 10 eq10(Projected Equity 10 Trades In Future Using Equity Curve Least Squares 2nd Order Polynomial Line) values in the rows that are left from the PF-LR filter. In this case just the rows that have Top 10 eq10 values since there is no PF-LR filter.


• Metric = The PWFO performance metric (defined on the PWFO page). For this WFPME filter, t10mwb|pf<25|3 Metric=b0 (Equity curve slope), this PWFO filter produced the following average 52 week statistics on this line.


• tOnp = Total out-of-sample(oos) net profit for these 52 weeks.


• mOsp = median oos net profit for the 52 weeks


• aOsp = Average oss net profit for the 52 weeks


• aanp = Average oos profit per trade


• aOnT = Average number of oos trades per week


• B0 = The 52 week trend of the out-of-sample weekly profits


• %P = The percentage of oos weeks that were profitable


• t = The student t statistic for the 52 weekly oos profits. The higher the t statistic the higher the probability that this result was not due to pure chance


• std = The standard deviation of the 52 weekly oos profits


• llw = The largest losing oos week


• eqDD = The oos equity drawdown


• lr = The largest number of losing oos weeks in a row


• # = The number of weeks this filter produced a weekly result. Note for some weeks there can be no strategy inputs that satisfy a given filter's criteria.


• b00 = The straight line trend of the oos equity curve in $/week.


• Blw = The maximum number of weeks the oos equity curve failed to make a new high.


• BE = Break even weeks. Assuming the average and standard deviation are from a normal distribution, this is the number of weeks you would have to trade to have a 99% probability that your oos equity is above zero.


• ndd = The normalized equity drawdown = 100*eqDD/tONet


• tONet = Total out-of-sample net profit(tOnp) minus the total trade cost. tONet=tOnp - #*aOnT*Cost.


• Prob = The probability that the filter's tONet was due to pure chance. For example, for row 3 where tONet=9744 there is a 1 in 14749 (1/0.0000678) chance that the tONet obtained by this filter was due to chance.


• Notice: Past performance is no guarantee of future results

The Walk Forward Performance Metric Explorer comes with a detailed manual explaining:

• How to setup, install and run the Walk Forward Performance Metric Explorer. The Walk Forward Performance Metric Explorer is a stand alone exe file that can be executed directly from your desktop icon or from the Windows Start Program menu.


• How to use the WFPME with your PWFO files.


• An explanation of each of the performance statistics columns.


• How to select the best WFPME filter to use on real time runs (Past performance is no guarantee of future results).

The WFPME Input Finder Excel Add-In and WFPME Strategy Input finder EXE.

Supplied with the WFPME is an Excel Add-In Filter and a separate WFPME Input Finder EXE. For the Excel Add-In, as shown below, you just click on a special Add-In icon on the Excel Toolbar and a popup window displays. Fill in the WFPME parameters and click on the Run button. The Excel Add-In macro will filter any number of PWFO files loaded in Excel, and will display the PWFO Excel file row that satisfies the typed in criteria. For the WFPME Input finder, as shown below, fill in the WFPME parameters click on Run and the Strategy Inputs are displayed as shown below.

   

The The Walk Forward Performance Metric Explorer package consisting of Manual, WFPME EXE file, WFPME Excel Add-In, and WFPME Input finder EXE is being offered, for $395 plus shipping. Please note that the WFPME will only read files generated by the PWFO product. In addition, the WFPME has a "Key Licence" that only allows it to be installed on two computers. With special pricing, the Walk Forward Performance Metirc Explorer package can be combined with the  PWFO program and this combo is being offered for $895, a savings of $95, if purchased at the same time.

How To Order
To order online click Order Online. To order via Fax or mail using a Visa or Master Card please fill out the order form on the Order Form page and Fax it to the telephone number on the order form or mail it to the address on the order form. If you would like to talk to me about the product, please call me at (312) 280-1687 M-F 12pm to 5pm CST. All E-mail queries can be sent to info@meyersanalytics.com.

Thank you for your Interest....Dennis Meyers

---