Articles

Those who perform optimization for the first time can find the description of necessary procedures in the Help menu of the MetaTrader 4 Client Terminal: <Help - Help Topics F1 - Auto Trading - Expert Optimization - Setup>. Or they can read the article named Testing of Expert Advisors in the MetaTrader 4 Client Terminal: An Outward Glance.

Then let's check variables to be optimized as shown in the screenshot below.

        datetime SetHour    = 0;  // Optimization starting hour; 
        datetime SetMinute  = 1;  // Optimization starting minute.

- set the amount of days for optimization (it must be the same as that for pre-optimization):

        int TestDay = 3;

- set the optimization end waiting time, in minutes, that we have calculated before, for example, 4 minutes:

        int TimeOut = 4;

- type in the Expert Advisor name:

        string NameMTS = "MACD Sample_1";  // EA's name

- type in the set-file name with the settings:

        // Set-file name with the settings
        string NameFileSet = "MACD Sample_1.set"; 

- type in the path to the folder containing the installed Terminal-Tester, for instance:

        // Path to the tester        
        string PuthTester = "D:\Program Files\Forex Best Trade Station";

- set the priority for filtering:

        // Sorting by Maximal profit
        int Gross_Profit = 1;                      
        // Sorting by Maximal profit factor        
        int Profit_Factor = 2;     
        // Sorting by Maximal expected payoff
        int Expected_Payoff = 3;

- write variable names for optimization:

        string Per1 = "FastEMA";
        string Per2 = "SlowEMA";
        string Per3 = "SignalSMA";
        string Per4 = "";

- copy attached file auto_optimization.mqh to the 'include' folder';
- include the library file in the Expert Advisor:

//--- Including the auto optimizer's library
#include <auto_optimization.mqh>

- it is only remains to copy the code below to the beginning of the start() function of your Expert Advisor. MACD Sample_1.mq4 contains it already.

// Not to be launched at testing and optimizing   
  if(!IsTesting() && !IsOptimization())
    {                
      // Compare the current hour with that preset for launching
      if(TimeHour(TimeLocal()) == SetHour)
        {
          // Protection against restarting
          if(!StartTest)
            {
              // Compare the minute range to the minute
              // preset for launching
              if(TimeMinute(TimeLocal()) > SetMinute - 1)
                {     
                  // the range is necessary, in case that for some reasons 
                  // no new tick is available for a long time
                  if(TimeMinute(TimeLocal()) < SetMinute + 1)
                    {  
                      // Flag of tester launching
                      StartTest    = true;
                      TimeStart    = TimeLocal();
                      Tester(TestDay, NameMTS, NameFileSet, PuthTester, 
                             TimeOut, Gross_Profit, Profit_Factor, 
                             Expected_Payoff, Per1, Per2, Per3, Per4);
                    
                    }
                }
            }
        }
    }
    
                        
   FastEMA      = GlobalVariableGet(Per1);
   SlowEMA      = GlobalVariableGet(Per2);
   SignalSMA    = GlobalVariableGet(Per3);
   TrailingStop = GlobalVariableGet(Per4);  
// If the tester launching is flagged  
  if(StartTest)
    {                                        
      // If more time has elapsed the launching than it was set 
      // to be the test waiting time
      if(TimeLocal() - TimeStart > TimeOut*60)
        {            
          // Zeroize the flag
          StartTest = false;                              
        }
    }

That's all. After the automated optimizer has been recompiled, it can be launched, but it may be used for only the same symbol and timeframe, for which the pre-optimization was performed. In our case, it is EURUSD on H1. To check the auto optimizer, you can insert the code given below in the int init () function, then the auto optimizer will be launched at the Expert Advisor's start.

Tester(TestDay,NameMTS,NameFileSet,PuthTester,TimeOut, Gross_Profit,Profit_Factor, 
       Expected_Payoff, Per1,Per2,Per3,Per4);

How Automated Optimizer Works

The automated optimizer works on the basis of using Terminal-Tester to optimize parameters of the Expert Advisor attached to the chart in Terminal. For this, the program sends to the Terminal-Tester a file containing optimization parameters (optimise.ini) and launches the Terminal Tester in the optimization mode. Then it copies the obtained results of "FileReport........htm" back to the Terminal and filters the best values from the obtained results.

 // Path to the terminal 
string PuthTerminal = TerminalPath() + "\experts\files";
// Name of the ini file for the tester
string FileOptim    = "optimise.ini";
string FileOptim1   = "\optimise.ini";                                  
// Calculation of the starting date
datetime DayStart   = TimeLocal()-86400*TestDay;
// Optimization starting date
string DateStart    = TimeToStr(DayStart,TIME_DATE);
// Optimization ending date
string DateStop     = TimeToStr(TimeLocal(),TIME_DATE);
// Tester report file name
string FileReport   = "FileReport_" + Symbol() + "_" + DateStop + ".htm";
string FileReport1  = "\FileReport_" + Symbol() + "_" + DateStop + ".htm";
// Limitation for the minimum amount of trades per day
double MinTr        = TestDay - 2;
// Limitation for maximal amount of trades per day
double MaxTr        = (60 / Period()*TestDay) + 2;
// The amount of attempts to copy the report file
int    KvoPptk      = 10;
// The amount of lines for sorting
int    StepRes      = 12;

Then the parameters of the ini file are written in the string array:

// Prepare the ini file for optimization
ArrayOpttim[0] = ";optimise strategy tester";             
// Enable/Disable Expert Advisors
ArrayOpttim[1] = "ExpertsEnable = false";
// Name of the EA file
ArrayOpttim[2] = "TestExpert=" + NameMTS;
// Name of the file containing parameters
ArrayOpttim[3] = "TestExpertParameters=" + NameFileSet;
// Symbol
ArrayOpttim[4] = "TestSymbol=" + Symbol();
// Timeframe
ArrayOpttim[5] = "TestPeriod=" + Period();
// Modeling mode
ArrayOpttim[6] = "TestModel=" + 0;
// Recalculate
ArrayOpttim[7] = "TestRecalculate=false";
// Optimization
ArrayOpttim[8] = "TestOptimization=true";
// Use date
ArrayOpttim[9] = "TestDateEnable=true";
// From
ArrayOpttim[10] = "TestFromDate=" + DateStart;
// To
ArrayOpttim[11] = "TestToDate=" + DateStop;
// Report file name
ArrayOpttim[12] = "TestReport=" + FileReport;
// Rewrite the report file
ArrayOpttim[13] = "TestReplaceReport=true";
// Shut down the terminal upon completion
ArrayOpttim[14] = "TestShutdownTerminal=true";

The optimization parameters are recorded into the ini file from the array. You can also read about how to create an ini file in the MetaTrader 4 Client Terminal Help, see <Help - Help Topics F1 - Tools - Configuration at Startup>.

// Write data into the ini file                
// Find out about the array size
OptimArraySize = ArraySize(ArrayOpttim);
// Open a file to write
opttim = FileOpen(FileOptim, FILE_CSV|FILE_WRITE, 0x7F);
if(opttim > 0)
  {
    for(int i = 0; i < OptimArraySize; i++)
      {
        // from the array into the variable
        ini = ArrayOpttim[i];                                     
        // from the variable into the file
        FileWrite(opttim, ini);
      } 
    // close the file
    FileClose(opttim);
  }
else
  {
    Print("Failed writing data into the ini file. Error No ", 
          GetLastError());
    return(0);
  }

After the parameters have been recorded in the ini file, the shell32.dll included in the standard Windows delivery is connected and function ShellExecuteA is launched.

#import  "shell32.dll"               //Connect a dll (provided with Windows)       
  int ShellExecuteA(int hwnd,string Operation,string 
                    File,string Parameters,string Directory,int ShowCmd); 
#import

File containing the parameters will be sent to the Terminal Tester folder.

// copy the ini file into the tester folder 
copyini = ShellExecuteA(0,"Open","xcopy", "\"" + PuthTerminal + 
                        FileOptim1 + "\" \"" + PuthTester + "\" /y", 
                        "", 3);
// wait until the file is copied
Sleep(1200);                                                    
if(copyini < 0)
  {
    Print("Failed copying ini file");
    return(0);
  }

// Start Tester 
start = ShellExecuteA(0, "Open", "terminal.exe", FileOptim,
                      PuthTester, 3);
if(start < 0)
  {
    Print("Failed starting Tester");
    return(0);
  }
Comment("Wait until optimization is complete");
// wait until optimization is complete
Sleep(60000*TimeOut);

 for(Pptk = 0; Pptk < KvoPptk; Pptk++)
  {                    
    //Start a cycle attempting to compy the resport file
    Comment("Attempt # " + Pptk + " to copy the report file");
    ShellExecuteA(0, "Open", "xcopy", "\"" + PuthTester + FileReport1 + 
                  "\" \"" + PuthTerminal + "\" /y", "", 3);
    // wait until the file is copied
    Sleep(1200);
    // Try to open the report file
    file = FileOpen(FileReport, FILE_READ, 0x7F);
    if(file < 0)
      {
        // if it fails to open, wait some more and try again
        Sleep(60000);
      }                    
    else 
        break;             
  }
if(file < 0)
  {
    Print("Failed copying the report file");
    return(0);
  }

Then the data from the report file will be placed in the string array for further processing.

// Read from file into the array
// Cycle, until the file ends
while(FileIsEnding(file) == false)
  {                 
    // Read a string from the report file
    FileLine = FileReadString(file);
    // Find the necessary string and set the reference point there
    index = StringFind(FileLine, "title", 20);
    if(index > 0)
      {
        // Increase the array in size
        ArrayResize(ArrayStrg, NumStr + 1);
        // Record the strings from the file in the array
        ArrayStrg[NumStr] = FileLine;
        NumStr++;
      }
  }
// Close the file
FileClose(file);
// Delete the file in order not to produce too many copies
FileDelete(FileReport);
// Set the array size by the amount of data read from the file
ArrayResize(ArrayData, NumStr); strings
    
.

Then the necessary values are selected in the array:

  for(text = 0; text < NumStr; text++)
     {
      select = ArrayStrg[text]; 
    //-------------------------------------------------------------------------
    //   Reporting text processing (These are apples and oranges)              | 
    //-------------------------------------------------------------------------
    // Position Pass 
    ClStep=StringFind(select, "; \">",20)+4;
    // Find the end of position
    ClStepRazm = StringFind(select, "td>", ClStep);
    // Read the value
    CycleStep = StringSubstr(select, ClStep, ClStepRazm - ClStep);
    // Position Profit 
    // Find the beginning of the position
    GrProf = StringFind(select, "", ClStepRazm);
    // Find the end of position
    GrProfRazm = StringFind(select, "td>", GrProf);
    // Read value
    GrossProfit = StringSubstr(select, GrProf+4, GrProfRazm - (GrProf + 4));
    // Position Total Trades
    // Find the beginning of position
    TotTrad = StringFind(select, "", GrProfRazm);
    // Find the end of position
    TotTradRazm = StringFind(select, "td>", TotTrad);
    // Read the value
    TotalTrades = StringSubstr(select, TotTrad+4, TotTradRazm - 
                               (TotTrad + 4));
    // Position Profitability
    // Find the beginning of position
    ProfFact = StringFind(select, "", TotTradRazm);
    // Find the end of position
    ProfFactRazm = StringFind(select, "td>", ProfFact);
    // Read the value
    ProfitFactor = StringSubstr(select, ProfFact + 4, ProfFactRazm - 
                                (ProfFact + 4));
    // Position Expected Payoff 
    // Find the beginning of position
    ExpPay = StringFind(select, "", ProfFactRazm);
    // Find the dn of position
    ExpPayRazm=StringFind(select, "td>", ExpPay);
    // Read the value
    ExpectedPayoff = StringSubstr(select, ExpPay+4, ExpPayRazm - 
                                  (ExpPay + 4));
    // Variables' positions starting with the second one
    // Find the beginning of position
    P1 = StringFind(select, Per1, 20);
    // Find the end of position
    P1k = StringFind(select, ";", P1);
    // Read the Variable
    Perem1 = StringSubstr(select, P1 + StringLen(Per1) + 1, P1k - 
                          (P1 + 1 + StringLen(Per1)));
    // Find the beginning of position
    P2 = StringFind(select, Per2, 20);
    // Find the end of position
    P2k = StringFind(select, ";", P2); 
    // Read the Variable
    Perem2 = StringSubstr(select, P2 + StringLen(Per2) + 1, P2k - 
                          (P2 + 1 + StringLen(Per2)));
    // Find the beginning of position
    P3 = StringFind(select, Per3, 20);
    // Find the end of position
    P3k = StringFind(select, ";", P3);
    // Read the Variable
    Perem3 = StringSubstr(select, P3 + StringLen(Per3) + 1, P3k - 
                          (P3 + 1 + StringLen(Per3)));
    // Find the beginning of position
    P4 = StringFind(select, Per4, 20);
    // Find the end of position
    P4k = StringFind(select, ";", P4);
    // Read the Variable 
    Perem4 = StringSubstr(select, P4 + StringLen(Per4) + 1, P4k - 
                          (P4 + 1 + StringLen(Per4)));
    Comment("The obtained results are being analyzed");

After that, the obtained results, before they are transformed into number format, have been filtered by the minimal and the maximal amount of trades. Zero in the value of Profit_Factor is replaced with 1000 for correct sorting and subsequent sifting.

// Transform into number format
TotalTradesTransit = StrToDouble(TotalTrades);
GrossProfitTransit = StrToDouble(GrossProfit);
ExpectedPayoffTran = StrToDouble(ExpectedPayoff);
nodubl = true;
if(MinTr < TotalTradesTransit && MaxTr > TotalTradesTransit)
  {                    
    // Filter by the amount of trades
    PrFactDouble = StrToDouble(ProfitFactor);
    // Replace 0 in the Profit_Factor for proper analysis
    if(PrFactDouble == 0)
      {
        PrFactDouble = 1000;
      }

Then the values are checked for duplications and filtered out.

// Filter data having identical values
for(Dubl = 0; Dubl <= ResizeArayNew; Dubl++)
  {                    
    // Start the loop searching for identical values
    if(GrossProfitTransit == ArrayData[Dubl][1])
      {          
        // check whether the results for maximal profit coincide
        if(TotalTradesTransit == ArrayData[Dubl][2])
          {       
            // check whether the results for the amount of trades coincide
            if(PrFactDouble == ArrayData[Dubl][3])
              {          
                // check whether the results for Profit Factor coincide
                if(ExpectedPayoffTran == ArrayData[Dubl][4])
                  { 
                    // check whether the results for expected payoff coincide
                    nodubl=false;                              
                    // If everything coincides, flag it as coincided
                  }
              }
          }
      }
  }

Then the values prepared for sorting are written in the array.

// Write the filtered data in the array
if(nodubl)
  {
    ArrayData[text][1] = GrossProfitTransit;                                
    ArrayData[text][2] = TotalTradesTransit;
    ArrayData[text][3] = PrFactDouble;
    ArrayData[text][4] = ExpectedPayoffTran;
    ArrayData[text][5] = StrToDouble(Perem1);
    ArrayData[text][6] = StrToDouble(Perem2);
    ArrayData[text][7] = StrToDouble(Perem3);
    ArrayData[text][8] = StrToDouble(Perem4);
    ResizeArayNew++; 
  }

Then the data start to be analyzed in the preset priority order. The analysis is performed as follows:

• the loop is launched and, at the first pass, the values are sorted by the first parameter, for instance, by maximal profit; several best values are selected (12, by default), others are cut off;
• at the second pass, the values are sorted by the second parameter, for example, by Profit Factor; some best values are selected, a half after the first sorting, others are cut off;
• at the third pass, the last sorting is performed for the third parameter, for example, by expected payoff; a half of values is taken after the second sorting, others are cut off.

// Analyzer
// Analyzing principle is the sequential checking of maximal 
// values according to the predefined filtering priority   
ArrayResize(ArrayTrans, ResizeArayNew - 1);
for(int PrioStep = 1; PrioStep < 4; PrioStep++)
  {
    for(PrCycle = 0; PrCycle < ResizeArayNew; PrCycle++)
      {
        Sort     = ArrayData[PrCycle][0];
        Prior1   = ArrayData[PrCycle][1];             
        transit  = ArrayData[PrCycle][2];
        Prior2   = ArrayData[PrCycle][3];             
        Prior3   = ArrayData[PrCycle][4];             
        transit1 = ArrayData[PrCycle][5];
        transit2 = ArrayData[PrCycle][6];
        transit3 = ArrayData[PrCycle][7];
        transit4 = ArrayData[PrCycle][8]; 
           
        if(PrioStep == 1)
          {
            //Prepare for the 1st sorting
            if(Gross_Profit ==1)
              {
                SortTrans = Prior1;
              }
            if(Profit_Factor == 1)
              {
                SortTrans = Prior2;
              }
            if(Expected_Payoff == 1)
              {
                SortTrans = Prior3;
              }
          }
        if(PrioStep == 2)
          {
            // Restore
            if(Gross_Profit ==1)
              {
                Prior1 = Sort;
              }
            if(Profit_Factor == 1)
              {
                Prior2 = Sort;
              }
            if(Expected_Payoff == 1)
              {
                Prior3 = Sort;
              } 
            //Prepare for the 2nd sorting
            if(Gross_Profit == 2)
              {
                SortTrans = Prior1;
              }
            if(Profit_Factor == 2)
              {
                SortTrans = Prior2;
              }
            if(Expected_Payoff == 2)
              {
                SortTrans = Prior3;
              }
          }
        if(PrioStep == 3)
          {
            // Restore
            if(Gross_Profit == 2)
              {
                Prior1 = Sort;
              }
            if(Profit_Factor == 2)
              {
                Prior2 = Sort;
              }
            if(Expected_Payoff == 2)
              {
                Prior3 = Sort;
              } 
            //Prepare for the 3rd sorting
            if(Gross_Profit ==3)
              {
                SortTrans = Prior1;
              }
            if(Profit_Factor == 3)
              {
                SortTrans = Prior2;
              }
            if(Expected_Payoff == 3)
              {
                SortTrans = Prior3;
              }
          }          
        ArrayTrans[PrCycle][0] = SortTrans;
        ArrayTrans[PrCycle][1] = Prior1;
        ArrayTrans[PrCycle][2] = transit;
        ArrayTrans[PrCycle][3] = Prior2;
        ArrayTrans[PrCycle][4] = Prior3;
        ArrayTrans[PrCycle][5] = transit1;
        ArrayTrans[PrCycle][6] = transit2;
        ArrayTrans[PrCycle][7] = transit3;
        ArrayTrans[PrCycle][8] = transit4;
      }
    ArraySort(ArrayTrans,StepRes, 0, MODE_DESCEND); // Sort the array
    ArrayResize(ArrayTrans, StepRes);               // Cut off the unnecessary things
    for(int CopyAr = 0; CopyAr < StepRes; CopyAr++)
      {
        ArrayData[CopyAr][0] = ArrayTrans[CopyAr][0];
        ArrayData[CopyAr][1] = ArrayTrans[CopyAr][1];
        ArrayData[CopyAr][2] = ArrayTrans[CopyAr][2];
        ArrayData[CopyAr][3] = ArrayTrans[CopyAr][3];
        ArrayData[CopyAr][4] = ArrayTrans[CopyAr][4];             
        // Per1    Variable 1
        ArrayData[CopyAr][5] = ArrayTrans[CopyAr][5];              
        // Per2    Variable 2
        ArrayData[CopyAr][6] = ArrayTrans[CopyAr][6];
        // Per3    Variable 3
        ArrayData[CopyAr][7] = ArrayTrans[CopyAr][7];
        // Per4    Variable 4
        ArrayData[CopyAr][8] = ArrayTrans[CopyAr][8];
      }
   StepRes = StepRes / 2;
  }

The values filtered in this manner are written in global variables. The values from global variables will be substituted in the EA.

     // Write the obtained results in variables
   double Peremen1 = ArrayTrans[0][5];                         
   double Peremen2 = ArrayTrans[0][6];
   double Peremen3 = ArrayTrans[0][7];
   double Peremen4 = ArrayTrans[0][8];
   // If the variable name is specified, write the result in 
   // global variables
   if(Per1 != "")
     {
       GlobalVariableSet(Per1, Peremen1);
     }             
   if(Per2 != "")
     {
       GlobalVariableSet(Per2,Peremen2);
     }
   if(Per3 != "")
     {
       GlobalVariableSet(Per3,Peremen3);
     }
   if(Per4 != "")
     {
       GlobalVariableSet(Per4,Peremen4);
     }
   Comment(Per1, " ", Peremen1, "  | ", Per2, " ", Peremen2, "  | ", Per3, 
           " ", Peremen3, "  | ", Per4, " ", Peremen4);
   Print(Per1, " ", Peremen1, "  | ", Per2, " ", Peremen2, "  | ", Per3, 
         " ", Peremen3,"  | ",Per4," ",Peremen4);
  }  // Function ends. That's all, automated optimization is complete.

Auto Optimizer Operation Results

The operation results of the automated optimizer can be observed using messages that appear in the upper left corner of the chart as shown in the screenshot below:

 // Limitation of minimal amount of trades per day
double MinTr   = TestDay - 2; 
// Limitation on maximal amount of trades per day
double MaxTr   = (60 / Period()*TestDay) + 2;
// The amount of attempts to copy the report file
int    KvoPptk = 10;
// The amount of strings to be sorted
int    StepRes = 12;

Conclusion

This article is not aimed at teaching newbies elements of optimization, so it is highly recommended to learn normal optimization before setting up automated optimization of your Expert Advisor. It is better to use the automated optimizer after you have chosen the basic variables that will influence your Expert Advisor differently at different times. I.e., it is better to use this automated optimizer to fit parameters of the variables the changes in which influence the EA operation more than those in other variables, depending on the market volatility.

Besides, it is better not to set a very large automated optimization period. Suppose the Expert Advisor has been optimized for 6-12 hours every day. Then a question occurs: When will it trade? In other words, optimization is not necessary as itself. It is recommended to set optimization periodicity (the periodicity of optimizer launching is meant) considering the timeframe, on which the EA is supposed to trade. This means that it is necessary to consider that the historical data will be pumped when the Tester-Terminal is started and it is possible that the broker just does not possess the necessary historical data for the specified period of time. To verify the hypothesis described at the beginning of this article, you will need 24-hour and stable internet connection.

The automated optimization programs developed are located in the attached files: auto_optimization.mqh - the library itself, MACD Sample_1.mq4 - slightly changed Expert Advisor included in the MetaTrader 4 Client Terminal standard delivery set.