Data Structure in MetaTrader 4 Build 600 and Higher

MetaTrader 4 — Examples | 17 February 2014, 11:45

27 599

Introduction

MetaTarder 4 build 600 features the new structure and location of the client terminal files. Now, MQL4 applications are placed in separate directories according to the program type (Expert Advisors, indicators or scripts). In most cases, the terminal data is now stored in a special data folder separated from the terminal installation location. In this article, we will describe in details how data is transferred, as well as the reasons for introducing the new storage system.

Why Has the New Data Storage System Been Implemented

Microsoft Windows XP released 13 years ago allows applications to write their own data at the place of their installation even if the latter took place in Program Files system folder. A user should only have an administrator permission to write data to any folder.

When working in 64-bit systems, separate installation directories are provided for 32 and 64-bit programs: Program Files and Program Files (x86). Operation features described in the article apply to both directories.

Starting from Windows Vista, Microsoft has introduced a restriction on writing to Program Files directory. If User Account Control (UAC) system is enabled, programs are not allowed to store their data in Program Files folder. All data should be located in a separate user directory. This limitation has been introduced in order to protect users against malicious programs and to prevent applications under one user account to change or damage the same program's data necessary for another user account. Since that time, security requirements in Microsoft operating systems have been tightening further. In particular, starting with Windows 8, UAC system cannot be disabled even if "Never notify" option is selected in its settings.

Microsoft has implemented virtualization process in order to provide compatibility with older applications when using UAC. If a program tries to save its data to Program Files directory, the data is actually (physically) saved to a separate folder having the following look - C:\Users\<user-name>\AppData\Local\VirtualStore\Program Files, while Windows File Explorer shows the files as if they are saved in the installation directory. Microsoft claims that this mode has been provided only for compatibility and can be removed later.

In order to comply with Microsoft recommendations, the data storage structure has been changed in MetaTrader 4 client terminal starting from build 600. Now, the terminal will also save its data in a user directory.

User Data Directory

In the new version, all data of a certain user working with a certain copy of MetaTrader 4 terminal are stored in a special place called terminal data folder. This folder can be found on a system disk (a disk with installed Windows operating system) along the following path

C:\Users\User_account_name\AppData\Roaming\MetaQuotes\Terminal\Instance_id

where:

С: - name of a system disk;
User_account_name - user's account for working in Windows;
Instance_id - unique name of the folder where all the user's data for working with a certain copy of MetaTrader 4 terminal is stored. A unique name consists of 16 characters. The name is generated based on the path to the terminal installation directory, since the path to the terminal installation directory cannot be explicitly used as a folder name. Significant length of the unique name is explained by the fact that multiple copies of the terminal can be installed on a single PC.

"Open Data Folder" command has been added to File menu of the terminal for searching and opening the data folder.

Open Data Folder

The root of each terminal data folder also contains origin.txt file where you can find the path to the installation folder of the terminal this data refers to. This allows users to match each terminal data folder with a certain terminal installation directory, for example, in case when several copies of MetaTrader 4 terminal are installed by a user. This type of working with the terminal when the data folder is separated from the installation one is the main mode.

For more convenience, an entry containing the path to the data folder is made in the terminal's journal each time the terminal is launched. For example:

2014.02.10 12:48:28.477 Data Folder: C:\Users\JohnSmith\AppData\Roaming\MetaQuotes\Terminal\9F86138A4E27C7218E9EC98A5F8D8CA1

Copying MQL4 Application Files When Updating to MetaTrader 4 Build 600 and Higher

When the newly updated terminal is launched, it checks if the data folder is present. If the data folder is not present yet, then it is created. If that folder is different from the installation one, the terminal's regular data (standard MQL4 programs, historical data, configuration files, templates, etc.) is copied into it. The files that are not changed during the terminal operation (executable files, mql.dll compiler, sound files, etc.) are left in the installation directory. The terminal data folder is different from the installation one in the following cases:

UAC system is enabled. The exception is when the terminal is installed on a portable device (external hard drive, usb flash drive, etc.).
Current PC user has limited rights to write data to the installation directory.
A user is working via remote connection (RDP).

If none of the above conditions is satisfied, the terminal data is stored in the installation directory.

Then, the user files are moved to the data folder. At this stage, the directory where the terminal's user data has been stored is determined. If the data has been stored in the terminal installation folder, they are copied the following way:

Source folder in the installation directory	Contents	Destination folder in the terminal data directory
\experts	Expert Advisors (trading robots) Note: only files from the root \experts directory are copied. No subdirectories are copied, since it is impossible to reliably determine their contents	\MQL4\Experts
\experts\indicators	Custom indicators	\MQL4\Indicators
\experts\scripts	Scripts (MQL4 applications for a single run on the chart)	\MQL4\Scripts
\experts\include	Source code MQH and MQ4 files implemented into other programs	\MQL4\Include
\experts\libraries	Libraries in the form of MQ4 source codes and EX4 executable files compiled from them. They are used for the dynamic call of the functions contained there by other MQL4 programs	\MQL4\Libraries
\experts\files	Special "file sandbox". MQL4 applications are allowed to execute file operations only within this directory	\MQL4\Files
\experts\logs	Log files of MQL4 applications	\MQL4\Logs
\experts\presets	Files of presets for MQL4 applications	\MQL4\Presets
\experts\images	Image files for being used in resources	\MQL4\Images

Next, it is checked whether the client terminal has stored data in virtualization directory (the operating system's virtual storage described above). If the terminal has been installed in Program Files directory and the operating system is Windows Vista or higher, the terminal data is most probably stored in that directory. If the data is found, it is copied according to the above table.

The files are copied and not moved during the migration. The copied files are not deleted from the source folders.

During the migration, the entries containing source and destination paths of the copied files are made in the terminal's journal. To view all the logs, open Journal tab of Terminal window and execute Open command in the context menu. The folder containing the terminal log files will open.

If the migration has been completed successfully for the current terminal copy, it is not repeated any more during subsequent MetaTrader 4 terminal updates. If the data folder is different from the installation one, and custom MQL4 applications have been copied together with the standard files during the migration, the following dialog window appears:

Migration completed successfully

Portable Mode

Portable launch mode is provided for the terminal operation on portable devices and non-system directories, as well as for working in Windows XP. When launched in this mode, the terminal tries to save its data in the installation folder. However, using Portable mode does not guarantee that an operating system will allow storing data in the installation folder (for example, if the terminal is installed in Program Files directory and UAC system is enabled).

The following conditions should be met for working in Portable mode:

Installation directory	Operating system	Requirements
Program Files or another system directory (for example, Windows)	Windows XP	Permission to write to the installation folder
Another non-system directory	Windows XP	Permission to write to the installation folder
External hard drive, usb flash drive, etc.	Windows XP	The terminal will always be launched in Portable mode.
Program Files or another system directory (for example, Windows)	Windows Vista\Windows 7	Launching the terminal as administrator and disabled UAC
Another non-system directory	Windows Vista\Windows 7	Permission to write to the installation folder
External hard drive, usb flash drive, etc.	Windows Vista\Windows 7	The terminal will always be launched in Portable mode.
Program Files or another system directory (for example, Windows)	Windows 8 and higher	It is impossible to use Portable mode, as UAC system cannot be disabled
Another non-system directory	Windows 8 and higher	Permission to write to the installation folder
External hard drive, usb flash drive, etc.	Windows 8 and higher	The terminal will always be launched in Portable mode.

It is not recommended to use Portable mode in case you use Windows Vista or higher, and the terminal is installed in Program Files. This mode is provided for the terminal operation on portable devices and Windows XP.
If you have updated the terminal and all the data has been copied to a separate user data folder, it is also not recommended to switch to Portable mode.
You should manually copy the data to the installation folder in order to use Portable mode after copying the data in the user data folder. Launching the terminal in Portable mode does not copy the data from the data folder to the installation one.

In order to launch the terminal in Portable mode, use "/portable" key. For more convenience, you can create an additional terminal launch shortcut with the appropriate name on your desktop and add the key directly to the shortcut:

Launching MetaTrader 4 in Portable mode

Conclusion

Changes in MetaTrader 4 terminal's data storage system have been made in order to comply with the requirements of Windows operating systems developed by Microsoft. This data storage system has originally been implemented in MetaTrader 5, and many traders successfully use it at the moment.

MetaTrader 4 terminal's data is transferred to the user directory automatically. No additional actions are required from users in most cases. Information about all data copying operations is displayed in MetaTrader 4 journal.

Translated from Russian by MetaQuotes Ltd.
Original article: https://www.mql5.com/ru/articles/1388

Warning: All rights to these materials are reserved by MetaQuotes Ltd. Copying or reprinting of these materials in whole or in part is prohibited.

Other articles by this author

Last comments | Go to discussion (10)

tmourad | 24 Apr 2014 at 15:45

Really i didnt understand anything from up there ..it is like a Chinese for me.I just need a simple way of coping some files into indicators folder, templates and experts ...how can I do it and see it in the software and cancel this stupid thing called administrator ....nobody else is using my PC

Please Help

[Deleted] | 29 May 2014 at 10:25

The data storage solution is a problem for users that have several platforms installed on other partitions from their system drive. Lets say you have 4-5 platforms from different brokers. You test on some and you trade on some.

The folder size of these platforms might be around 1-3GB if you have historic data for testing, it might even be more. This patch then copies several gigs of these history files into the users profile. My own got 8 GB larger!

Coming from a system administrators perspective, it is bad practice to install that much into a user profile. Coming from a coders perspective I can see how it might be convenient, but it is NOT a good or clean solution.

The way this patch has been solved just seems very sloppy and not thought through.

You should really be able to let users move all those files out from the users profile and solve it with a config file of some sort that points to the correct location instead, especially with the files and history folders.

[Deleted] | 16 Jun 2014 at 22:55

This messed up everything!! I cannot even transfer custom indicators from one platform to another. I putt ALL MT4 folders to the impossible-to-remember-obscure-no-one-cam-find path C:\Users\<user-name>\AppData\.... and none of my indicatord are showing up!! Time to start learning a new programming language and finally trash the MT/MQL! This platform is just regressing and the mt5 "upgrade!" was living proof that they are so out of touch that no reason to stay on board!

pc-wiz | 15 Jul 2016 at 23:02

This was a really bad decision because it doesn't take into account the fact that you can install metatrader somewhere other than in a program files directory, but when you do your files go into your c: drive under this unreadable appdata directory. If you need to reinstall windows, or you redo your c: drive with an image, it'll get blasted over in the users directory and you're screwed. I keep my MQ4 files on a separate drive just for this reason. The Metaquotes authors should fix this immediately! If your installallation is not in Program Files or Program Files (x86) there is no need to comply with Microsoft and it should save everything in your MQ4 directory structure!

Rudolf De Heer | 1 Aug 2016 at 18:00

From a system administrator view, 8gb used to be a lot in a user profile. In perspective of the smaller but faster solid state disks, its still a lot. For instance a 8 GB user profile with a 80 GB system partition, results in a 10% of disk usage. The data processing design of MQL4 and 5 seems to take place in the two C:\users\username\appdata\roaming\Metaquotes\ and C:\Users\username\AppData\Roaming\MetaTrader (Broker specific name) folders

Because the virtual private service, probably uses terminal server technology, the roaming user data folder is by Microsoft design, the best place to synchronise Expert advisors, charts and indicators, to a microsoft user profile on a virtual terminal server.

A design solution could be a postgress SQL database where for instance history data is stored and accessed through a form of dlll authorisation per user. Although the history centre is gone in MQL 5. Therefore i decided to use MS excel to backtest algorhytms on historic price data. All log processing takes place in the user profile. Is a maximum log folder size applicable?

Another point of view is, to keep all standard EAs, Indicators and scripts in the C:\Program files\Metatrader 5\MQL5 directory. This only works when writing paths in the "C:\users\username\appdata\roaming\Metaquotes\terminal specific to user" are authenticated correctly. E.g. custom Expert advisors should be permitted to write and read, correctly,consistently, timely and valid.

I am not sure if the cause is Microsoft design standards or a lack of man power, budget, or priorities at the Metaquotes team. In my opinion the solution lies in application authentication and authorisation, if applicable at all. For users with no IT knowledge copying and moving indicators or EAs seems cumbersome.

MQL5 Programming Basics: Lists

The new version of the programming language for trading strategy development, MQL [MQL5], provides more powerful and effective features as compared with the previous version [MQL4]. The advantage essentially lies in the object-oriented programming features. This article looks into the possibility of using complex custom data types, such as nodes and lists. It also provides an example of using lists in practical programming in MQL5.

Offline Charts in the New MQL4

Updated MQL4 has the new format for storing historical data and provides the appropriate MqlRates structure for convenient storage of Time, Open, Low, High, Close and Volume values. For many years, traders have developed their MQL4 applications that collect and store their data in HST files for generating offline charts. We can assure you that all previously compiled EX4 files will work in the new MetaTrader 4 terminal the same way as before.

Working with GSM Modem from an MQL5 Expert Advisor

There is currently a fair number of means for a comfortable remote monitoring of a trading account: mobile terminals, push notifications, working with ICQ. But it all requires Internet connection. This article describes the process of creating an Expert Advisor that will allow you to stay in touch with your trading terminal even when mobile Internet is not available, through calls and text messaging.

Creating Non-Lagging Digital Filters

The article describes one of the approaches to determining a useful signal (trend) in stream data. Small filtering (smoothing) tests applied to market quotes demonstrate the potential for creating non-lagging digital filters (indicators) that are not redrawn on the last bars.