Tradestation´s slow FileAppend

Posted by admin

Tradestation and Multicharts Fileappend command is too slow for many operations. Here i would like to introduce a solution

Compatibility

This library is not dedicated for Tradestation. It will work with every other platform such as Multicharts, Ninjatrader, C#. Our demonstration here is just for demo.

Details

Lets imagine you want to export OHLC Data as CSV to be able to analyse from an external tool. The problem is that Fileappend will be very slow operating, this is probably because Tradestation has only single command (its easy to handle) to Append and it probably does open and close by every single line and this requires time.

If we wanted export all Bar OHLC Data as external .csv file we need a code like seen:

// code snipplet
OHLC="ohlc=O="+Numtostr(o,6)+";"
		+"H="+Numtostr(h,6)+";"
		+"L="+Numtostr(l,6)+";"
		+"C="+Numtostr(c,6)+";"
		+"DATE="+Numtostr(Date,0)+";"
		+"TIME="+Numtostr(TIME,0)+";"		
		+"VOLUME="+Numtostr(Volume,1)+";"+Newline;

Fileappend(FN,OHLC);

Exporting 1 Year of M1 Data contains around 726000 Bars in Tradestation (EURUSD), to export all these bars Fileappend() needs 65 seconds to complete the operation (at least on my very fast computer with Flash/SSD Harddisk). If you need to export 10 years of data it would require over 12 minutes to complete, not very efficient.

Solution Approach

I have developed a library called fast_appender.dll, this library is allowed to accept lines to append, it does not open, close by every single append operation. It opens the file and keeps appending until you say to close.

The import is like seen:

external: "fast_appender.dll", string, "faVersionA";  
external: "fast_appender.dll", int, "faInitA",LPSTR,int; 
external: "fast_appender.dll", int, "faAppendA",int,LPSTR; 
external: "fast_appender.dll", int, "faClose",int;

I have made tests for both variants to append, with fileAppend native command of Easylanguage and FastAppender Library:

As you see FastAppender is over 5 times faster in appending to text files. 5 times speed improvement is not something you should underestimate, there is huge difference between 5 seconds and 25 seconds per chart and symbol and exported file.

FastAppender

int faInitA(string Filename,bool Append)

This is first function you need to call and call it only once (once begin….end block) at start. First parameter is name of file, second parameter accepts either 0 (false) or 1 (true), if you call it with 1, it will Append to existing file, if you call it with 0, it will Overwrite file and start Appending from first byte. This function returns a handle as int, you need to remember this handle, it will be reused in other commands

int faAppendA(int Handle,string LineToAppend)

This function will append LineToAppend to previously opened File, the Handle is taken from faInitA.

int faClose(int Handle)

This closes the handle, you will not need this function in Multicharts / Tradestation, you can need this function if you need to call from other platforms. Once you close the handle, you also close the file and you cannot Append anymore. If you are sure you will not append anymore, then you can close it.

string faVersionA()

Returns Version information of library.

Demonstration

This example exports OHLC Data of all bars as .csv file

Inputs: ExportDirectory("c:\DATA1\");
Vars: OHLC(""),intrabarpersist FN(""),BInterval(0),intrabarpersist fa(0);

// Import fast_appender
external: "fast_appender.dll", string, "faVersionA";  
external: "fast_appender.dll", int, "faInitA",LPSTR,int; 
external: "fast_appender.dll", int, "faAppendA",int,LPSTR; 
external: "fast_appender.dll", int, "faClose",int;

Once Begin
	if BarType=1 then BInterval=BarInterVal;
	if BarType=2 then BInterval=24*60; 
	if BarType=3 then BInterval=24*60*7;
	if BarType=4 then BInterval=24*60*30;
	If BarType=12 then BInterval=(Range/1 Point);	
	FN=ExportDirectory+getsymbolname+"-"+Numtostr(BInterval,0)+".csv";	
	// open File Handle and remember Handle in fa variable
	fa=faInitA(FN,1);  
end;

OHLC="ohlc=O="+Numtostr(o,6)+";"
		+"H="+Numtostr(h,6)+";"
		+"L="+Numtostr(l,6)+";"
		+"C="+Numtostr(c,6)+";"
		+"DATE="+Numtostr(Date,0)+";"
		+"TIME="+Numtostr(TIME,0)+";"		
		+"VOLUME="+Numtostr(Volume,1)+";"+Newline;
 
if (fa>0) then faAppendA(fa,OHLC);

Download

You can download it here

Final Words

FastAppender library is working with threads, you submit the data which needs to be appended and it will do the actual work in background to ensure speed. This library is over 5x faster than native command. It replaces native Fileappend command completely, you can use it for all your Append, Export operations.Have fun using it