ASE Home Page Products Download Purchase Support About ASE
ChartDirector Support
 Forum HomeForum Home   SearchSearch

Message ListMessage List     Post MessagePost Message

  Python Memory leak when looping FinanceChart
Posted by Shane Lee on Feb-24-2014 21:20
| Reply | Monitor
FinanceChart has serious memory problem.
You can test it with python codes at the bottom.

Memory is going up to 2GB and crashes around 663rd loops
Especially, addVolBars and makeChart consumes memory which is not freed
0.5 MiB       c.addVolBars(75, 0x99ff99, 0xff9999, 0x808080)
2.2 MiB       c.makeChart("finance.png")

Test environment:
64bit windows 7
32bit python and ChartDirector

#!/usr/bin/python
from FinanceChart import *
from memory_profiler import profile

@profile
def test(i):
    """ enter codes here from pythondemo\\finance.py """

if __name__ == "__main__":

    for i in range (0, 10000):
        test(i)

  Re: Python Memory leak when looping FinanceChart
Posted by Peter Kwan on Feb-25-2014 02:07
| Reply | Monitor
Attachments:
Hi Shane,

Thanks for reporting this issue to us. I confirm this is in fact a bug in ChartDirector for
financial charts. The error is due to a circular reference that occurs when using
FinanceChart. (The Python garbage collector sometimes cannot handle circular references.)

I have attached an updated "pychartdir.py" that should fix this problem. We will update the
ChartDirector for Python release to incorporate this fix.

Again, thank you very much for reporting this issue to us.

Regards
Peter Kwan
pychartdir.py
pychartdir.py
61.45 Kb

  Re: Python Memory leak when looping FinanceChart
Posted by Shane Lee on Feb-25-2014 11:55
| Reply | Monitor
Dear Peter,

It's very nice of you to fix this issue so quickly.
Now, it's working fine.

Thanks and regards,
Shane

  Re: Python Memory leak when looping FinanceChart
Posted by Chris on Apr-13-2014 21:35
| Reply | Monitor
Hello Peter

I am also experiencing memory leak problems with the PHP version when looping
FinanceChart creation.

Is it possible that the same problem is causing issues with the PHP version too?

Thank you.

  Re: Python Memory leak when looping FinanceChart
Posted by Peter Kwan on Apr-15-2014 00:17
| Reply | Monitor
Hi Chris,

The same problem would not occur in PHP.

However, ChartDirector for PHP is designed not to automatically free charts until the PHP
script ends, and there would be no memory leak after all. This method normally works
because PHP is designed to process web request, and it is not likely a single web page will
have hundreds or thousands of dynamically generated charts. ChartDirector is designed this
way to make it compatible with both PHP 4.x and PHP 5.x. (In PHP 4.x, this is the only
method to free memory allocated by a shared object.)

Occasionally, people may use PHP as command line scripts, and use a loop to create
thousands of charts. In this case, the code may need to call "garbageCollector();" to free
the charts before the script ends. The "garbageCollector();" will free all previous charts
created by ChartDirector in the same script instance.

For your case, are you using PHP as command line scripts (or to run it in batch files or in
CRON jobs), and creating hundreds or thousands of chart?

Regards
Peter Kwan

  Re: Python Memory leak when looping FinanceChart
Posted by Chris on Apr-17-2014 05:25
| Reply | Monitor
Hi Peter

Sorry, I missed your reply.

As you suggested, I am using a Cron job to loop through the creation of about 6000 finance
charts, however after about 250 iterations, the script runs out of memory, which is why I
thought there might be a memory leak.

I'm therefore having to run the creation of the finance charts in batches of 250 which works,
but is painful to set up Cron tabs for, and to control.

I'll have a look at the Garbage Collector to see if that helps, but would welcome any other
ideas or solutions to the creation of so many charts using PHP, as the number of finance
charts is very likely to increase soon.

Thanks
Chris

  Re: Python Memory leak when looping FinanceChart
Posted by Chris on Apr-17-2014 16:49
| Reply | Monitor
Hi Peter

I have added garbageCollector() to the PHP script, but it is not making any real difference.
There are still only around 250 charts being created before the memory is depleted.

This is the code that I use to generate each image:

# Create a FinanceChart object of width 640 pixels
$c = new FinanceChart(640);

# Add a title to the chart
$c->addTitle($name);

# Set the data into the finance chart object
$c->setData($timeStamps, $high, $low, $open, $close, $volData, $extraDays);

# Add the main chart with 240 pixels in height
$c->addMainChart(240);

# Add a custom text at (40, 45) (the upper left corner of the plotarea). Use 10
pts
$c->addText(40, 45, "ChartSpy.com", "arialbd.ttf", 10, 0xc0c0c0);

# Add a 20 period simple moving average to the main chart, using purple color
$c->addSimpleMovingAvg(20, 0x9900ff);

# Add a 5 period simple moving average to the main chart, using grey color
$c->addSimpleMovingAvg(50, 0x808080);

# Add candlestick to the main chart, using green/red for up/down days
$c->addCandleStick(0x008000, 0xcc0000);

# Add a 75 pixels volume bars sub-chart to the bottom of the main chart, using
# green/red/grey for up/down/flat days
$c->addVolBars(75, 0x99ff99, 0xff9999, 0x808080);


# Append a 12-days Slow Stochastic indicator chart (75 pixels high) after the
main chart.
$c->addSlowStochastic(75, 12, 3, 0x9900ff,0x99ff99);


# Output the chart
if ( $chartSymbol != "" ) {
$chartPath = "../charts/" . strtolower($chartSymbol) . "_" . $chartDate .
".png";
if ($c->makeChart($chartPath)) {
writeToLogFile( $logFile, $symbol . " : Creating finance chart " .
$chartPath . " : Success");
} else {
writeToLogFile( $logFile, $symbol . " : Creating finance chart " .
$chartPath . " : Failed");
}
}

garbageCollector();


This is the error message that occurs after about 250 iterations. The script is running on a
VPS:

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 71 bytes) in
/home/********/public_html/chartdirector/lib/phpchartdir.php on line 3490

  Re: Python Memory leak when looping FinanceChart
Posted by Peter Kwan on Apr-18-2014 02:24
| Reply | Monitor
Hi Chris,

I have just performed some testing myself. In my test case, I have not found any evidence
of memory leak.

In my testing, I created 1000 charts on Windows. I found the memory usage increases by
around 100K bytes per chart. By the time the script created around 500 charts, the memory
usage was 50M to 60M. Then the memory usage suddenly dropped to 15M and then the
memory usage increased again until the script finished producing 1000 charts. The memory
usage never exceeded 60M.

The above memory usage pattern is typically of a system using garbage collection to
manage memory, of which PHP is one. The PHP interpreter intentionally not freeing the
memory until it reaches a certain threshold - such as 50M to 60M. Then it frees all the
memory in one batch. This saves CPU time. I am not sure how PHP determines the
threshold. It may depend on how much memory the computer have. For a computer with
4GB of RAM, 60M is a small amount of memory, and can be a reasonable threshold.

For your case, the 33M limit is probably the default limit configured in "php.ini" for each
script. The default probably assumes PHP is used in a web application. For this type of
usage, many requests can come at the same time, and the script needs to be short running
and shoudl not be resource intensive, so a small script limit is appropriate. As you are using
PHP in a long running command line or batch application, the default limit may not be
appropriate. I suggest you may use a bigger limit, such as 512M.

Hope this can help.

Regards
Peter Kwan

  Re: Python Memory leak when looping FinanceChart
Posted by Chris on Apr-18-2014 02:52
| Reply | Monitor
Thank you, Peter, for investigating this.

We are using default settings in PHP.ini, hence the low memory allocation, as you explained
so well.

Given these issues with PHP, I have decided to offload the chart creation to a Windows VPS
and created a VB Script version of our batch chart processing.

The Windows VPS easily handled the creation of 6000 charts without any issues at all, so this
is definitely the way to go. This system should scale very well from here, which is
encouraging.

I am highly impressed with ChartDirector and its low resource utilisation and small file sizes.

Thank you for your support.

Chris