ASE Home Page Products Download Purchase Support About ASE
ChartDirector General
 Forum HomeForum Home   SearchSearch

Message ListMessage List     Post MessagePost Message

  Having out of memory issue.
Posted by niket on Sep-23-2014 17:15
| Reply | Monitor
Attachments:
Hi,

  I am using a graph having multiple y axis scale that can be enable at run time by clicking check boxes that you can see from screen shots that i have attached.
  I am also using two cursor to display parameters value.  graph values come from a file   which might be over a GB, when i start the timer to read file content and plot on graph,
it works fine for up to some extent of data from the file, but after some limit that i could not able to find, it just shows a cross mark in red color on plot area.

I have attached two screen shots one for running mode and another one for the problem that i am facing.

thanks in advance for any help.

regards
Niket singh
error.png
running.png

  Re: Having out of memory issue.
Posted by Peter Kwan on Sep-24-2014 01:38
| Reply | Monitor
Hi niket,

From your screen shot, it seems you are using ASP.NET in a web application. The "Red X"
in this case means the browser is requesting a chart image that does not exist on the
server. To diagnose the problem, I would like to know more details about the application
and how do you know it is due to out of memory.


(a) Does the chart update itself periodically and automatically like in the "Realtime Chart"
sample code? What is the time interval between updates?

(b) Is the server directly under your control, or is it a leased server or a remote server
not under your control? Is it a single physical server, or is it a server pool using load-
balancing or a cloud based system?

(c) How large are your data series in terms of total number of data values (number of
values in each series x number of series)?

(d) From your description, it seems you start from 00:00:00, and then your code read
more and more data from the file (as the timer elapsed) and update the chart, and at a
certain point, the red X occurs. Is it possible to directly jump from 00:00:00 to say
17:00:00? I would to determine if the issue is due to too much data at 17:00:00, or too
many large charts generated too fast?

(e) Have your run your system in Visual Studio debug mode and did the Visual Studio
debugger shows any error message (such as out of memory exceptions)?

(f) Have you monitor the server memory usage and the ASP.NET process memory usage,
and are they using excessive memory?

(g) Does you code use image maps (call getHTMLImageMap)?


ChartDirector can easily support a few million data points. Of course, you must also have
sufficient also memory. For huge number of data points, the chart image itself does not
use that much memory, but the "image map" can use a lot of memory. So you may
consider to disable the image map if you want to conserve memory.

Also, after the chart is created, ChartDirector by default caches the chart in memory for
60 seconds. It is because the server cannot know if the browser has received the chart
image. Due to networking or other issues, it is not uncommon for the browser not to
receive the image correctly, and can attempt retransmission. So the server must keep
the image in case the browser asks for retransmission later. If your code is producing
very large charts and the browsers are request charts at a very fast rate and the server
memory is not enough, it is possible for the cache to be full, and so new charts cannot
be created. If this is the cause of the problem, you can ask ChartDirector to generate
the chart as temporarily files on the hard disk (see BaseChart.makeTmpFile) instead of
using memory to cache charts. This can converse memory.

Regards
Peter Kwan

  Re: Having out of memory issue.
Posted by niket on Sep-25-2014 02:08
| Reply | Monitor
hi Peter,
    first of all thanks for your attention towards my problem, first i am not      using ASP.NET, actually it is a desktop application which is based on WPF and as language i am using C#.
Now for your all question these are answers :-
a) yes as sample realtime chart application mine is also update automatically after timeout, see i am using timer but after every timeout the timeout function called where i disable timer and after process various parameters approx 150 parameter are there, after processing i invoke drawchart function to render new image on plot area, after that i enable the timer again,
so basically it is taking 500ms or depends upon processing speed.
   b) there is no server as it is standalone desktop application which uses log file where all data contains.
c) see, number of values purely depends upon log file, see i used to read one line from log file and update list for particular parameter. actually i maintaining list for each parameters and in  my application i have 150 numbers of parameters which i can enable on plot area by dividing the scale y.

for example one line of file is like this :-
00:00:00:010,0,0,0,0,0,0,0,0,0,1,0,0,1,0,1,0,0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,1107,2271,2269,2271,2,F,5.4,8.2,88.0,0,0,0,1,0,0,0,0,65,F,,0,0,0,0,0,0,0,1,0,0,1,0,1,0,0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0,11,0,0,0,01,1,2,4,0,0,0,0,0........for 150 paramters
   d)  I am coming to this point later in this post.
e) yes i have run mine system in visula studio debug mode and for making my chart clickable i use getHTMLImageMap,  i got an exception in this call only saying out of memory exception.
f) as i already said  i have not used ASP.NET
g) yes i am using getHTMLImageMap to make my chart clickable .  i need clickable chart because i need to select parameters from pause chart mode by clicking on line which is already plotted and by pressing del button i can perform delete the parameters from chart plot area, which i am doing also by using  tick marks for various parameters which can be seen in screen shots.
Now important points is , i used to plot 0 to 1000 points in timescale in running mode,  i mean to say in running mode a graph image makes of only 1000 points  , but say if run for 5 min and then press pause button i used to plot entire graph with whole points from 0 to whatever point was last, and then if i start moving the cursor fastly around chart plot area it hangs and display x mark. In movement of cursor i used to make track change call and then i call draw chart image.

regards
Niket

  Re: Having out of memory issue.
Posted by Peter Kwan on Sep-25-2014 16:04
| Reply | Monitor
Hi niket,

Sorry for incorrectly assuming it is an ASP.NET application. When I saw the "red X", I
thought about the WebChartViewer because the WebChartViewer would output a "red X" to
indicate the browser is requesting a chart that does not exist. Now looking more carefully, I
realize this "red X" is different from the WebChartViewer "red X". May be this "red X" is
output by the WPF framework to indicate some issues in rendering the control.

For your case, a quick try is add the following code immediately after
"viewer.updateDisplay();" (the line that updates the track line on the screen):

System.GC.Collect();

The above code forces the .NET framework to perform garbage collection to reclaim
memory.

Each time the screen is updated, certain memory is used depending on screen size, and an
equivalent amount of memory is unused. So there should be no net memory usage.
However, the .NET framework will not immediately reclaim any unused memory. It will delay
for sometime, by then there can be a lot of unused memory, and then reclaim them all at
once. It is possible the .NET system delays by too much, and the system runs out of
memory even when there is a lot of unused memory pending to be reclaimed. If this is the
cause of the problem, the System.GC.Collect(); should solve the problem.

Please kindly let me know if the above works or not.

Regards
Peter Kwan

  Re: Having out of memory issue.
Posted by Niket on Oct-30-2014 03:04
| Reply | Monitor
Attachments:
Hi Peter,

First of all sorry for the late response due to some reason, i have attached the garbage
collector code right
after viewer.updateDisplay() which is in drawCursor function, and also after
chartWinRef.ImageMap assignment statement in DrawChart function which you can see
in exception screenshot that i have attached. I mentioned in the last post that X mark
comes when i pause the chart and try to move cursor around chart plot area which
having huge data. I can surely say that garbage collector code help me very much.

Now X mark is not coming but it crashed  at " chartWinRef.Chart = c; ". One thing that i
want to clear that when chart is in play mode it means application reads one line from
text file which contains data for 150 parameters and plot graph for first data point for
150 parameters(or depend on selected parameters) then in next read it plot for two data
points for selected parameters , this will go up to 1000 data points after that in next
read 1000th  place data  replaced by new read data and this shifting will be happen until
file ends.

so each plot happening in play mode having only 1000 data points for each parameters.
But in pause mode it plot the chart from data point 1 to whatever data points last read ,
for example if application read 20000 data point at particular time and if i pause the
graph then it will plot the graph using 20000 data points for each parameters.


so in our case i left the graph to plot for some hour then i press pause button then it
tried to plot the graph but it crashed, at that time number of data points was  255843.

i observed the size of memory in task manager was around 882440K , but sometimes it
was different also.

one more thing that when i press continue button when the app crashed it again show
the exception at   chartWinRef.ImageMap =
chartWinRef.Chart.getHTMLImageMap("clickable", "",
                  "title='{dataSetName}'");
snippet which is right after   chartWinRef.Chart = c;

when i try to press continue button again again it shows the exception at
chartWinRef.ImageMap assignment statement.

I am looking for your help

Regards
Niket
outOfmemoryScreen.png
outofMemeory2.png

  Re: Having out of memory issue.
Posted by Peter Kwan on Oct-30-2014 23:34
| Reply | Monitor
Hi Niket,

I am thinking, is it possible the system really runs of our memory due to the large amount
of data?

My understanding is that there are 255843 points for each parameter, which means there
are 255843 x 150 = 38376450 values. If each data point is a double value, which is 8
bytes per value, it needs around 300M of memory. It means your own would need at
least 300M of memory just to store the data, even before it starts to use ChartDirector.

If your storage is based on something like a List<double> or ArrayList, it may need twice
of triple the amount. It is because the List<double> internally usually allocate more
memory than necessary, so that you can add more data without having to reallocate the
memory. Also, eventually you would need to convert the data to an array (using the
ToArray method) in order to pass the data to ChartDirector, and this creates yet another
copy of data.

The above just assumes all data are double values. If there are text strings or DateTime
objects, they can easily use 20 times the memory compared to double values.

I am not sure how your code reads the data. The database drivers or I/O buffers
probably needs memory, and the .NET itself needs memory.

You can see your own code can easily use many hundreds of MB of memory just to store
your data.

When you pass the data to ChartDirector, ChartDirector needs to maintain a copy of the
data, so it is again a lot of memory used. For each data point (which is 8 byte for a
double value), ChartDirector needs to compute and remember its coordinates and shape
and other housekeeping information. So you can see each data point can easily consume
several dozens of bytes, and it can easily adds up to many hundreds of MB or even over
1GB of memory.

Now no matter how many memory you have, a 32-bit process can use at most 2GB of
memory. (See http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx).
So it is quite possible the system simply runs out of memory.

For the number of data points you have, the image map certainly will use much more
memory that the chart itself, but I see in your case your system has run out of memory
even before the image map is generated. In any case, you should consider to not use
any image map if there are more than 10000 values.

To solve the problem, you may note that in practice, the screen is only around 2000
pixels wide, and has 2M pixels in total. So it is never necessary to plot so many points, as
there are insufficient pixels to show the points. The points simply overlap and overwrite
each others, so most of the points are invisible anyway.

If you are drawing a line chart, ChartDirector has a fast line mode (see
LineLayer.setFastLineMode). With this method, you code still passes a lot of data points
to ChartDirector, and so ChartDirector still needs quite some memory to store the data.
ChartDirector would then try to analyze the data (which needs more memory per point)
and determine which points can be skipped (as the plot area does not have so many
pixels to plot all points). So it can somewhat reduces memory usage as ChartDirector only
needs to compute the coordinates, draw and generate the image map for some of the
points, but it would still need to store and analyze all the points.

A more efficient method is for your code to reduce the data before passing the
ChartDirector. For example, you may reduce the 255843 points to 2000 points to better
match with the screen size. In this way, ChartDirector does not even need to store and
analyze all the points, but your own code would still needs a lot of memory.

The most efficient method is for your code to not to store all the data in memory,
because they can never be plotted at once anyway (due to limited number of pixels). For
example, the data can be stored in a database, and you can use an aggregate query to
aggregate the data to 2000 points when you need to plot all of them.

Hope this can help.

Regards
Peter Kwan