ASE Home Page Products Download Purchase Support About ASE
ChartDirector Support
 Forum HomeForum Home   SearchSearch

Message ListMessage List     Post MessagePost Message

  Help with dynamic data and scrolling
Posted by Mauricio on Aug-28-2012 01:04
| Reply | Monitor
Hi,

In my application I have to plot huge data sets: 200+ channels and each channel has
150,000,000+ data points. With a total of over 30,000,000,000
points (that is right, 30 billion samples), and since each data point is stored in a double, I
would need over 223 GB of memory just to hold the data. Since
that is obviously not a option, I'm dynamically loading the data in manageable chunks.

My code works as follows:
  - I have three buffers of data: currentDataSet, previousDataSet, nextDataSet. These
sets of data overlap in half their size.
  - I display the currentDataSet set always, and I track how close I am to the limit of my
array.
  - If I get "too close" to the end of my array, then I switch to make the currentDataSet
= nextDataSet (and I update the data for the previousDataSet
and nextDataSet). Similarly, if I get "too close" to the beginning of my array, then I
switch to make the currentDataSet = previousDataSet
  - Finally I adjust the x-axis and I'm good to go.

I started my code based on the "ZoomScrollTrack" example. However, I'm having a few
bugs that I need help with:
1.  I'm loading the new data (for the currentDataSet, etc) on the drawChart function. On
that same function I update the x-axis. However, when I use
the drag function to move to the right, as soon as I update the x-axis, then the
"viewPortStartIndex" has new values, which are used for the next
section of the drag. Therefore, my x-axis keeps increasing like crazy.  This would not
happen if I only updated the dataSets at the end of the drag (and
not in all intermediate steps). My question is: How can I know, in a  ViewPortChange
event, that the mouse drag is complete?

This is how the code looks:

void CZoomscrolltrackDlg::drawChart(CChartViewer *viewer)
{
    // Get the start index and end index that are visible on the chart.
    double viewPortStartIndex = viewer->getValueAtViewPort("x", viewer-
>getViewPortLeft());
double viewPortEndIndex = viewer->getValueAtViewPort("x", viewer-
>getViewPortLeft() +
        viewer->getViewPortWidth());

viewPortStartIndex = viewer->
// load new data sets
loadData(viewPortStartIndex, viewPortEndIndex);

    // Get the array indexes that corresponds to the visible start and end dates
int startIndex =  (int)floor(Chart::bSearch(m_timeStamps, viewPortStartIndex));
int endIndex =  (int)ceil(Chart::bSearch(m_timeStamps, viewPortEndIndex));
    int noOfPoints = endIndex - startIndex + 1;

    // Extract the part of the data array that are visible.
DoubleArray viewPortDataSeries[m_channels];
    DoubleArray viewPortTimeStamps = DoubleArray(m_timeStamps.data + startIndex,
noOfPoints);
for (int ch=0; ch < m_channels; ch++)
{
viewPortDataSeries[ch] = DoubleArray(m_dataCurrentSet[ch].data +
startIndex, noOfPoints);
}

    //
    // At this stage, we have extracted the visible data. We can use those data to plot
the chart.
    //

///////////////////////////////////////////////////////////////////////////////////////
    // Configure overall chart appearance.
    ///////////////////////////////////////////////////////////////////////////////////////

    // Create an XYChart object of size 650 x 350 pixels, with a white (ffffff) background
and grey
// (aaaaaa) border
    XYChart *c = new XYChart(1050, 650, 0xffffff, 0xaaaaaa);

    // Set the plotarea at (55, 55) with width 90 pixels less than chart width, and height
90 pixels
    // less than chart height. Use a vertical gradient from light blue (f0f6ff) to sky blue
(a0c0ff)
    // as background. Set border to transparent and grid lines to white (ffffff).
    c->setPlotArea(55, 55, c->getWidth() - 90, c->getHeight() - 90, c-
>linearGradientColor(0, 55, 0,
c->getHeight() - 35, 0xf0f6ff, 0xa0c0ff), -1, Chart::Transparent, 0xffffff,
0xffffff);

    // As the data can lie outside the plotarea in a zoomed chart, we need enable clipping.
    c->setClipping();

    // Add a title to the chart using 18 pts Times New Roman Bold Italic font
    c->addTitle("   Zooming and Scrolling with Track Line (1)", "timesbi.ttf", 18);

    // Set legend icon style to use line style icon, sized for 8pt font
    c->getLegend()->setLineStyleKey();
    c->getLegend()->setFontSize(8);

    // Set the axis stem to transparent
    c->xAxis()->setColors(Chart::Transparent);
    c->yAxis()->setColors(Chart::Transparent);

    // Add axis title using 10pts Arial Bold Italic font
    c->yAxis()->setTitle("Ionic Temperature (C)", "arialbi.ttf", 10);

    ///////////////////////////////////////////////////////////////////////////////////////
    // Add data to chart
    ///////////////////////////////////////////////////////////////////////////////////////

    //
    // In this example, we represent the data by lines. You may modify the code below to
use other
// representations (areas, scatter plot, etc).
    //

    // Add a line layer for the lines, using a line width of 2 pixels
    LineLayer *layer = c->addLineLayer();
    layer->setLineWidth(1);

    // In this demo, we do not have too many data points. In real code, the chart may
contain a lot
    // of data points when fully zoomed out - much more than the number of horizontal
pixels in this
    // plot area. So it is a good idea to use fast line mode.
    layer->setFastLineMode();

    // Now we add the 3 data series to a line layer, using the color red (ff0000), green
    // (00cc00) and blue (0000ff)
    layer->setXData(viewPortTimeStamps);

for (int ch = 0; ch < m_channels; ch++)
{
std::string ch_prefix = "ch";
std::stringstream ch_name;
ch_name << ch_prefix << ch;
layer->addDataSet(viewPortDataSeries[ch], 0xff3333, "");
}

    ///////////////////////////////////////////////////////////////////////////////////////
    // Configure axis scale and labelling
    ///////////////////////////////////////////////////////////////////////////////////////

    // Set the x-axis as numeric axis with the scale according to the view port x range.
viewer->syncLinearAxisWithViewPort("x", c->xAxis());
viewer->syncLinearAxisWithViewPort("y", c->yAxis());

    ///////////////////////////////////////////////////////////////////////////////////////
    // Output the chart
    ///////////////////////////////////////////////////////////////////////////////////////

    delete viewer->getChart();
    viewer->setChart(c);
}

void CZoomscrolltrackDlg::loadData(double startIndex, double endIndex)
{

if (  ((startIndex >= m_indexCurrentSet + m_switchDatasize) && (endIndex <=
m_indexCurrentSet + m_chunkSize - m_switchDatasize)) || //
indexes within rage
  (startIndex  <= m_minIndex + m_switchDatasize)  || // index already at the
beginning of all data set
  (endIndex    >= m_maxIndex - m_switchDatasize)     // index already at the
end of all data set
   )
{
// No need to load new data
return;
}

if (startIndex < m_indexCurrentSet  &&  endIndex > m_indexCurrentSet +
m_chunkSize)
{
// ERROR: the new indexes span over data sets. THIS SHOULD NEVER HAPPEN
exit(-1);
}

// Fill in the data
double *data = NULL;
if (startIndex < m_indexCurrentSet + m_switchDatasize)
{
m_indexCurrentSet = (((int) startIndex / (int) m_halfChunkSize) - 1) * (int)
m_halfChunkSize;

int chunkNumber = m_indexCurrentSet / m_halfChunkSize;

// going back, therefore, delete the m_dataNextSet data
for (int ch=0; ch < m_channels; ch++)
{
delete m_dataNextSet[ch].data;
m_dataNextSet[ch] = m_dataCurrentSet[ch];  // new next data set
m_dataCurrentSet[ch] = m_dataPrevSet[ch];  // new current data set
}

// new previous data set -> must be loaded from file ( THIS SHOULD BE
SPAWN ON A DIFFERENT THREAD )
for (int ch=0; ch<m_channels; ch++)
{
data = new double[m_chunkSize];
for (int i = 0; i < m_chunkSize; i++)
{
int r = rand();
data[i] = ch*m_ch_offset + ((r%10)/100.0) + (chunkNumber-
1)/10.0;
}
m_dataPrevSet[ch] = DoubleArray(data, m_chunkSize);
}
}
else // endIndex > m_indexCurrentSet + m_chunkSize - m_switchDatasize
{
m_indexCurrentSet = ((int) endIndex   / (int) m_halfChunkSize) * (int)
m_halfChunkSize;
int chunkNumber = m_indexCurrentSet / m_halfChunkSize;
// going forward, therefore, delete the m_dataPrevSet data
for (int ch=0; ch < m_channels; ch++)
{
delete m_dataPrevSet[ch].data;
m_dataPrevSet[ch] = m_dataCurrentSet[ch];  // new previous data set
m_dataCurrentSet[ch] = m_dataNextSet[ch];  // new current data set
}

// new next data set -> must be loaded from file ( THIS SHOULD BE SPAWN
ON A DIFFERENT THREAD )
// TODO: HANDLE CASE, AT THE END OF THE FILE, WHERE THE LAST CHUNK
OF DATA IS NOT A MULTIPLE OF m_chunkSize
// if we are already at the end of the file, then this is not touched (it is left
with the ending data)
for (int ch=0; ch<m_channels; ch++)
{
data = new double[m_chunkSize];
for (int i = 0; i < m_chunkSize; i++)
{
int r = rand();
data[i] = ch*m_ch_offset + ((r%30)/100.0 +
(chunkNumber+1)/10.0);
}
m_dataNextSet[ch] = DoubleArray(data, m_chunkSize);
}
}

// build the new "x" axis values
// TODO: Handle the case for the last chunk which might not be a multiple of the
m_chunkSize.
data = new double[m_chunkSize];
for (int i = 0; i < m_chunkSize; i++)
{
data[i] = (double) i + m_indexCurrentSet;
}

//if (m_timeStamps.data) delete m_timeStamps.data; // delete previously held
memory.
m_timeStamps.data = data;
m_timeStamps.len = m_chunkSize;

m_ChartViewer.setFullRange("x", m_timeStamps[0],
m_timeStamps[m_timeStamps.len - 1]);
double viewPortPercent = (startIndex-
m_timeStamps[0])/(m_timeStamps[m_timeStamps.len - 1] - m_timeStamps[0]);
m_ChartViewer.setViewPortLeft(viewPortPercent);
//m_ChartViewer.updateViewPort(false, true);
}

If I did not explained myself don't hesitate on letting me know.

I appreciate your help.

As a side info, Is there a way to store the data points in an IntArray instead of a
DoubleArray? I assume that would save at least half of the memory.

  Re: Help with dynamic data and scrolling
Posted by Peter Kwan on Aug-28-2012 06:47
| Reply | Monitor
Hi Mauricio,

When the chart is scrolling, the scroll events can occur very rapidly. In most cases, ChartDirector should be fast enough to plot the charts even if the scroll events are very rapid. For your case, it also depends on how fast you can get the new data when the chart is scrolling. (Note: For C++, the "Release Mode" can be significantly faster than "Debug Mode".)

Anyway, if you thinking the scrolling events are firing too rapidly, and you would like to only process the last scroll event, the followings are some ideas:

(a) If the drag to scroll is because the user drags on the chart itself, ChartDirector will always send CVN_ViewPortChanged messages with CChartViewer.needUpdateImageMap being false for all the view port changed events except the final event. So by checking the CChartViewer.needUpdateImageMap, you can determine if it is the final scroll event.

(b) If the scrolling is initiated by some other controls or code unrelated to ChartDirector (such as by a Windows scrollbar control), it is not possible for ChartDirector to know when that external control has finished. You would need to refer to the documentation on that control to see when does it finish. For the Windows scrollbar control, you can probably check the SB code (see http://msdn.microsoft.com/en-us/library/e14hhbe6(v=vs.80).aspx) to see if it is SB_ENDSCROLL. I believe currently the sample code does not check for SB_ENDSCROLL. You can modify it to check for SB_ENDSCROLL and to trigger the view port changed event with needUpdateImageMap = true (so that your other code knows it is the final scroll event).


I am not sure if you need to support "zoom in/out" as well. If you are 100% zoom out, you would need to plot 30,000,000,000 points on the chart all at once (and it is only for 1 channel).

Of course, it is not meaningful to plot more points than the number of pixels on the screen. ChartDirector already has a "Fast Line" mode (see LineLayer.setFastLineMode) to reduce the line to fit the graphical resolution, but it still takes sometime to process 30,000,000,000 points to determine how to reduce them.

One common method to deal with extremely large data series is to create multi-resolution versions of the data series. In addition to the 30,000,000,000 points series, you can create a series that reduces the original series to 10% of its size 3,000,000,000 points, and then another series with size of 300,000,000, then to 30,000,000 and so on. All these extra lower resolution series together will only take 11.1% more space, so there is minimal overhead for all these series, and these series can be pre-generated and store in the database. When you zoom/in out, your code can select the most apprpriate data series to use based on the zoom level.

In some sense, the above method is similar to Google Map. There are multiple versions of the map at different resolution, and the map to display depends on the zoom level. The overhead is small but the speed improvement can be large.

Hope this can help.

Regards
Peter Kwan

  Re: Help with dynamic data and scrolling
Posted by Mauricio on Aug-28-2012 22:09
| Reply | Monitor
Hi Peter,

Thanks for your reply! That's exactly what I was looking for. I had read in a few different
places about the needUpdateImageMap(), but it was not clear to me how it worked.

Excellent support!!

Cheers,

- Mauricio