|
 Real-time chart: performance |
| Posted by Philip on Sep-16-2020 04:31 |
|
I want to write a desktop application so I will write the code in C++ or C#.
The real-time XY chart should support at least 100 million data points.
After reading your "Real-Time Charts and Multithreading" feature at https://www.advsofteng.com/tutorials/real_time_chart_multithread/real_time_chart_multithread.html , does C++ or C# code give better performance for reading from file, writing to file and scrolling 100 million data points?
Philip |
| Re: Real-time chart: performance |
| Posted by Peter Kwan on Sep-17-2020 01:02 |
|
Hi Philip,
C++ is faster, while C# is easier to write due to automatic memory management.
The current version of ChartDirector can definitely handle 100M data points, but the number of visible data points is at the order of 1 million. That means you would need to scroll back and forth to see the whole data. The process is CPU related, which means it can be faster with more visible data points with a faster CPU.
The next version of ChartDirector is planned to be available by early next year (or even at the end of this year). Currently, we have tested it to 3 x 100M data points (3 lines earch 100M points), all updated in realtime. It allows complete zoom in/out (that is, you can zoom out to display all 300M data points in one chart) and it can zoom and scroll fast even with that many data points. The process is memory bound. Our current testing is that it needs around 20 bytes per data value (counting both x and y values). We think with sufficient memory it may support more data points, but we are yet to test that.
ChartDirector only plots the data points in memory. Your code will need to read the data points from disk or save them to disk. I believe for large amount of data, the process will be I/O bound (it depends on the speed of your hard disk), and either language may have similar performance.
Regards
Peter Kwan |
| Re: Real-time chart: performance |
| Posted by Philip on Sep-17-2020 02:14 |
|
Hello Peter,
For the DAQ (Data AcQuisition) industry, 8 channels is very popular. Please google with phrases "DAQ" and "8 channel". This means 8 lines in your real-time XY chart. Is it possible to support 8 lines each 100M points in the next release? Maybe it could be a configuration option with higher RAM requirements?
Regards,
Philip |
| Re: Real-time chart: performance |
| Posted by Peter Kwan on Sep-17-2020 23:21 |
|
Hi Philip,
Yes, 8 x 100M should have no problem. We just have not tested it as this stage, but I have no reason to believe it would have issue.
By 8 x 100M, we mean we can plot all 8 x 100M points in "real time" (within 1 second), and update the lines many times per second.
We have some customers who have even larger data series, but they just need to display around 1M points at a time. In this case, only the visible part of the data need to be in memory, and the rest can stay on the hard disk and be loaded on demand (load the data only when the user scrolls to that part of the data). For this kind of usage, only a small amount of RAM is needed.
Regards
Peter Kwan |
| Re: Real-time chart: performance |
| Posted by Philip on Oct-03-2020 04:22 |
|
Hi Peter,
When will the "100 million points per line" feature be available for evaluation?
Philip |
| Re: Real-time chart: performance |
| Posted by Peter Kwan on Oct-06-2020 01:23 |
|
Hi Philip,
Currently, we have a beta version for .NET C#/VB, but we have not yet written the documentation on how to use it. We can provide an example Visual Studio solution that you can compile and run for testing, but without documentation, you may need to guess what the code means.
We are porting to C++ right now, and because we have many other features to port, it may take a few weeks before a beta release can be available for C++.
If you are interested in testing the .NET C#/VB now, I can ask our developer to create a Visual Studio example that generates 100 million data points per line in real-time so you can see how it works.
Regards
Peter Kwan |
| Re: Real-time chart: performance |
| Posted by Philip on Oct-14-2020 03:42 |
|
Hello Peter,
I would like to take you up on your offer from your last post. Please forward a .NET C# solution that generates 100 million data points per line.
Will the demo show complete zoom in/out to show all the data points in one chart that you mentioned in an earlier post?
In a earlier post you mentioned that ChartDirector only plots data in memory and that programmers would have to read data points from the disk when a user scrolled. Will ChartDirector provide some callback functionality? Programmers would have to perform a seek and read on the file to obtain the necessary data points, put the data points into an array(s) and pass the array(s) to ChartDirector. Is this correct?
Philip |
| Re: Real-time chart: performance |
| Posted by Peter Kwan on Oct-15-2020 01:27 |
|
Hi,
The following is the WPF code and the next version "netchartdir.dll" that we use for internal testing. I have configured it to plot 3 lines each with up to 100M points. Initially, it will generate 3 random lines with 3 x 99M data points, then it will add 3 x 1000 points per second in real time. You can zoom out to display all 3 x 100M points, and zoom in so that individual points are visible and can see the chart updating in real time.
Note that it takes some time to generate 3 random lines with 99M each. In my computer, it takes around 15 seconds. So after pressing the "Plot Chart" button, there will be a delay of around 15 seconds for it to generate the data, and in my computer it needs 7G of RAM. Once it starts running, you can try the zooming and scrolling and it should be responsive.
Initially, it will display the last 20% of the data, which is around 3 x 20M points. As the real time update is only 3 x 1000 points, so you may not see the update in a chart with that many points. You would need to zoom in to the right to clearly see the real time update.
For memory usage, you need 8 bytes to store each value (each double value is eight bytes), and ChartDirector needs around 12 bytes to process and plot the value. So in total the system needs 20 bytes per value.
For the 3 x 100M lines, it has 3 y-coordinate series and 1 x-coordinate series. So there are 4 series, and the estimated memory requirement should be 20 x 4 x 100M = 8G. In actual testing, it takes around 7G in my computer. I suggest you try it only if your computer has that much memory, or you may need to modify the code to reduce the number of points.
https://www.advsofteng.com/support/megarealtimechartbeta.zip
Regards
Peter Kwan |
|
Forum Home
Search
Message List
Post Message