Major performance problem when viewing big scenarios in high resolution

[Jan]

When viewing a scenario in wrowser, the updateFileList() method can take a long time consuming gigabytes of memory if the scenario is big and/or the scan output files have a high resolution. The reason for this is that to display the available probe names, all probe *contents* are also parsed and read into memory.

I propose this patch to fix it:

=== modified file 'wrowser/scenario/widgets.py'
--- wrowser/scenario/widgets.py 2009-05-28 13:48:58 +0000
+++ wrowser/scenario/widgets.py 2009-08-19 22:39:26 +0000
@@ -102,17 +102,20 @@
             import wrowser.Probe
             self.fileList.clear()

- if os.path.exists(self.workingDir + '/output/'):
- self.viewScenarioProbes = wrowser.Probe.readAllProbes(self.workingDir + '/output/')
- else:
- self.viewScenarioProbes = {}
+ self.viewScenarioProbes = {}
+ dirname = self.workingDir + '/output/'
+ if os.path.exists(dirname):
+ for ff in os.listdir(dirname):
+ filename = os.path.join(dirname, ff)
+ if os.path.isfile(filename):
+ for suffix in wrowser.Probe.TableProbe.fileNameSigs:
+ if suffix in filename:
+ self.viewScenarioProbes[ff] = "found"

- doNotShowThese = []
- for k,v in self.viewScenarioProbes.items():
- if v.probeType is not 'Table':
- doNotShowThese.append(k)
- for k in doNotShowThese:
- self.viewScenarioProbes.pop(k)
+ ### Here we basically only want to read the names of all table probes. The way this was
+ ### written meant that all probes (including all contents) were read into memory only to
+ ### throw away everything except the Table probenames. In the process, Gigabytes of memory
+ ### were used and the parsing took several minutes for bigger scenarios.

             self.fileList.addItems(self.viewScenarioProbes.keys())

[Maciej Muehleisen (mue-comnets)]

Thank you for that one. We use a similar approach inside Probes.py when dealing with PDF probes. We do not read the histogram data as long as it is not accessed. Maybe this could be the place to do the changes for TableProbes as done in getHistogram of the PDF probe.

endswith() instead of "in" should be used to compare the suffix.

My suggestion: Call TableProbe.readProbes(dirname) rather than readAllProbes, change TableProbe to not read data unless required.

[Marc Schinnenburg (marc.schinnenburg)]

Perhaps the following is even nicer (it will however descend into any available subdirectories):

self.viewScenarioProbes = {}
dirname = self.workingDir + '/output/'
for root, dirs, files in os.walk(dirname):
    for ff in files:
        ext = filename.split('_')[-1]
        sig = '_' + ext
        if sig in wrowser.Probe.TableProbe.fileNameSigs:
            self.viewScenarioProbes[ff] = "found"

TableProbe could have a static method 'iterProbes(dirname)' to just return a generator object with the filenames of TableProbes in a directory (with the above code essentially):

(static method of TableProbe)
def iterProbes(dirname):
    for root, dirs, files in os.walk(dirname):
        for ff in files:
            ext = filename.split('_')[-1]
            sig = '_' + ext
            if sig in fileNameSigs:
                yield os.path.join(root, ff)

a listProbe method would look as follows:

def listProbes(dirname):
    return [filename for filename in iterProbes(dirname)]

[Jan (jan-ellenbeck)]

Thanks for your suggestions, Maciej and Marc!

I played with it a little bit more and here is my second suggestion:

1) We do have to check if the fileNameSig is "in" the filename because it's not a real suffix as the ".m" is still attached to it, e.g. Scanner_RxPwr_BSID20_subBand1_max.m

2) I would like to raise the question whether we really want to display all the TableProbes in that list. Personally, I have never looked at the "trials" probe.

3) We don't need a dict for the viewScenarioProbes so I changed it to a list

4) I recommend sorting that list to make finding the right probe to plot easier

5) I like Marc's idea to walk the directory but I had to add the "filename = ..." line to it.

Cheers
Jan

Here is the updateFileList method:

        def updateFileList(self):
            import wrowser.Probe
            self.fileList.clear()

            self.viewScenarioProbes = []
            dirname = self.workingDir + '/output/'
            for root, dirs, files in os.walk(dirname):
                for ff in files:
                    filename = os.path.join(dirname, ff)
                    for infix in ['_mean', '_max']: #wrowser.Probe.TableProbe.fileNameSigs:
                        if infix in filename:
                            self.viewScenarioProbes.append(ff)

            self.fileList.addItems(sorted(self.viewScenarioProbes))