Error selecting more than 1000 features from SQLite FeatureSource

[gBecker]

When trying to select more than 1000 Features from a SQLite FeatureSource an error occured and is logged in Error.log:

 StackTrace:
  - MgMappingUtil.StylizeLayers() line 893 file c:\working\build_area\mapguide\3.0.0\x64\mgdev\server\src\services\mapping\MappingUtil.cpp	
<2016-10-25T12:14:08> 	4876	Ajax Viewer	::1	Administrator
 Error: An exception occurred in FDO component.
        Error occurred in Feature Source (Library://FFM/CSOGIS/Daten/Sqlite/basisgis.FeatureSource): Expression tree is too large (maximum depth 1000) (Cause: , Root Cause: Expression tree is too large (maximum depth 1000))
 StackTrace:
  - MgFeatureServiceHandler.ProcessOperation() line 83 file c:\working\build_area\mapguide\3.0.0\x64\mgdev\server\src\services\feature\FeatureServiceHandler.cpp
  - MgOpSelectFeatures.Execute() line 142 file c:\working\build_area\mapguide\3.0.0\x64\mgdev\server\src\services\feature\OpSelectFeatures.cpp
  - MgServerFeatureService.SelectFeatures(Resource=Library://FFM/CSOGIS/Daten/Sqlite/basisgis.FeatureSource	ClassName=Default:KS31101_Gebaeude	
Options={{ClassProps={{KGF_WERT}{Geometrie}}}{Operator=1}{Filter=(PRIMARYINDEX=1928) OR (PRIMARYINDEX=2024) OR (PRIMARYINDEX=2190) OR (PRIMARYINDEX=2442) OR (PRIMARYINDEX=3281) OR 
...
(PRIMARYINDEX=19556) OR (PRIMARYINDEX=19568) OR (PRIMARYINDEX=19717) OR (PRIMARYINDEX=19918) OR (PRIMARYINDEX=19929) OR (PRIMARYINDE<2016-10-25T12:14:09> 	13036	Ajax Viewer	::1	Administrator
 Warning: An exception occurred in FDO component.
        String does not represent a valid filter. 
 StackTrace:
  - MgStylizationUtil.ExceptionTrap() line 751 file GeometryAdapter.cpp	
<2016-10-25T12:14:09> 	13036	Ajax Viewer	::1	Administrator
 Warning: An exception occurred in FDO component.
        String does not represent a valid filter.

This error leads to an terminated MapGuideAppPool. You have to restart the application pool.

Application Error in Windows application.Log:

Name der fehlerhaften Anwendung: w3wp.exe, Version: 10.0.14393.0, Zeitstempel: 0x57899b8a
Name des fehlerhaften Moduls: isapi_MapAgent.dll, Version: 3.0.0.8701, Zeitstempel: 0x559fd022
Ausnahmecode: 0xc0000409
Fehleroffset: 0x0000000000009060
ID des fehlerhaften Prozesses: 0x3214
Startzeit der fehlerhaften Anwendung: 0x01d22ea886d41d6f
Pfad der fehlerhaften Anwendung: c:\windows\system32\inetsrv\w3wp.exe
Pfad des fehlerhaften Moduls: \\?\C:\Program Files\OSGeo\MapGuide3.0\Web\www\mapagent\isapi_MapAgent.dll
Berichtskennung: 345f7d37-abe7-4ef2-bb64-1e745f1d4e02

[jng]

Is this a feature query from a MgFeatureService.SelectFeatures API call or from a map tool selection?

Is your filter literally 1000s of PRIMARYKEY=value conditions OR'd together, or is that from a MgSelection?

I ask because I know MgSelection has the ability to sub-divide filters into manageable parts, but I'm not sure that SelectFeatures has the same ability.

[gBecker]

The feature query is from the default selection tool (rectangle selection) of the AjaxViewer. Apparently 1000 "PRIMARYKEY=value" conditions are concatenated by "OR". The error message in the logfile is truncated at around 200 conditions.

[jng]

Okay, I see where this is happening (building the attributes of the QUERYMAPFEATURES response).

We need to sub-divide the selection set filter to a reasonable number and incrementally write out the attributes from all the sub-divided feature readers.

[jng]

Firstly, does this happen with MGOS 3.1 as well?

Secondly, if the above answer is yes, I'm going to build and attach a patched MgMapGuideCommon.dll for MGOS 3.1 64-bit that incorporates the attached source code changes. Could you check if this dll fixes the issue in 3.1? If it does, I'll backport this fix for 3.0 and 2.6 as well.

[jng]

Patched MgMapGuideCommon.dll for MGOS 3.1 64-bit

[gBecker]

In MGOS 3.1 the selection issue doesn't happen (without the patched MgMapGuideCommon.dll). I can select up to approxamately 15.000 features. When selecting more features at once, then there seems to be another limit, but luckily without terminating the IIS AppliationPool. The selection layer isn't updated anymore.

But, you can select even more features while pressing the Ctrl-Button and selecting a smaller amount of features. When selecting approxemately 25.000 features another odd behaviour occurs. The selection layer only shows the last added features while the number of selected features in the statusbar increaeses.

These behaviours aren't restricted to SQLite FeatureSources. This also belongs to Shapes oder SDF FeatureSources. This is happening in MGOS 3.0 and 3.1.

I believe that this has nothing to do with the initial issue. Maybe this is more a problem of the SQLite FDO Provider (4.0) in MGOS 3.0 and was fixed in MGOS 3.1?

[jng]

Hmmm, I see no changes in the SQLite FDO provider that could indicate a fix at the FDO provider level. I'll make a patched MgMapGuideCommon.dll for 3.0 and attach that here.

As for the strange selection behaviour when you hold the CTRL key, I backported the selection appending fix (r9036). I just haven't made a 3.0.1 release yet that incorporates this fix.

[jng]

Patched MgMapGuideCommon.dll for MGOS 3.0 64-bit

[jng]

Can you try this MgMapGuideCommon.dll for your 3.0 install and let me know how it goes?

[gBecker]

Unfortunately, there is no difference. Selecting the 1,000th feature makes the application pool stop working.