error registering maps outside mapset

[lucadelu]

Hi all,

if I try to register raster (but I think also vector and raster3d) maps outside my mapset I obtain error because the raster map is not found.

I think to have fixed it with the attached patch, please try it.

[lucadelu]

patch to fix register for maps outside mapset

[huhabla]

The current behavior is a design decision. Space time datasets accept only maps from the same mapset for registration. A new space time datasets will be dedicated to the current mapset. The current mapset is part of its temporal database identifier. Hence space time datasets from different mapsets can be stored in the same temporal database. This assures that mapsets can share the same temporal database.

Space time datasets are new datatypes in GRASS and should behave like raster or vector maps. Hence these datatypes are mapsets specific. It is possible to access space time datasets from different mapsets for reading, if the mapsets share the same temporal database. But it should not be possible to modify space time datasets from different mapsets.

As default the temporal database will be created in the PERMANENT directory, so that space time datasets and maps from different mapsets can be registered there. This allows the read only access to the datasets from different mapsets in the temporal database. To avoid this, the temporal database can be set mapset specific. Hence each mapset sees only their space time datasets and registered maps.

This decision was made to reduce confusion, to avoid unexpected behavior and to assure a convenient behavior of the temporal framework in the GRASS way.

Examples to support this decision:

If you register maps from a foreign mapset in a space time raster dataset and you remove the maps in the foreign mapset, then the temporal database might not be aware of these changes, pointing to maps that have not been unregistered. In all following operations the effected space time datasets will produce errors.

If you use "t.remove" with the "-rf" option, the provided space time datasets and its registered maps will be removed from the temporal and spatial database (g.remove will be called). This operation will fail if the registered maps are from different mapsets.

Attention when several mapsets share the same temporal database:

It is still a work in progress to grant only access to space time datasets and raster maps in the temporal database, that the user has permission to read --> configured using the GRASS mapset permission system. Hence you might see space time datasets in the output of "t.list" from different mapsets, to which you have no permission to access.

[annakrat]

I ran into problems when I tried to register maps from different mapsets: although t.register reported that it couldn't register the maps into the dataset (which is ok according to the desicion), it seemed to register them into the database. Because then, I switched to that mapset and tried to register the maps again with a new dataset but it failed because the maps were already registered (--o helped). I guess the maps should not have been registered at all?

[huhabla]

Replying to annakrat:

I ran into problems when I tried to register maps from different mapsets: although t.register reported that it couldn't register the maps into the dataset (which is ok according to the desicion), it seemed to register them into the database. Because then, I switched to that mapset and tried to register the maps again with a new dataset but it failed because the maps were already registered (--o helped). I guess the maps should not have been registered at all?

That is correct, this check was not implemented.

I have hopefully solved this issue with commit r58020. Many thanks for the feedback.

[lucadelu]

Replying to huhabla:

The current behavior is a design decision. Space time datasets accept only maps from the same mapset for registration. A new space time datasets will be dedicated to the current mapset. The current mapset is part of its temporal database identifier. Hence space time datasets from different mapsets can be stored in the same temporal database. This assures that mapsets can share the same temporal database.

Space time datasets are new datatypes in GRASS and should behave like raster or vector maps. Hence these datatypes are mapsets specific. It is possible to access space time datasets from different mapsets for reading, if the mapsets share the same temporal database. But it should not be possible to modify space time datasets from different mapsets.

As default the temporal database will be created in the PERMANENT directory, so that space time datasets and maps from different mapsets can be registered there. This allows the read only access to the datasets from different mapsets in the temporal database. To avoid this, the temporal database can be set mapset specific. Hence each mapset sees only their space time datasets and registered maps.

I'm not sure that this is the right way to follow because PERMANENT may be owned by a user differing from the user that is going to create a new temporal database in the location (think multi-user location!) so there should arise a write permission problem.

For example, in the location where our group members store MODIS LST data, when the user MarkusN previously connected his data to the temporal framework, but myself I also connect to it later (here MarkusN's data are hence in PERMANENT) I can do that. But when I try to actually create a new temporal dataset I obtain this error:

t.connect -d    
Default driver / database set to:
driver: sqlite
database: /grassdata/eu_laea/PERMANENT/tgis/sqlite.db

t.create out=lst_2003 title=lst_2003 description="LST 2003 dataset"
ERROR: Unable to execute transaction:
       INSERT INTO strds_base ( name ,creator ,mapset ,creation_time
       ,temporal_type ,id ,semantic_type ) VALUES ('lst_2003' ,'lucadelu'
       ,'lucadelu_test_temporal' ,'2013-10-17 10:33:20.660192' ,'absolute'
       ,'lst_2003@lucadelu_test_temporal' ,'mean') ;
       INSERT INTO strds_absolute_time ( start_time ,end_time ,granularity
       ,timezone ,id ,map_time ) VALUES (NULL ,NULL ,NULL ,NULL
       ,'lst_2003@lucadelu_test_temporal' ,NULL) ;
       INSERT INTO strds_spatial_extent ( north ,bottom ,west ,top ,proj
       ,east ,id ,south ) VALUES (NULL ,NULL ,NULL ,NULL ,'XY' ,NULL
       ,'lst_2003@lucadelu_test_temporal' ,NULL) ;
       INSERT INTO strds_metadata ( max_min ,description ,ewres_min ,title
       ,min_max ,nsres_min ,nsres_max ,number_of_maps ,raster_register
       ,command ,min_min ,ewres_max ,id ,max_max ) VALUES (NULL ,'LST 2003
       dataset' ,NULL ,'lst_2003' ,NULL ,NULL ,NULL ,NULL ,NULL ,'#
       2013-10-17 10:33:20
       t.create out="lst_2003" title="lst_2003"
       description="LST 2003 dataset"
       ' ,NULL ,NULL ,'lst_2003@lucadelu_test_temporal' ,NULL) ;
Traceback (most recent call last):
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/scripts/t.create", line 83, in <module>
    main()
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/scripts/t.create", line 79, in main
    semantic, None, grass.overwrite())
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/open.py", line 169, in open_new_space_time_dataset
    sp.insert(dbif)
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/abstract_dataset.py", line 395, in insert
    dbif.execute_transaction(statement)
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.py", line 628, in execute_transaction
    self.cursor.executescript(statement)
sqlite3.OperationalError: attempt to write a readonly database

It is naturally "read-only" because PERMANENT is owned by MarkusN and not by me.

Another example: I try to create a new temporal DB in another location which yet does not contain any temporal DB, but the owner of PERMANENT is again MarkusN, I get this error:

t.connect -d
Default driver / database set to:
driver: sqlite
database: /grassdata/patUTM32/PERMANENT/tgis/sqlite.db

t.create out=lst_2003 title=lst_2003 description="LST 2003 dataset"
Create temporal database: /grassdata/patUTM32/PERMANENT/tgis/sqlite.db
Traceback (most recent call last):
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/scripts/t.create", line 83, in <module>
    main()
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/scripts/t.create", line 76, in main
    tgis.init()
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.py", line 298, in init
    create_temporal_database(dbif, database)
  File "/home/lucadelu/compilati/grass7/dist.x86_64-unknown-linux-gnu/etc/python/grass/temporal/core.py", line 371, in create_temporal_database
    os.makedirs(tgis_dir)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/grassdata/patUTM32/PERMANENT/tgis'

So, t.create fails because I cannot write there due to the permission management, say, PERMANENT is not and should not be group writable.

This decision was made to reduce confusion, to avoid unexpected behavior and to assure a convenient behavior of the temporal framework in the GRASS way.

It is unexpected behaviour since it is contrary to the long-term known GRASS permission management.

Examples to support this decision:

If you register maps from a foreign mapset in a space time raster dataset and you remove the maps in the foreign mapset, then the temporal database might not be aware of these changes, pointing to maps that have not been unregistered. In all following operations the effected space time datasets will produce errors.

The same would apply to imagery groups, but this implementation is there for many years without real problems. Similar also for r.external and r.reclass.area. It is unavoidable that a user takes care of the data management. This should not limit the overall usability, hence more flexibility is better.

If you use "t.remove" with the "-rf" option, the provided space time datasets and its registered maps will be removed from the temporal and spatial database (g.remove will be called). This operation will fail if the registered maps are from different mapsets.

I understand to some extent your design decision but I see here a big problem for people with a shared data storage infrastructure like in our group (FEM PGIS).

For example, MarkusM is reconstructing MODIS LST data and he is at this point the owner of the mapset in which the results are stored. This mapset's size is of several TB and I have to work with the entire dataset (time series). We cannot change the owner of that mapset because it is continuously updated through an automated procedure.

Until now we cannot use temporal framework in our Unit, for my point of view this is really a limitation. Please consider to add the possibility to work with maps outside the working mapset and I'll be able to continue to test the temporal framework.

Thanks Luca

[huhabla]

Replying to lucadelu:

Replying to huhabla:

The current behavior is a design decision. Space time datasets accept only maps from the same mapset for registration. A new space time datasets will be dedicated to the current mapset. The current mapset is part of its temporal database identifier. Hence space time datasets from different mapsets can be stored in the same temporal database. This assures that mapsets can share the same temporal database.

Space time datasets are new datatypes in GRASS and should behave like raster or vector maps. Hence these datatypes are mapsets specific. It is possible to access space time datasets from different mapsets for reading, if the mapsets share the same temporal database. But it should not be possible to modify space time datasets from different mapsets.

As default the temporal database will be created in the PERMANENT directory, so that space time datasets and maps from different mapsets can be registered there. This allows the read only access to the datasets from different mapsets in the temporal database. To avoid this, the temporal database can be set mapset specific. Hence each mapset sees only their space time datasets and registered maps.

I'm not sure that this is the right way to follow because PERMANENT may be owned by a user differing from the user that is going to create a new temporal database in the location (think multi-user location!) so there should arise a write permission problem.

If you have a multi-user mapset environment, then you have to take care where to put the temporal database that should be accessed by several users from different mapsets. You can simply put the temporal database into a directory to which your group of users has write permissions? You can use PostgreSQL as temporal database, then you have to take care about user access permissions at the server as well. You don't depend on the default settings. However, i can change the default location of the temporal database to be mapset specific if everyone agrees.

But, changing this default behavior results in the drawback that the temporal database will be created for each mapset separately. Hence the user will not be able to see space time datasets from other mapsets. This is a principal problem of the SQL database Python interface. It is not possible to merge several sqlite databases to a single one and open them at once. This is AFAIK also not possible with the PostgreSQL backend. Different mapsets must share a single database to be able to see the space time datasets of each other. I would love to see a solution of multi-database management to fix this issue.

The temporal database approach is different from the vector database approach, where each vector has its own metadata file that describes the database connection for each layer. This approach can not be applied to the temporal database design, otherwise i need to create a database metadata connection file for each map and space time dataset. Hence if you want to access the temporal metadata information for 10000 maps, then you need to visit 10000 metadata files, connect 10000 times to the databases and read information for each map. Simple SQL where conditions would take hours to process. Sorting and many other features would not be available as SQL query. This approach would simply not work.

Hence, if the user wants to access space time datasets from different mapsets, then the temporal database path in each of these mapsets must point to single writable database location (sqlite -> path, postgresql -> server).

For example, in the location where our group members store MODIS LST data, when the user MarkusN previously connected his data to the temporal framework, but myself I also connect to it later (here MarkusN's data are hence in PERMANENT) I can do that. But when I try to actually create a new temporal dataset I obtain this error:

[snip]

It is naturally "read-only" because PERMANENT is owned by MarkusN and not by me.

This will fail even if the temporal frameworks allows that you can register maps from different locations in a space time dataset in your current location. In your case the registration process will modify the metadata of the raster maps in MarkusN location, it will try to set the time stamps for the maps. And because you have no write permission this will fail.

Space time datasets and their content are mapset specific, maps will be modified if you register them in the temporal database. It is IMHO an important functionality. It assures that you have access to the map time stamps using the C libraries.

Another example: I try to create a new temporal DB in another location which yet does not contain any temporal DB, but the owner of PERMANENT is again MarkusN, I get this error:

[snip]

So, t.create fails because I cannot write there due to the permission management, say, PERMANENT is not and should not be group writable.

This decision was made to reduce confusion, to avoid unexpected behavior and to assure a convenient behavior of the temporal framework in the GRASS way.

It is unexpected behaviour since it is contrary to the long-term known GRASS permission management.

I agree that putting the temporal database by default into the PERMANENT mapset might not be the most agreeable solution. But the design decision that space time datasets and their content are mapset specific follows the GRASS permission management.

Examples to support this decision:

If you register maps from a foreign mapset in a space time raster dataset and you remove the maps in the foreign mapset, then the temporal database might not be aware of these changes, pointing to maps that have not been unregistered. In all following operations the effected space time datasets will produce errors.

The same would apply to imagery groups, but this implementation is there for many years without real problems. Similar also for r.external and r.reclass.area. It is unavoidable that a user takes care of the data management. This should not limit the overall usability, hence more flexibility is better.

If you use "t.remove" with the "-rf" option, the provided space time datasets and its registered maps will be removed from the temporal and spatial database (g.remove will be called). This operation will fail if the registered maps are from different mapsets.

I understand to some extent your design decision but I see here a big problem for people with a shared data storage infrastructure like in our group (FEM PGIS).

For example, MarkusM is reconstructing MODIS LST data and he is at this point the owner of the mapset in which the results are stored. This mapset's size is of several TB and I have to work with the entire dataset (time series). We cannot change the owner of that mapset because it is continuously updated through an automated procedure.

In this case the automated procedure should update the space time datasets that manages the time series in the mapset of MarkusM? If the temporal database connection of the mapset from MarkusM and your mapset point to the same SQL database, then you can access the updated space time datasets.

Until now we cannot use temporal framework in our Unit, for my point of view this is really a limitation. Please consider to add the possibility to work with maps outside the working mapset and I'll be able to continue to test the temporal framework.

Thanks Luca

I hope i could explain the design decision a bit better. Of course, if you have arguments that make my completely obsolete, then we can change the design. But so far i don't see a reason to change this. :)

Best regards and many thanks for this discussion, we should have had it in the design phase of the temporal framework ;) Soeren

[huhabla]

I have discussed this topic off the list with Luca and investigated the problem in more detail. Luca suggested to register only maps from foreign mapsets that are not already in the temporal database and that have a timetsamp attached with r.timestamp, so that there is no need anymore to modify the timestamp in the spatial database metadata. Revision r58084 allows now to use existing time stamps in the registration process that were set with (r|r3|v).timestamp. This works only if the map is not already registered in the temporal database. However, timestamps will be rewritten in the spatial database metadata, since the registration method is generic and does not care where the timestamps originated from.

A more fundamental problem that occurs when registering maps from foreign mapsets is the following:

Space time datasets are defined by the spatio-temporal extents and the metadata of their maps. Hence, if you modify a map, then you will modify the space time datasets in which this map is registered. That's one reason why space time datasets are mapset specific.

Consider to have a work-mapset A and foreign mapset B. Now you register maps from mapset B in a space time dataset of mapset A. Now the owner of mapset B creates several space time datasets in his mapset and modifies some maps that are registered in the space time dataset of mapset A using the temporal framework (t.snap, t.shift, ...). The metadata modification of maps in mapset B will result in the modification of the space time dataset in mapset A. Hence the owner of mapset B can compromise the space time datasets of the owner of mapset A. This will violate the GRASS permission system, hence, its not allowed will be prevented by the temporal framework.

[lucadelu]

The initial problem is fixed (since long time).

Please reopen if needed.