Opened 4 years ago

Closed 2 years ago

#2247 closed task (fixed)

Python and Jupyter

Reported by: darkblueb Owned by: osgeolive@…
Priority: normal Milestone: OSGeoLive15.0
Component: OSGeoLive Keywords: jupyter, python
Cc:

Description (last modified by darkblueb)

ubuntu focal 20.04 LTS LXQt

Install python and jupyter from the default Ubuntu repositories; note the results.

my first goal was to have a packaged setup that satisfies the pySAL mapclassify requirement, which is sensible and they wrote it recently . it is now [ numpy matplotlib descartes scipy sklearn pandas seaborn geopandas ]

​https://github.com/pysal/mapclassify

​https://github.com/pysal/mapclassify/blob/master/requirements.txt 

$ grep Get /home/shared/OSGEO_Misc/o14_misc/py_installs_raw_24jun2020.txt | sed -e 's/^Get.*focal\///' -e 's/ amd64 /\t/' -e 's/\]//' -e 's/ \[/\t/'  | sort -u

main	fonts-font-awesome all 5.0.10+really4.7.0~dfsg-1	514 kB
main	libamd2 amd64 1:5.7.1+dfsg-2	20.0 kB
main	libcamd2 amd64 1:5.7.1+dfsg-2	21.7 kB
main	libccolamd2 amd64 1:5.7.1+dfsg-2	23.2 kB
main	libcholmod3 amd64 1:5.7.1+dfsg-2	308 kB
main	libmetis5 amd64 5.1.0.dfsg-5	169 kB
main	libumfpack5 amd64 1:5.7.1+dfsg-2	229 kB
main	python3-click all 7.0-3	64.8 kB
main	python3-colorama all 0.4.3-1build1	23.9 kB
main	python3-defusedxml all 0.6.0-2	37.0 kB
main	python3-munch all 2.3.2-2	7,984 B
main	python3-packaging all 20.3-1	26.8 kB
main	python3-psutil amd64 5.5.1-1ubuntu4	141 kB                 
main	python3-wcwidth all 0.1.8+dfsg1-3	17.4 kB
main	python3-wrapt amd64 1.11.2-2build1	30.6 kB
universe	fonts-glyphicons-halflings all 1.009~3.4.1+dfsg-1	117 kB
universe	ipython3 all 7.13.0-1	4,988 B
universe	jupyter all 4.6.3-3	3,368 B
universe	jupyter-client all 6.1.2-1	5,812 B
universe	jupyter-console all 6.0.0-2	2,820 B
universe	jupyter-core all 4.6.3-3	5,196 B
universe	jupyter-nbconvert all 5.6.1-1	4,284 B
universe	jupyter-nbextension-jupyter-js-widgets all 6.0.0-6	214 kB
universe	jupyter-nbformat all 5.0.4-1	3,228 B
universe	jupyter-notebook all 6.0.3-2	5,952 B                   
universe	jupyter-qtconsole all 4.6.0-1	5,328 B                  
universe	libblosc1 amd64 1.17.1+ds1-1	32.2 kB
universe	libdsdp-5.8gf amd64 5.8-9.4build1	184 kB
universe	libglpk40 amd64 4.65-2	378 kB
universe	libjs-backbone all 1.3.3~dfsg-5	176 kB
universe	libjs-bootstrap all 3.4.1+dfsg-1	124 kB
universe	libjs-bootstrap-tour all 0.12.0+dfsg-2	20.4 kB
universe	libjs-codemirror all 5.51.0-1	609 kB
universe	libjs-es6-promise all 4.2.8-6	14.7 kB                  
universe	libjs-jed all 1.1.1-1	12.2 kB                          
universe	libjs-jquery-typeahead all 2.11.0+dfsg1-1	47.0 kB      
universe	libjs-marked all 0.5.1+dfsg-1	17.7 kB                  
universe	libjs-moment all 2.24.0+ds-2	126 kB                    
universe	libjs-requirejs all 2.3.6-1	29.9 kB                    
universe	libjs-requirejs-text all 2.0.12-1	8,894 B              
universe	libjs-text-encoding all 0.7.0-1	104 kB                 
universe	libjs-xterm all 3.8.1-3	56.0 kB                        
universe	liblbfgsb0 amd64 3.0+dfsg.3-7build1	29.5 kB
universe	libspatialindex-c6 amd64 1.9.3-1build1	58.3 kB
universe	libspatialindex-dev amd64 1.9.3-1build1	16.0 kB        
universe	pandoc amd64 2.5-3build2	15.4 MB                       
universe	pandoc-data all 2.5-3build2	76.0 kB                    
universe	python3-affine all 2.3.0-1	13.8 kB
universe	python3-atomicwrites all 1.1.5-2build1	7,016 B
universe	python3-backcall all 0.1.0-2	11.2 kB
universe	python3-bleach all 3.1.1-1	108 kB
universe	python3-blosc amd64 1.7.0+ds1-2ubuntu2	22.0 kB
universe	python3-click-plugins all 1.1.1-2	9,704 B
universe	python3-cligj all 0.5.0-2	7,708 B
universe	python3-cloudpickle all 1.3.0-1	23.4 kB
universe	python3-cvxopt amd64 1.2.3+dfsg-2build1	259 kB
universe	python3-dask all 2.8.1+dfsg-0.4	532 kB
universe	python3-deprecated all 1.2.7-1	8,432 B
universe	python3-descartes all 1.1.0-3	6,108 B
universe	python3-et-xmlfile all 1.0.1-2.1	9,224 B
universe	python3-fiona amd64 1.8.13-1build3	469 kB
universe	python3-fsspec all 0.6.1-0.1	45.7 kB
universe	python3-geographiclib amd64 1.50.1-1build1	32.2 kB
universe	python3-geojson all 2.5.0-2	13.3 kB                    
universe	python3-geopandas all 0.7.0-1	861 kB                   
universe	python3-geopy all 1.20.0-1	53.4 kB                     
universe	python3-ipykernel all 5.2.0-1	76.8 kB
universe	python3-ipython all 7.13.0-1	494 kB
universe	python3-ipywidgets all 6.0.0-6	31.0 kB                 
universe	python3-jdcal all 1.0-1.3	7,944 B
universe	python3-jedi all 0.15.2-1	502 kB
universe	python3-joblib all 0.14.0-3	192 kB                     
universe	python3-jupyter-client all 6.1.2-1	69.5 kB
universe	python3-jupyter-console all 6.0.0-2	18.2 kB
universe	python3-locket all 0.2.0-1	4,562 B
universe	python3-mistune all 0.8.4-2	14.3 kB
universe	python3-nbconvert all 5.6.1-1	102 kB
universe	python3-nose all 1.3.7-5	116 kB                        
universe	python3-notebook all 6.0.3-2	1,094 kB                  
universe	python3-numexpr amd64 2.7.1-1build1	133 kB
universe	python3-openpyxl all 3.0.3-1	144 kB
universe	python3-pandas all 0.25.3+dfsg-7	1,968 kB              
universe	python3-pandas-lib amd64 0.25.3+dfsg-7	4,628 kB
universe	python3-pandocfilters all 1.4.2-2	18.9 kB
universe	python3-parso all 0.5.2-1ubuntu1	62.8 kB
universe	python3-partd all 1.0.0-1	14.8 kB                      
universe	python3-patsy all 0.5.1-1	168 kB
universe	python3-pickleshare all 0.7.5-2	7,560 B
universe	python3-pluggy all 0.13.0-2	18.4 kB                    
universe	python3-prometheus-client all 0.7.1-1.1	33.3 kB
universe	python3-prompt-toolkit all 2.0.10-2	220 kB
universe	python3-py all 1.8.1-1	65.4 kB                         
universe	python3-pysal all 2.1.0-1	14.3 MB                      
universe	python3-pytest all 4.6.9-1	178 kB                      
universe	python3-qtconsole all 4.6.0-1	87.6 kB                  
universe	python3-rasterio amd64 1.1.3-1	997 kB
universe	python3-rtree all 0.9.4-1	54.0 kB                      
universe	python3-scipy amd64 1.3.3-3build1	11.1 MB              
universe	python3-seaborn all 0.10.0-1	155 kB
universe	python3-send2trash all 1.5.0-2	9,704 B
universe	python3-shapely amd64 1.7.0-1build1	248 kB             
universe	python3-sklearn all 0.22.2.post1+dfsg-5	1,773 kB       
universe	python3-sklearn-lib amd64 0.22.2.post1+dfsg-5	2,358 kB 
universe	python3-snuggs all 1.4.7-1	6,744 B
universe	python3-statsmodels all 0.11.1-1	3,939 kB              
universe	python3-statsmodels-lib amd64 0.11.1-1	1,574 kB        
universe	python3-tables all 3.6.1-2build1	333 kB                
universe	python3-tables-lib amd64 3.6.1-2build1	501 kB          
universe	python3-terminado all 0.8.2-3	9,312 B
universe	python3-testpath all 0.4.4+dfsg-1	9,040 B
universe	python3-toolz all 0.9.0-1	42.0 kB
universe	python3-tornado amd64 6.0.3+really5.1.1-3	304 kB
universe	python3-tqdm all 4.30.0-1	68.4 kB                      
universe	python3-widgetsnbextension all 6.0.0-6	4,888 B         
universe	python3-xlrd all 1.1.0-5	90.8 kB                       
universe	python3-xlwt all 1.3.0-3	83.7 kB                       
universe	python3-zmq amd64 18.1.1-3	294 kB                      
universe	python-tables-data all 3.6.1-2build1	45.7 kB
universe	rasterio all 1.1.3-1	8,868 B

Change History (14)

comment:1 by darkblueb, 4 years ago

Description: modified (diff)

comment:2 by darkblueb, 4 years ago

hmm regarding pandas 

  pandas 0.25.3 (2019-10-31)
  pandas 1.0.0 (2020-01-29)
  Latest version:  pandas 1.0.5  (Jun 17, 2020)

  installed:
   pd.__version__  =>  '0.25.3'

comment:3 by darkblueb, 4 years ago 

$ jupyter troubleshoot
/usr/bin/python3: No module named pip
$PATH:
	/usr/local/sbin
	/usr/local/bin
	/usr/sbin
	/usr/bin
	/sbin
	/bin
	/usr/games
	/usr/local/games
	/snap/bin

sys.path:
	/usr/bin
	/usr/lib/python38.zip
	/usr/lib/python3.8
	/usr/lib/python3.8/lib-dynload
	/usr/local/lib/python3.8/dist-packages
	/usr/lib/python3/dist-packages

sys.executable:
	/usr/bin/python3

sys.version:
	3.8.2 (default, Apr 27 2020, 15:53:34) 
	[GCC 9.3.0]

platform.platform():
	Linux-5.4.0-39-generic-x86_64-with-glibc2.29

which -a jupyter:
	/usr/bin/jupyter
	/bin/jupyter
Version 0, edited 4 years ago by darkblueb (next)

comment:4 by darkblueb, 4 years ago

stable, first pass at python3 installs using Ubuntu packaging 

gdal   numpy   rasterio
fiona   shapely

matplotlib   descartes   seaborn

dask

owslib
psycopg2   sqlalchemy   sqlalchemy-ext   geoalchemy2
statsmodels   statsmodels-lib

pandas   pandas-lib
geopandas   geojson
pysal

rtree   geographiclib
scipy   sklearn   sklearn-lib
geopy

jupyter-client   jupyter-console   jupyter-core
nbconvert   nbformat   notebook
ipykernel   ipython-genutils   ipython3   ipywidgets

qgis   qgis-common

affine   apport
apt   aptdaemon
atomicwrites
attr   backcall   bleach
blinker   blosc   bs4
certifi   cffi-backend
chardet   click   click-plugins   cligj
cloudpickle   colorama
commandnotfound   cryptography
cups   cupshelpers
cvxopt   cycler
dateutil   dbus   debconf   debian   decorator
defer   defusedxml   deprecated   dev
distro   distro-info   distupgrade
distutils   entrypoints   examples
fsspec   future   gi
html5lib   httplib2
idna   importlib-metadata   jdcal   jedi
jinja2   joblib
jsonschema   jwt
keyring   kiwisolver
launchpadlib   lib2to3
locket   lxml
markupsafe   minimal
mistune   more-itertools   munch   nacl
netifaces   nose
numexpr   oauthlib   olefile
opengl   openpyxl
packaging   pandocfilters   parso   partd
patsy   pexpect   pickleshare   pil   pip
pkg-resources   plotly   pluggy
problem-report   prometheus-client   prompt-toolkit
psutil   ptyprocess
pygments   py   pymacaroons   pyparsing    pyproj
pyqt5   pyqt5.qsci   pyqt5.qtsql   pyqt5.qtsvg   pyqt5.qtwebkit
pyrsistent   pytest   qtconsole
renderpm   reportlab-accel   reportlab
requests   requests-unixsocket   retrying
secretstorage   send2trash
setuptools   simplejson
sip   six   snuggs   software-properties   soupsieve
systemd   tables   tables-lib
terminado   testpath
toolz   tornado   tqdm   traitlets   tz
uno   update-manager   urllib3
wadllib   wcwidth   webencodings   wheel
widgetsnbextension   wrapt   xapian   xdg   xkit   xlrd   xlwt
yaml   zipp   zmq
Last edited 4 years ago by darkblueb (previous) (diff)

comment:5 by darkblueb, 4 years ago

Homepage: ​https://dask.org -- Sources: ​https://github.com/dask/dask -- Blog: ​https://blog.dask.org 

Package: python3-dask
Version: 2.8.1+dfsg-0.4
Section: universe/python
 -
Recommends: 
   python3-cloudpickle,  python3-numpy,  python3-pandas,  python3-partd, 
   python3-requests
Suggests: 
   ipython,  python-dask-doc,  python3-bcolz,  python3-blosc,  python3-boto, 
   python3-distributed (>= 1.21),  python3-graphviz,  python3-h5py,  python3-psutil, 
   python3-scipy,  python3-skimage,  python3-sklearn,  python3-sqlalchemy, 
   python3-tables
Last edited 4 years ago by darkblueb (previous) (diff)

comment:7 by darkblueb, 4 years ago 

Ubuntu/Debian/GNU Linux System Python v3.8
Linux-5.4.0-42-generic-x86_64-with-glibc2.29

SciPy  SkLearn  Matplotlib  installed via system .deb
# -- import mapclassify ------------------------------

import 'platform' #
import 'multiprocessing' #
import 'numpy' #  1.17.4 
import 'scipy.spatial.kdtree' #  1.3.3
import 'scipy.spatial.ckdtree' #
import 'scipy.spatial.qhull' #
import 'scipy.spatial._voronoi' #
import 'scipy.spatial._distance_wrap' #
import 'scipy.spatial._hausdorff' #
import 'scipy.spatial.distance' #
import 'scipy.spatial._spherical_voronoi' #
import 'scipy.spatial.transform.rotation' #
import 'scipy.spatial.transform._rotation_spline' #
import 'scipy.spatial.transform' #
import 'scipy.spatial' #
import 'scipy.optimize._shgo_lib.triangulation' #
import 'scipy.optimize._dual_annealing' #
import 'scipy.optimize' #
import 'scipy.stats' #
import 'sklearn._config' #
import 'sklearn._distributor_init' #
import 'sysconfig' #
import 'json' #
import 'zipfile' #
import 'xml' #
import 'xml.parsers.expat' #
import 'joblib' #
import 'sklearn.utils._joblib' #
import 'sklearn.exceptions' #
import 'distutils' #
import 'sklearn.externals' #
import 'sklearn.neighbors._dist_metrics' #
import 'sklearn.neighbors._ball_tree' #
import 'sklearn.neighbors._kd_tree' #
import 'sklearn.neighbors._base' #
import 'sklearn.neighbors._unsupervised' #
import 'sklearn.neighbors._graph' #
import 'sklearn.neighbors._classification' #
import 'sklearn.neighbors._regression' #
import 'sklearn.neighbors._nearest_centroid' #
import 'sklearn.neighbors._kde' #
import 'sklearn.neighbors._lof' #
import 'sklearn.tree._utils' #
import 'sklearn.tree._tree' #
import 'sklearn.tree._splitter' #
import 'sklearn.tree._criterion' #
import 'sklearn.tree._classes' #
import 'sklearn.tree._reingold_tilford' #
import 'sklearn.tree._export' #
import 'sklearn.tree' #
import 'sklearn.neighbors._quad_tree' #
import 'sklearn.manifold' #
import 'sklearn.cluster._k_means_fast' #
import 'sklearn.cluster._k_means_elkan' #
import 'sklearn.cluster._kmeans' #
import 'sklearn.cluster._spectral' #
import 'sklearn.cluster._mean_shift' #
import 'sklearn.cluster._affinity_propagation' #
import 'sklearn.utils._fast_dict' #
import 'sklearn.cluster._hierarchical_fast' #
import 'sklearn.cluster._feature_agglomeration' #
import 'sklearn.cluster._agglomerative' #
import 'sklearn.cluster._dbscan_inner' #
import 'sklearn.cluster._dbscan' #
import 'sklearn.cluster._optics' #
import 'sklearn.cluster._bicluster' #
import 'sklearn.cluster._birch' #
import 'sklearn.cluster' #  0.22.2.post1
import 'mapclassify.classifiers' #
import 'mapclassify.pooling' #
import 'mapclassify.greedy' #
import 'mapclassify' # 2.3.0

##   pandas  shapely  installed by system .deb
##-- import geopandas --------------------------------
import 'six' #
import 'pandas._config.config' #  0.25.3
import 'pandas._libs' #
import 'pandas.core' #
import 'pandas.io' #
import 'pandas.io.formats' #
import 'pandas.io.formats.console' #
import 'unicodedata' #
import 'csv' #
import 'gzip' #
import 'http' #
import 'pandas.io.common' #
import 'pandas.io.formats.format' #
import 'matplotlib' #  3.1.2
import 'matplotlib.cm' #
import 'pandas.plotting._matplotlib.style' #
import 'matplotlib.font_manager' #
import 'pandas.api' #
import 'pandas.io.spss' #
import 'pandas.io.sql' #
import 'pandas.io.stata' #
import 'pandas.io.api' #
import 'pandas' #  0.25.3
import 'shapely' #  1.7.0
import 'rtree' #
import 'pyproj.compat' #
import 'pyproj' #  2.5.0 
import 'shapely.geos' # libgeos-3.8.0
import 'shapely.speedups._speedups' #
import 'shapely.prepared' #
import 'geopandas.io' #
import 'geopandas.geodataframe' #
import 'fiona' #
import 'geopandas.io.file' #
import 'geopandas.io.arrow' #
import 'geopandas.io.sql' #
import 'geopandas._version' #
import 'geopandas' # 0.8.0+5.g89908f8
Last edited 4 years ago by darkblueb (previous) (diff)

comment:9 by darkblueb, 4 years ago

STAC support August 2020

Tutorial for pySTAC ​https://pystac.readthedocs.io/en/latest/tutorials/pystac-spacenet-tutorial.html

​https://github.com/stac-utils

note: boto3 and s3fs are supplied as dot-deb, but specific to Amazon S3

compare & contrast

​https://github.com/cogeotiff/rio-cogeo

Last edited 4 years ago by darkblueb (previous) (diff)

comment:10 by darkblueb, 4 years ago

folium install via pip3 

user@ol14b187:~$  pip3 install --user --upgrade-strategy=only-if-needed folium
Collecting folium
  Downloading folium-0.11.0-py2.py3-none-any.whl (93 kB)
     |████████████████████████████████| 93 kB 247 kB/s 
Requirement already satisfied: branca>=0.3.0 in /usr/lib/python3/dist-packages (from folium) (0.3.1)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from folium) (1.17.4)
Requirement already satisfied: jinja2>=2.9 in /usr/lib/python3/dist-packages (from folium) (2.10.1)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from folium) (2.22.0)
Installing collected packages: folium
Successfully installed folium-0.11.0
Last edited 4 years ago by darkblueb (previous) (diff)

comment:11 by darkblueb, 4 years ago

joris and crew use their own maptile utility -- ​contextily 

user@ol14b187:$ pip3 install --user contextily
Collecting contextily
  Downloading contextily-1.0.1-py3-none-any.whl (23 kB)
Requirement already satisfied: pillow in /usr/lib/python3/dist-packages (from contextily) (7.0.0)
Collecting geopy
  Downloading geopy-2.0.0-py3-none-any.whl (111 kB)
     |████████████████████████████████| 111 kB 381 kB/s 
Requirement already satisfied: rasterio in /usr/lib/python3/dist-packages (from contextily) (1.1.8)
Requirement already satisfied: joblib in /usr/lib/python3/dist-packages (from contextily) (0.14.0)
Requirement already satisfied: matplotlib in /usr/lib/python3/dist-packages (from contextily) (3.1.2)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from contextily) (2.22.0)
Requirement already satisfied: mercantile in /usr/lib/python3/dist-packages (from contextily) (1.1.6)
Requirement already satisfied: geographiclib<2,>=1.49 in /usr/lib/python3/dist-packages (from geopy->contextily) (1.50)
Installing collected packages: geopy, contextily
Successfully installed contextily-1.0.1 geopy-2.0.0
Last edited 4 years ago by darkblueb (previous) (diff)

comment:12 by darkblueb, 4 years ago

Jupyter info on build ol14b187 

user@ol14b187:~/jupyter$ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.0.3
qtconsole        : not installed
ipython          : 7.13.0
ipykernel        : 5.2.0
jupyter client   : 6.1.2
jupyter lab      : not installed
nbconvert        : 5.6.1
ipywidgets       : 6.0.0
nbformat         : 5.0.4
traitlets        : 4.3.3

comment:13 by kalxas, 3 years ago

Milestone: OSGeoLive14.0 → OSGeoLive15.0

Ticket retargeted after milestone closed

comment:14 by darkblueb, 2 years ago

Resolution: → fixed
Status: new → closed

closed for ol15 new OS

Note: See TracTickets for help on using tickets.