netCDF et al

[darkblueb]

There has been some discussion about netCDF, python-netCDF and versions.. This ticket is to clarify and record information relating to netCDF versions.

[darkblueb]

there is a python package called ScientificPython which contains IO.NetCDF. A revision date shows 2008. It is present on the disk but not referenced.

user@live-79a-042:~/gisvm/bin$ locate NetCDF.py
/usr/lib/python2.7/dist-packages/Scientific/IO/NetCDF.py
/usr/lib/python2.7/dist-packages/Scientific/IO/NetCDF.pyc
/usr/share/pyshared/Scientific/IO/NetCDF.py

[darkblueb]

Two .so libs present. One is 2.3M, the other is 1.1M. Both have very similar lists of linked libs.

/usr/local/lib/python2.7/dist-packages/netCDF4.so
/usr/lib/libnetcdf.so.6

user@live-79a-042:~$ ldd /usr/local/lib/python2.7/dist-packages/netCDF4.so | sort
	libasn1.so.8 => /usr/lib/i386-linux-gnu/libasn1.so.8 (0x00ab8000)
	libcom_err.so.2 => /lib/i386-linux-gnu/libcom_err.so.2 (0x004c8000)
	libcrypt.so.1 => /lib/i386-linux-gnu/libcrypt.so.1 (0x009e6000)
	libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0x00110000)
	libcurl-gnutls.so.4 => /usr/lib/i386-linux-gnu/libcurl-gnutls.so.4 (0x002ba000)
	libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0x00730000)
	libgcrypt.so.11 => /lib/i386-linux-gnu/libgcrypt.so.11 (0x006a6000)
	libgnutls.so.26 => /usr/lib/i386-linux-gnu/libgnutls.so.26 (0x003a6000)
	libgpg-error.so.0 => /lib/i386-linux-gnu/libgpg-error.so.0 (0x0072b000)
	libgssapi_krb5.so.2 => /usr/lib/i386-linux-gnu/libgssapi_krb5.so.2 (0x004e2000)
	libgssapi.so.3 => /usr/lib/i386-linux-gnu/libgssapi.so.3 (0x0053c000)
	libhcrypto.so.4 => /usr/lib/i386-linux-gnu/libhcrypto.so.4 (0x009b1000)
	libhdf5_hl.so.6 => /usr/lib/libhdf5_hl.so.6 (0x00312000)
	libhdf5.so.6 => /usr/lib/libhdf5.so.6 (0x00c02000)
	libheimbase.so.1 => /usr/lib/i386-linux-gnu/libheimbase.so.1 (0x0088d000)
	libheimntlm.so.0 => /usr/lib/i386-linux-gnu/libheimntlm.so.0 (0x00735000)
	libhx509.so.5 => /usr/lib/i386-linux-gnu/libhx509.so.5 (0x00bb2000)
	libidn.so.11 => /usr/lib/i386-linux-gnu/libidn.so.11 (0x0074c000)
	libk5crypto.so.3 => /usr/lib/i386-linux-gnu/libk5crypto.so.3 (0x0084f000)
	libkeyutils.so.1 => /lib/i386-linux-gnu/libkeyutils.so.1 (0x0073d000)
	libkrb5.so.26 => /usr/lib/i386-linux-gnu/libkrb5.so.26 (0x00a35000)
	libkrb5.so.3 => /usr/lib/i386-linux-gnu/libkrb5.so.3 (0x00780000)
	libkrb5support.so.0 => /usr/lib/i386-linux-gnu/libkrb5support.so.0 (0x00fbb000)
	liblber-2.4.so.2 => /usr/lib/i386-linux-gnu/liblber-2.4.so.2 (0x00345000)
	libldap_r-2.4.so.2 => /usr/lib/i386-linux-gnu/libldap_r-2.4.so.2 (0x00354000)
	/lib/ld-linux.so.2 (0x0089d000)
	libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0x00484000)
	libnetcdf.so.6 => /usr/lib/libnetcdf.so.6 (0x0058e000)
	libp11-kit.so.0 => /usr/lib/i386-linux-gnu/libp11-kit.so.0 (0x00579000)
	libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0x00fcb000)
	libresolv.so.2 => /lib/i386-linux-gnu/libresolv.so.2 (0x004b0000)
	libroken.so.18 => /usr/lib/i386-linux-gnu/libroken.so.18 (0x00877000)
	librtmp.so.0 => /usr/lib/i386-linux-gnu/librtmp.so.0 (0x0046a000)
	librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0x00a2c000)
	libsasl2.so.2 => /usr/lib/i386-linux-gnu/libsasl2.so.2 (0x00520000)
	libsqlite3.so.0 => /usr/lib/i386-linux-gnu/libsqlite3.so.0 (0x00fe6000)
	libtasn1.so.3 => /usr/lib/i386-linux-gnu/libtasn1.so.3 (0x004cd000)
	libwind.so.0 => /usr/lib/i386-linux-gnu/libwind.so.0 (0x008bf000)
	libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0x00b9c000)
	linux-gate.so.1 =>  (0x00a2b000)
user@live-79a-042:~$ ldd /usr/lib/libnetcdf.so.6 | sort
	libasn1.so.8 => /usr/lib/i386-linux-gnu/libasn1.so.8 (0x00e55000)
	libcom_err.so.2 => /lib/i386-linux-gnu/libcom_err.so.2 (0x00110000)
	libcrypt.so.1 => /lib/i386-linux-gnu/libcrypt.so.1 (0x00ccc000)
	libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0x007de000)
	libcurl-gnutls.so.4 => /usr/lib/i386-linux-gnu/libcurl-gnutls.so.4 (0x00c36000)
	libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0x0011a000)
	libgcrypt.so.11 => /lib/i386-linux-gnu/libgcrypt.so.11 (0x00a4c000)
	libgnutls.so.26 => /usr/lib/i386-linux-gnu/libgnutls.so.26 (0x00988000)
	libgpg-error.so.0 => /lib/i386-linux-gnu/libgpg-error.so.0 (0x00115000)
	libgssapi_krb5.so.2 => /usr/lib/i386-linux-gnu/libgssapi_krb5.so.2 (0x001c8000)
	libgssapi.so.3 => /usr/lib/i386-linux-gnu/libgssapi.so.3 (0x00d0a000)
	libhcrypto.so.4 => /usr/lib/i386-linux-gnu/libhcrypto.so.4 (0x00be0000)
	libhdf5_hl.so.6 => /usr/lib/libhdf5_hl.so.6 (0x00d84000)
	libhdf5.so.6 => /usr/lib/libhdf5.so.6 (0x003a7000)
	libheimbase.so.1 => /usr/lib/i386-linux-gnu/libheimbase.so.1 (0x00789000)
	libheimntlm.so.0 => /usr/lib/i386-linux-gnu/libheimntlm.so.0 (0x00281000)
	libhx509.so.5 => /usr/lib/i386-linux-gnu/libhx509.so.5 (0x00f20000)
	libidn.so.11 => /usr/lib/i386-linux-gnu/libidn.so.11 (0x00142000)
	libk5crypto.so.3 => /usr/lib/i386-linux-gnu/libk5crypto.so.3 (0x00bb8000)
	libkeyutils.so.1 => /lib/i386-linux-gnu/libkeyutils.so.1 (0x00f1c000)
	libkrb5.so.26 => /usr/lib/i386-linux-gnu/libkrb5.so.26 (0x00db7000)
	libkrb5.so.3 => /usr/lib/i386-linux-gnu/libkrb5.so.3 (0x00ad1000)
	libkrb5support.so.0 => /usr/lib/i386-linux-gnu/libkrb5support.so.0 (0x00cc3000)
	liblber-2.4.so.2 => /usr/lib/i386-linux-gnu/liblber-2.4.so.2 (0x0079d000)
	libldap_r-2.4.so.2 => /usr/lib/i386-linux-gnu/libldap_r-2.4.so.2 (0x00176000)
	/lib/ld-linux.so.2 (0x00120000)
	libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0x0074b000)
	libp11-kit.so.0 => /usr/lib/i386-linux-gnu/libp11-kit.so.0 (0x00777000)
	libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0x0024a000)
	libresolv.so.2 => /lib/i386-linux-gnu/libresolv.so.2 (0x00e3d000)
	libroken.so.18 => /usr/lib/i386-linux-gnu/libroken.so.18 (0x007ac000)
	librtmp.so.0 => /usr/lib/i386-linux-gnu/librtmp.so.0 (0x00206000)
	librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0x00f76000)
	libsasl2.so.2 => /usr/lib/i386-linux-gnu/libsasl2.so.2 (0x00265000)
	libsqlite3.so.0 => /usr/lib/i386-linux-gnu/libsqlite3.so.0 (0x00f7f000)
	libtasn1.so.3 => /usr/lib/i386-linux-gnu/libtasn1.so.3 (0x00220000)
	libwind.so.0 => /usr/lib/i386-linux-gnu/libwind.so.0 (0x00c8e000)
	libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0x00234000)
	linux-gate.so.1 =>  (0x007dd000)

[darkblueb]

There is a project at ​code.google.com/p/netcdf4-python What is the relationship to apt-get install python-netcdf ?

After svn co http://code.google.com/p/netcdf4-python/ netcdf4-python-trunk and reading a bit, I noticed that checkversion.py read three variables from the netCDF4 module. These three variable match in netCDF4 supplied by apt-get install python-netcdf.

[darkblueb]

in python, making a call to netCDF4 Dataset.filepath() returns

filepath method requires netcdf lib >= 4.1.2, you have 4.1.1

[darkblueb]

a python example

In [2]: import netCDF4

In [3]: from netCDF4 import Dataset

In [4]: tDS = Dataset('/usr/local/share/data/netcdf/txxETCCDI_yr_MIROC5_rcp45_r2i1p1_2006-2100.nc')

In [5]: tDS.file_format
Out[5]: 'NETCDF3_CLASSIC'

Apparently the Canadian Climate Centre is using netCDF3 format data.. others mention using v3 data also, and a current rev of NetCDF is needed to do both netCDF3 and read/write HDF5 format.

[darkblueb]

Note that this page has no reference to python... more chance that there are several python implementations then..

​http://www.unidata.ucar.edu/downloads/netcdf/index.jsp

[darkblueb]

hiebert writes : -- With respect to netcdf pacakges in python, Scientific.IO.NetCDF is a pure python implementation that only handles classic (version 3) NetCDF files. To my knowledge it is not under active development (like you have noted with the 6 year old revision date). There is definitely merit to using a pure python implementation (it can run on different interpreters like PyPy and Jython, there is no build/linking process, and it's substantially more portable). However, not handling version 4 NetCDF files can be problematic.

It's worth noting that the NetCDF4 format is a subset of the HDF5 format, so you can use either library to open NetCDF4 files. We use the netcdf4-python and the h5py packages. Both sets of developers are extremely responsive and I trust their code. We typically install both using python's pip, rather than relying on the Debian packages.

I hadn't noticed that I only gave you NetCDF 3 files. Hopefully, that's OK. We use both depending on the needs of the data storage for a particular project.

--

[darkblueb]

oddly, netCDF4 install is erroring out in LUbuntu 14.04.1 change to setuptools method (for now)

[darkblueb]

package libhdf4-alt-dev is pulled in by gdal-dev:

user@live8-a4:~/libhdf4-4.2r10$ less debian/libhdf4-alt-dev.install

usr/lib-alt/lib*.a  usr/lib
usr/lib-alt/lib*.la usr/lib
usr/lib-alt/lib*.so usr/lib
usr/include-alt/hdf usr/include
usr/share/man/man1/hdf.3 usr/share/man/man3
debian/libhdf4-alt-dev.install (END)

[hamish]

fyi libhdf4-alt-dev is needed to move files around from their original, and perhaps user-expected positions, since GDAL wants to be able to read both HDF4 and HDF5 datasets, but the upstream authors only expect you to have one version installed at a time, which leads to conflicting namespace problems in the filesystem. there's a bit of discussion about it on the debiangis mailing list archives if anyone is curious.

what's the current status of this ticket? specifically the errors mentioned in comment:9?

do we have a viewer already on disc that the *.nc files could be associated with via a double click?

thanks, Hamish

[darkblueb]

When the Live switched to using the jtaylor PPA for IPython, it resolved most of the ambiguities in the component versions under Ubuntu 14.04. Please note that the SciPy stack is a confederation of previously independent projects.. there definitely *were* issues of components dissonance, but things are much better now. NetCDF libs are included in that.

Some of this ticket is meant to simply document obscure lore as the installs evolved.

[darkblueb]

Live 8.5 build 12028 -- note that package python-netcdf supplies :

    /usr/lib/python2.7/dist-packages/Scientific/_netcdf.so

while package python-netcdf4 supplies :

    /usr/lib/python2.7/dist-packages/netCDF4.so

[darkblueb]

version 12dev alpha1 ; netCDF4 python is stable and integrated