Kerchunk is a library that provides a unified way to represent a variety of
chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient
access to the data from traditional file systems or cloud object storage. It
also provides a flexible way to create virtual datasets from multiple files. It
does this by extracting the byte ranges, compression information and other
information about the data and storing this metadata in a new, separate object.
This means that you can create a virtual aggregate dataset over potentially many
source files, for efficient, parallel and cloud-friendly in-situ access without
having to copy or translate the originals. It is a gateway to in-the-cloud
massive data processing while the data providers still insist on using legacy
formats for archival storage.
We provide the following things:
- completely serverless architecture
- metadata consolidation, so you can understand a many-file dataset (metadata
plus physical storage) in a single read
- read from all of the storage backends supported by fsspec, including object
storage (s3, gcs, abfs, alibaba), http, cloud user storage (dropbox, gdrive)
and network protocols (ftp, ssh, hdfs, smb...)
- loading of various file types (currently netcdf4/HDF, grib2, tiff, fits,
zarr), potentially heterogeneous within a single dataset, without a need to go
via the specific driver (e.g., no need for h5py)
- asynchronous concurrent fetch of many data chunks in one go, amortizing the
cost of latency
- parallel access with a library like zarr without any locks
- logical datasets viewing many (>~millions) data files, and direct
access/subselection to them via coordinate indexing across an arbitrary number
of dimensions
NOTE: If this package has multiple flavors (see below), then use one of them instead of the name specified above. NOTE: This is a Python port. Instead of py312-kerchunk listed in the above command, you can pick from the names under the Packages section.
===> The following configuration options are available for py312-kerchunk-0.2.10:
CFTIME=off: Time handling support
FITS=off: FITS file format support
GRIB2=off: GRIB2 file format support
HDF=off: HDF file format support
NETCDF3=off: netCDF 4 file format support
===> Use 'make config' to modify these settings
filesystems/py-kerchunk: Add py-kerchunk 0.2.7
Kerchunk is a library that provides a unified way to represent a variety of
chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient
access to the data from traditional file systems or cloud object storage. It
also provides a flexible way to create virtual datasets from multiple files. It
does this by extracting the byte ranges, compression information and other
information about the data and storing this metadata in a new, separate object.
This means that you can create a virtual aggregate dataset over potentially many
source files, for efficient, parallel and cloud-friendly in-situ access without
having to copy or translate the originals. It is a gateway to in-the-cloud
massive data processing while the data providers still insist on using legacy
formats for archival storage.
We provide the following things:
(Only the first 15 lines of the commit message are shown above )