unidata的通用数据模型和NetCDFJava库API概述

《unidata的通用数据模型和NetCDFJava库API概述》由会员分享，可在线阅读，更多相关《unidata的通用数据模型和NetCDFJava库API概述（52页珍藏版）》请在装配图网上搜索。

1、2021/7/11Unidatas Common Data ModelandNetCDF Java Library APIOverviewJohn CaronUnidata/UCARNov 20082021/7/12Java=Programmer Productivity Portability Object Oriented Libraries everywhere Thriving open source development Strong typing(aka type safety)needed for large development projects Good tools:ID

2、Es,debuggers,profilers Very productive Java is faster than C for some applications eg multithreaded server2021/7/13Tomcat:The Definitive Guide,Jason Brittain(OReilley 2007)2021/7/14Java Virtual Machine/Operating Systems JVM options Linux,Solaris,Windows(Sun)Mac OS X(Apple)AIX,Linux,Windows,z/OS(IBM)

3、HP-UX(Hewlett-Packard)2021/7/15Java Negatives Linking Java with C/Fortran apps is difficult Arguably not suitable for large scale numerical computation Type safety,array safety,strict reproducibility Multicore CPU challenge could shift Specialized languages can be more productive 2021/7/16NetCDF-Jav

4、a library 100%Java Open Source(LGPL,MIT)Independent implementation Used as a component in other software(partial)Integrated Data Viewer,THREDDS Data Server(Unidata)Panoply(NASA)ncBrowse(EPIC/NOAA)Java NEXRAD Viewer(NCDC/NOAA)MyWorld GIS(Northwestern)EDC for ArcGIS,ERRDAP(SFSC/NOAA)Live Access Server

5、(PMEL/NOAA)ncWMS(Reading)Matlab plug-in(USGS)2021/7/17NetcdfDatasetApplicationScientific Feature TypesNetCDF-Java/CDM architectureOPeNDAPTHREDDSCatalog.xmlNetCDF-3HDF5I/O service providerGRIBGININIDSNetcdfFileNetCDF-4NexradDMSPCoordSystem BuilderDatatype Adapter2021/7/18NetCDF Java Release Plans Cur

6、rent Stable Release NetCDF-Java 2.2 Maintenance,bug fixes only Development version 4.0 Extensive refactor,enhance performance Extended data types for NetCDF4 Sequences:variable length Structures Scientific Feature Types refactor Nested Tables abstract model for point features(point,station,trajector

7、y,profile)By the end of the year2021/7/19Format Readers(CDM files)General:NetCDF,OPeNDAP,HDF5,NetCDF4,HDF4,HDF-EOS Gridded:GRIB-1,GRIB-2,GEMPAK Radar:NEXRAD 2&3,DORADE,CINRAD,Universal Format,TDWR Point:BUFR,ASCII Satellite:DMSP,GINI,McIDAS AREA Misc:GTOPO,Lightning,etc Others in development(partial

8、):AVHRR,GPCP,GACP,SRB,SSMI,HIRS(NCDC)2021/7/110 HTTP Tomcat ServerTHREDDS Data ServerDatasetscatalog.xmlmotherlode.ucar.eduTHREDDS ServerNetCDF-JavalibraryRemote AccessIDD DataHTTPServerNetcdfSubsetWCSOPeNDAPconfigCatalog.xml2021/7/111NcML DatasetsApplicationdatasetNcML datasetApplicationdatasetNcML

9、 datasetTHREDDS dataset2021/7/112 NcML example2021/7/113TDS/NcML example 2021/7/114TDS/NcML aggregation 2021/7/115Common Data Model2021/7/116Whats a Data Model?An Abstract Data Model describes data objects and what methods you can use on them.An API is the interface to the Data Model for a specific

10、programming language A file format is a way to persist the objects in the Data Model.A data access protocol like OPeNDAP plays the role of a file format(sort of).An Abstract Data Model removes the details of any particular API and the persistence format.2021/7/117Scientific Feature TypesGridPointRad

11、ialTrajectorySwathStationProfileCoordinate SystemsCommon Data ModelData AccessnetCDF-3,HDF5,OPeNDAPBUFR,GRIB1,GRIB2,NEXRAD,NIDS,McIDAS,GEMPAK,GINI,DMSP,HDF4,HDF-EOS,DORADE,GTOPO,ASCII2021/7/118NetcdfDatasetApplicationScientific Feature TypesNetCDF-Java/CDM architectureOPeNDAPTHREDDSCatalog.xmlNetCDF

12、-3HDF5I/O service providerGRIBGININIDSNetcdfFileNetCDF-4NexradDMSPCoordSystem BuilderDatatype Adapter2021/7/119Common Data Model(Data Access Layer)2021/7/120NetCDF-4 Data ModelDimensionname:Stringlength:intisUnlimited()Attributename:Stringtype:DataTypevalues:1D arrayVariablename:Stringshape:Dimensio

13、n type:DataTypearray:read(),Filelocation:Filenamecreate(),open(),PrimitiveTypecharbyteshort intfloatdoubleDataTypeUser-defined types,including compound types,may be stored with other data.A file has a top-level unnamed group.Each group may contain one or more named subgroups,variables,dimensions,and

14、 attributes.A variable may also have attributes.Variables may share dimensions,indicating a common grid.One or more dimensions may be of unlimited length.2021/7/121Coordinate Systems2021/7/122Coordinate Systems as FunctionsData variable V,with n dimensions Vdim=dimk,k=0,n-1is a function from domain

15、Vdim to RV:Vdim RA coordinate variable for V is also a function CV:Vdim RA coordinate system for V,CSV,is a set of m coordinate variables for V CSV=CVj,j=0,m-1CSV:Vdim Rm The coordinates of the(i,j,k)data point are the m values CV1(i,j,k),CV2(i,j,k),CV3(i,j,k),A coordinate system must be invertible.

16、2021/7/123Coordinate Systems NetCDF,OPeNDAP,HDF data models do not have integrated coordinate systems so georeferencing not part of API Need conventions to specify(eg CF-1,COARDS,etc)Contrast GRIB,HDF-EOS,other specialized formats2021/7/124Coordinate Variablesdimensions:lat=64;lon=128;variables:floa

17、t lat(lat);float lon(lon);float time;double temperature(lat,lon);coordinates=“lat lon time”;2021/7/125Limitations of 1D Coordinate Variables Non lat/lon horizontal grids:float temperature(y,x)float lat(y,x);float lon(y,x);Trajectory data:float NKoreaRadioactivity(pt);float lat(pt);float lon(pt);floa

18、t altitude(pt);float time(pt)2021/7/126Coordinate Systems UML2021/7/127Projections(CF)albers_conical_equal_area lambert_azimuthal_equal_area lambert_conformal_conic mcidas_area mercator orthographic rotated_pole stereographic(including polar)transverse_mercator UTM(ellipsoidal)vertical_perspective20

19、21/7/128Vertical Transforms(CF)atmosphere_sigma atmosphere_hybrid_sigma_pressure atmosphere_hybrid_height ocean_s ocean_sigma existing3DField2021/7/129Add your own Transform Pluggable framework Add at runtime CoordTransBuilder.registerTransform()Implement CoordTransBuilderIF2021/7/130Scientific Feat

20、ure Types2021/7/131Scientific Feature Types Based on datasets Unidata is familiar with APIs are evolving Intended to scale to large,multifile collections Intended to support“specialized queries”Space,Time These form the basis for NetCDF-Java implementations Two categories:Grids and Points 2021/7/132

21、Gridded Data Grid:multidimensional grid,separable coordinates Radial:a connected set of radials using polar coordinates collected into sweeps Swath:a two dimensional grid,track and cross-track coordinates Unstructured Grids:finite element models,coastal modeling2021/7/133Gridded Datafloat gridData(t

22、,z,y,x);float t(t);float y(y);float x(x);float z(z);float lat(y,x);float lon(y,x);float height(t,z,y,x);Cartesian coordinatesData is 2,3,4D All dimensions have 1D coordinate variables(separable)2021/7/134Radial Datafloat radialData(radial,gate):float distance(gate)float azimuth(radial)float elevatio

23、n(radial)float time(radial)float origin_lat;float origin_lon;float origin_alt;Polar coordinates 2D:radials collected into sweeps Not separate time dimension2021/7/135Swathfloat swathData(track,xtrack)float lat(track,xtrack)float lon(track,xtrack)float alt(track,xtrack)float time(track)two dimensiona

24、l track and cross-track not separate time dimension orbit tracking allows fast search2021/7/136Unstructured Grid Pt dimension not connected Need to specify the connectivity explicitly No implementation in the CDM yetfloat unstructGrid(t,z,pt);float lat(pt);float lon(pt);float time(t);float height(z)

25、;2021/7/137float data(sample);Point:measured at one point in time and space Station:time-series of points at the same location Profile:points along a vertical line Station Profile:a time-series of profiles at same location.Trajectory:points along a 1D curve in time/space Section:a collection of prof

26、ile features which originate along a trajectory.1D Feature Types(“point data”)2021/7/138Point Observation Data Table lat,lon,z,time;obs1,obs2,.obs(sample);Set of measurements at the same point in space and time=obs Collection of obs=dataset Sample dimension not connectedfloat obs1(sample);float obs2

27、(sample);float lat(sample);float lon(sample);float z(sample);float time(sample);2021/7/139Time-series Station Data Table stationId;lat,lon,z;Table time;obs1,obs2,.obs(*);/connected stn(stn);/not connectedfloat obs1(sample);float obs2(sample);int stn_id(sample);float time(sample);int stationId(stn);f

28、loat lat(stn);float lon(stn);float z(stn);float obs1(sample);float obs2(sample);float lat(sample);float lon(sample);float z(sample);float time(sample);float obs1(stn,time);float obs2(stn,time);float time(stn,time);int stationId(stn);float lat(stn);float lon(stn);float z(stn);2021/7/140Profile Data T

29、able profileId;lat,lon,time;Table z;obs1,obs2,.obs(*);/connected profile(profile);/not connectedfloat obs1(sample);float obs2(sample);int profile_id(sample);float z(sample);int profileId(profile);float lat(profile);float lon(profile);float time(profile);float obs1(profile,level);float obs2(profile,l

30、evel);float z(profile,level);float time(profile);float lat(profile);float lon(profile);float obs1(sample);float obs2(sample);float lat(sample);float lon(sample);float z(sample);float time(sample);2021/7/141Time-series Profile Station Data Table stationId;lat,lon;Table time;Table z;obs1,obs2,.obs(*);

31、/connected profile(*);/connected stn(stn);/not connectedfloat obs1(profile,level);float obs2(profile,level);float z(profile,level);float time(profile);float lat(profile);float lon(profile);float obs1(stn,time,level);float obs2(stn,time,level);float z(stn,time,level);float time(stn,time);float lat(st

32、n);float lon(stn);2021/7/142Table trajectory_id;Table lat,lon,z,time;obs1,obs2,.obs(*);/connected traj(traj)/not connectedTrajectory Datafloat obs1(sample);float obs2(sample);float lat(sample);float lon(sample);float z(sample);float time(sample);int trajectory_id(sample);float obs1(traj,obs);float o

33、bs2(traj,obs);float lat(traj,obs);float lon(traj,obs);float z(traj,obs);float time(traj,obs);int trajectory_id(traj);2021/7/143 Section Datafloat obs1(traj,profile,level);float obs2(traj,profile,level);float z(traj,profile,level);float lat(traj,profile);float lon(traj,profile);float time(traj,profil

34、e);Table section_id;Table surface_obs/data anywhere lat,lon,time Table depth;obs1,obs2,.obs(*);/connected profile(*);/connected section(*)/not connected2021/7/144Nested Table Notation(1)A feature instance is a row in a table.A table is a collection of features of the same type.The table may be fixed

35、 or variable length.A nested(child)table is owned by a row in the parent table.Both coordinates and data variables can be at any level of the nesting.A feature type is represented as nested tables of specific form.A feature collection is an unconnected collection of a specific feature type.Table dat

36、a1,data2 lat,lon,time;Table z;obs1,obs2,.obs(17);profile(*);2021/7/145Nested Table Notation(2)A constant coordinate can be factored out to the top level.This is logically joined to any nested table with the same dimension.dim level=17;float z(level);Table data1,data2 lat,lon,time;Table obs1,obs2,.ob

37、s(level);profile(*);2021/7/146Nested Table Notation(3)A coordinate in an inner table is connected;a coordinate in the outermost table is unconnected.Table stationId;lat,lon;Table time;Table z;obs1,obs2,.obs(*);/connected profile(*);/connected stn(stn);/not connected Table lat,lon,z,time;obs1,obs2,.p

38、oint(sample);Table trajectory_id;Table lat,lon,z,time;obs1,obs2,.obs(*);/connected traj(traj)/not connected2021/7/147Relational model Nested Tables are a hierarchical data model(tree structure)Simple transformation to relational model explicitly add join variables to tables Table stationId;lat,lon,z

39、;Table time;obs1,obs2,.obs(42);stn(stn);RTable stationId/primary key lat,lon,z;stn RTable stationId/secondary key time;obs1,obs2,.obs;2021/7/148Nested Model Summary Compact notation to describe 1D point feature types Connectivity of points is key property Variable/fixed length table dimensions can b

40、e notated easily Constant/varying coordinates can be easily seen Can be translated to relational model to get different performance tradeoffs More details2021/7/149Feature Type implementationsNetcdf-Java libraryGridGridDatatypeRadialRadialSweepFeatureSwath-Unstructured Grids-PointPointFeatureStation

41、StationTimeSeriesFeatureProfileProfileFeatureTrajectoryTrajectoryFeatureStationProfileStationProfileFeatureSectionSectionFeature2021/7/150Encoding Feature Types in NetCDFUsing CF Conventions CF-1.0 focused on Grids Other types are being studied/proposed Unidata proposal for point obs NCAR/EOL working on Radial data(netCDF4)NPOESS/GOES-R using netCDF4 for satellite(swath)Unidata has proposal to NOAA/NASA Working group for unstructured grids Happening now!2021/7/151 若有不当之处，请指正，谢谢！若有不当之处，请指正，谢谢！