

						19-Aug-1995
						Sami Saarinen
						(Draft #4)


	I/O-routines for CMA or Comprehensive Memory Array
	__________________________________________________

			(A proposal)


1. Introduction


Observational data found in the CMA-files is intended to be accessed
by a set of low level routines described in this document. The purpose
is to provide an easy and efficient way of storing or retrieving data.

In the underneath of these routines acts the ECMWF's pbio-library, a
C-based stdio-library. It transfers data in an unblocked fashion
to/from a disk by disregarding the logical record boundaries (opposite
to the Fortran binary I/O). Furthermore, the pbio-library enables
updating of certain fields in the middle (or beginning) of the
CMA-file, while retaining the data already written, untouched. For
example, an important update of the DDR-records that are situated in
the beginning of the file can be handled without loss (or truncation)
of the actual observation report data written previously just after --
at that time dummy -- DDRs.



2. Contents of the CMA-file


The CMA-file consists logically of two parts: 

- Data Description Records (DDRs)
- Report data

Currently there are two DDRs occupying totally less than 30kbytes.
They give essential information about subsequently found observations
in this file.

Report data contains a number of observations in a random geographical
locations. Each observation or report consists of a header describing
the name of the station, its location and some other crucial
information.  A report header is followed by one or more so called
body entries, in which each describe the actual variable name, its
current value and some status information. Thus each report occupies
variable amount of disk space ranging typically from a few kilobytes
to hundreds of kilobytes.

The CMA-file is a machine independent and always consists of IEEE
754-1985 double precision (64-bit) floating point numbers. Such a file
can be today generated practically in any machine that conforms this
IEEE-standard (even on Cray PVP-system, but with assistance of
CRAY2IEG- and IEG2CRAY-routines).  Since some of the data fields are
supposed to comprise of integers or even character symbols, they are
converted by means of applying the INT-function to the corresponding
double precision floating point entity. In addition to this, the
character symbols require an extra call to the CHAR-function to
retrieve their final value.  However, no true integer field can exceed
the value of 2**31-1 (2147483647) or to be less than -2**31
(-2147483648), since integers are always assumed to be 32-bit signed
quantities.




3. User interface to the CMA-files

This section describes the subroutines to be used to access the CMA-file
information:

	- CMA_OPEN :	Opens specific CMA-file and returns some size 
			information for use by dynamic memory allocators
	- CMA_READ :	Reads either DDR of a specific length or one, 
			many or all reports
	- CMA_WRITE:	Symmetric (almost) with the CMA_READ, 
			but of course writes data, instead of reading
	- CMA_CLOSE:	Closes the CMA-file
	- CMA_REWIND:	Repositions the CMA-file to the beginning


All routines are coded in Fortran-90. None of the routines is designed
to abort in case of abnormal situations. It is therefore solely left
to the user to take the corresponding actions based on return codes.

The user interface to these routines is as follows:


__________________________________________________________________________________

	SUBROUTINE CMA_OPEN(kunit, filename, mode, kinfo, kinfo_len, kret)
c*** Input variables
	character*(*), intent(in) :: filename	! Filename
	character*(*), intent(in) :: mode	! Open mode: 'r'=read, 
						! 'w'=write, 'a'=append
	integer, intent(in)  :: kinfo_len	! Length of the info vector
c*** Output variables
	integer, intent(out) :: kunit	! PBIO file pointer
	integer, intent(out) :: kret	! Return code
					!  0 = OK
					! -1 = Could open file
					! -2 = Invalid file name
					! -3 = Invalid open mode
	integer, intent(out) :: kinfo(kinfo_len)! Information vector (if mode 'r')
					! (1) = # of DDRs
					! (2) = total length of DDRs in dp-words
					! (3) = # of observations/reports
					! (4) = total length of report data
					
	! Relies on PBOPEN, PBREAD2, PBSEEK and C-routine fstat()

__________________________________________________________________________________

	SUBROUTINE CMA_READ(kunit, pbuf, kbufsize, kwhat, kret)
c*** Input variables
	integer, intent(in) :: kunit	! PBIO file pointer
	integer, intent(in) :: kwhat	! Type of read:
					! 0 = Read all DDRs into the buffer
					! 1 = Read the next report
					! n = Read the 'n' next reports
					!-1 = Read all the remaining reports
	                                !     or up to kbufsize w/o checking
        	                        !     how many reports actually read
	integer, intent(in) :: kbufsize ! Length of the input pbuf-array
c*** Output variables
	real(8), intent(out):: pbuf(kbufsize)	! Input array
	integer, intent(out):: kret		! Return code:
						!   >= 0 , # of dp-words read
						! -1 = Error in reading data
						! -2 = kbufsize was not large
						!      enough to hold the data

	! Relies on PBREAD2 and PBSEEK

****	Notes:  

	(1) If parameter kwhat == 0, then file is automatically repositioned
	    to the beginning of the CMA-file before the reading.

	(2) If parameter kwhat >= 1, then the reading will begin from the
	    current file position.

	(3) If parameter kwhat >= 1, then the array pbuf is scanned through
	    to locate the next kwhat-reports, assuming they will fit into the
	    given kbufsize. If not, an error (-2) will occur.

	    If too much data has been read, the file is automatically
	    repositioned just after the last valid report, so that the
	    subsequent read would start from the very next report boundary.

	    Also the return code is adjusted accordingly to match with the
	    actual data size which is therefore seldom equal to kbufsize.

	(4) If parameter kwhat == -1, no scan is made and data is simply read in
	    starting from the current file position.

	    Also, the number of reports read is generally not known, since
	    it is assumed to be known by the user.

	    Return code -2 will occur only if kbufsize was too small.

	(5) The following two codes are equivalent, except the first one is
	    much faster (the reading of DDRs has been preceded in both cases) :

		call CMA_READ(fp, pbuf, kbufsize, -1, kret)

		  and

		offset = 0
		do i = 1, number_of_reports
		  call CMA_READ(fp , pbuf(1+offset), kbufsize - offset, 1, kret)
		  offset = offset + kret
		enddo

	(6) It is assumed that the CMA-file is a true file that can be 
	    repositioned (unlike "stdin" or so)

__________________________________________________________________________________

	SUBROUTINE CMA_WRITE(kunit, pbuf, kbufsize, kwhat, kret)
c*** Input variables
	integer, intent(in) :: kunit	! PBIO file pointer
	integer, intent(in) :: kwhat	! Type of write:
					! 0 = Write/update DDRs at the beginning
					!-1 = Write all the reports in pbuf
	integer, intent(in) :: kbufsize ! Length of the output pbuf-array
	real(8), intent(in) :: pbuf(kbufsize)	! Output array
c*** Output variables
	integer, intent(out):: kret		! Return code:
						!   >= 0 , # of dp-words written
						! -1 = Error in writing data

	! Relies on PBWRITE and PBSEEK

****	Notes:

	(1) Two modes of output exist: DDR or (any) data. They are distinguished
	    from each other by the fact that before starting to write DDRs,
	    the file is repositioned to the beginning. Whereas with the other 
	    option, the routine simply dumps out the pbuf starting from the current 
	    file position.

	(2) It is solely under user responsibility to have full set of reports 
	    in the pbuf.

__________________________________________________________________________________

	SUBROUTINE CMA_CLOSE(kunit, kret)
c*** Input variables
	integer, intent(in) :: kunit	! PBIO file pointer
c*** Output variables
	integer, intent(out):: kret	! Return code
					!  0 = OK
					! -1 = File handling error

	! Relies on PBCLOSE

__________________________________________________________________________________

	SUBROUTINE CMA_REWIND(kunit, kret)
c*** Input variables
	integer, intent(in) :: kunit	! PBIO file pointer
c*** Output variables
	integer, intent(out):: kret	! Return code
					!  0 = OK
					! -1 = File handling error

	! Relies on PBSEEK

__________________________________________________________________________________



4. An example


The following example demonstrates how use the CMA I/O-routines in
conjunction with the Fortran-90 dynamic memory allocation.


	PROGRAM demo_with_CMA_file
	implicit none

	real(8), pointer :: all_ddrs(:)
	real(8), pointer :: all_reports(:)
	integer info_vector(4)
	integer ret_code
	integer number_of_ddrs, number_of_reports
	integer ddr_len, data_len
	integer fp

c**	Open CMA for read and retrieve size etc. information
	call CMA_OPEN(fp, 'CMAFILE', 'r', info_vector, 4, ret_code)

c**	Update size information and allocate memory
	number_of_ddrs    = info_vector(1)
	ddr_len           = info_vector(2)
	number_of_reports = info_vector(3)
	data_len          = info_vector(4)

	allocate(all_ddrs(ddr_len))
	allocate(all_reports(data_len))

c**	Read DDRs and reports
	call CMA_READ(fp, all_ddrs,    ddr_len,   0, ret_code)
	call CMA_READ(fp, all_reports, data_len, -1, ret_code)

c**	Close CMA-file
	call CMA_CLOSE(fp, ret_code)
	
c**	Work with DDRs and report data
	call work(all_ddrs, ddr_len, all_reports, data_len)

c**	Open new CMA-file for write
	call CMA_OPEN(fp, 'CMAFILE.new', 'w', info_vector, 0, ret_code)

c** 	Write dummy or not yet fully up-to-date DDRs to file
	call CMA_WRITE(fp, all_ddrs, ddr_len, 0, ret_code)

c**	Write report data as it is now
	call CMA_WRITE(fp, all_reports, data_len, -1, ret_code)

c**	Update DDR-information and write it to file
c	This won't destroy the contents of the previously written reports
	call update_ddr_info(all_ddrs, ddr_len, all_reports, data_len)
	call CMA_WRITE(fp, all_ddrs, ddr_len, 0, ret_code)

c**	Close CMA-file and deallocate memory
	call CMA_CLOSE(fp, ret_code)
	
	deallocate(all_ddrs)
	deallocate(all_reports)

	END


Note that the writing sequence shown here would of course have been
equivalent with the following code fraction, if one only could do
it in this way:

	call update_ddr_info(all_ddrs, ddr_len, all_reports, data_len)

	call CMA_WRITE(fp, all_ddrs,    ddr_len,   0, ret_code)
	call CMA_WRITE(fp, all_reports, data_len, -1, ret_code)
