Next Frame Processing Utilities

The TCL procedures in the nextFrameProc.tcl file provide support for reading a sequence of FITS files into a frame. A frame is defined as one FITS file (or possibly more than one) plus some overlap appended from the start of the next FITS file in the sequence.

The procedures of interest are:

EXAMPLEof how these things work

NOTE:


Example Usage

Assume there is a sequence of ten FITS files, each containing 100 rows and 100 columns. They are located in the current directory and named as follows:
        test-1.fits 
        test-2.fits
        ...
        test-10.fits
If you would like to process this sequence of files with an overlap of 10 rows, then the frame size would be 110 (i.e., number of rows in FITS file plus number of overlap rows). The following TCL line would initialize the state:
        dervish> set state [nextFrameInit . test- 1 10 .fits 10 110 0]
Then the nextFrameGet and nextEOF proc's could be used to read each FITS file in the sequence and determine when the last file has been read:
        dervish> set region [keylget state frameHandle]
        dervish> set state [nextFrameGet $state]
        dervish> if { ![nextEOF state] } then {...do something with $region...}
A loop in a TCL script to do this might look like:
        
        set state [nextFrameInit . test- 1 10 .fits 10 110 0]
        set region [keylget state frameHandle]  # Save frame region handle
        set state [nextFrameGet $state]         # Get first frame
        while { ![nextEOF $state] } {           # Loop over FITS file sequence
          ...do something with $region...
          set state [nextFrameGet $state]       # Get next frame
        }

nextFrameInit

Initializes the keyed list that controls nextFrame.
nextFrameInit <dirName> <baseName> <beginFile> <endFile> 
              <suffixName> <overlapRows> <frameRows> <nfDebug>

       <dirName>	Directory containing FITS files
       <baseName>	Base component of file names
       <beginFile>	Beginning (numbered) component of file name
       <endFile>	Ending (numbered) compenent of file name
       <suffixName>	Suffix of file names
       <overlapRows>	Number of rows in overlap
       <frameRows>	Number of rows in frame (FITS file rows + overlap rows)
       <nfDebug>	0 = No debug statements;  1 = Print debug statements

nextFrameGet

Loads up the frame with the next set of rows.
nextFrameGet <state>
       <state>	Frame state

nextEOF

Returns 1 if state says it is out of input files, 0 otherwise.
nextEOF <state>
       <state>	Frame state

nextFieldInit

Initializes the keyed list that controls nextField.
set fieldState [nextFieldInit <dirName> <filterList> <baseName> <beginFile>
              <endFile> <suffixName> <overlapRows> <frameRows> <nfDebug>]

       <filterList>	list of filter names, in the order they appear on the tape
       <dirName>	Directory containing FITS files
       <baseName>	Base component of file names
       <beginFile>	Beginning (numbered) component of file name
       <endFile>	Ending (numbered) compenent of file name
       <suffixName>	Suffix of file names
       <overlapRows>	Number of rows in overlap
       <frameRows>	Number of rows in frame (FITS file rows + overlap rows)
       <nfDebug>	0 = No debug statements;  1 = Print debug statements

nextFieldGet

Loads up the set of frames with the next field.
set fieldState [nextFieldGet <fieldState>]
       <fieldState>	Field state

regFromFilter

Get the handle to the region corresponding to the filter
regFromFilter <filter> <fieldState>

Example

Here is a proc that reads 7 fields:
proc nextFieldGetExample {dirName iRun iCol} {
    set baseName run[format %06d $iRun]-c$iCol
    set hdr [hdrNew]
    set infoChain [fits2Schema $dirName/$baseName.fieldInfo FI $hdr]
    set filterList [hdrGetAsAscii $hdr FILTERS]
    echo call initNextField:
    set iField 0
    set debug 1
    set fieldState [nextFieldInit $filterList \
	    $dirName $baseName- 0 1000 .fits 200 1554 $debug]

    echo ============================================================
    echo MemStatsPrint after nextFieldInit:
    echo [memStatsPrint]
    echo ============================================================

    echo [timerStart]
    loop iField 0 7 {
	# Get the next field
	echo call nextFieldGet for iField = $iField
	set fieldState [nextFieldGet $fieldState]
	echo ============================================================
	echo MemStatsPrint after nextFieldGet:
	echo [memStatsPrint]
	echo ============================================================

	# Do the analysis
	set memPt [expr 1+[memSerialNumber]]
	foreach filter $filterList {
	    echo - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
	    set reg [regFromFilter $filter $fieldState]
	    echo for filter=$filter and reg=$reg
	    set ccdRow [hdrGetAsInt $reg.hdr CCDROW]
	    set ccdCol [hdrGetAsInt $reg.hdr CCDCOL]
	    set frame  [hdrGetAsInt $reg.hdr FRAME]
	    echo ccdRow=$ccdRow    ccdCol=$ccdCol   frame=$frame
	    set if [chainElementGetByPos $infoChain $iField]
	    echo Information from the infoChain:
	    exprPrint $if
	    handleDel $if
	    set reg [regNew 1000 1000]
	    set reg [regNew 1000 1000]
	    set reg [regNew 1000 1000]
	    set reg [regNew 1000 1000]
	    set reg [regNew 1000 1000]
	}
	echo -----------> Free up the memory allocated during iField=$iField
	memFreeBlocks $memPt [memSerialNumber]
	echo [timerLap]
    }
    return 0
}

And here is its output:
al> nextFieldGetExample /data/dp1.3/astroline/run/601Input 601 3 
call initNextField:
--- bufferFill: thisFile=0 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000000.fits
--- bufferFill: thisFile=1 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000001.fits
--- bufferFill: thisFile=2 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000002.fits
--- bufferFill: thisFile=3 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000003.fits
--- bufferFill: thisFile=4 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000004.fits
============================================================
MemStatsPrint after nextFieldInit:

Number of memory allocation requests:     504
Number of memory de-allocation requests:  82
Total bytes currently in use:             103575705
Total bytes in Free Memory Pool:          9472
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        13.10 %
Percentage of memory allocation requests
  satisfied from the Operating System:    86.90 %


============================================================

call nextFieldGet for iField = 0
--- bufferFill: thisFile=5 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000005.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=6 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000006.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=7 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000007.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=8 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000008.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=9 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000009.fits
--- nextEOF: No EOF yet
============================================================
MemStatsPrint after nextFieldGet:

Number of memory allocation requests:     1164
Number of memory de-allocation requests:  492
Total bytes currently in use:             103617170
Total bytes in Free Memory Pool:          19200
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        40.81 %
Percentage of memory allocation requests
  satisfied from the Operating System:    59.19 %


============================================================
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=r and reg=h5
ccdRow=1 ccdCol=3 frame=15
Information from the infoChain:
lambda     -2.89377777
eta        0
time       50893.3391839
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=i and reg=h8
ccdRow=2 ccdCol=3 frame=17
Information from the infoChain:
lambda     -2.89377777
eta        0
time       50893.3391839
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=u and reg=h11
ccdRow=3 ccdCol=3 frame=19
Information from the infoChain:
lambda     -2.89377777
eta        0
time       50893.3391839
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=z and reg=h14
ccdRow=4 ccdCol=3 frame=21
Information from the infoChain:
lambda     -2.89377777
eta        0
time       50893.3391839
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=g and reg=h17
ccdRow=5 ccdCol=3 frame=23
Information from the infoChain:
lambda     -2.89377777
eta        0
time       50893.3391839
-----------> Free up the memory allocated during iField=0
{ELAPSED 40.881} {CPU {{OVERALL 30.970} {UTIME 25.360} {STIME 5.610} {CUTIME 0.000} {CSTIME 0.000}}}
call nextFieldGet for iField = 1
--- bufferFill: thisFile=10 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000010.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=11 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000011.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=12 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000012.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=13 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000013.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=14 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000014.fits
--- nextEOF: No EOF yet
============================================================
MemStatsPrint after nextFieldGet:

Number of memory allocation requests:     1984
Number of memory de-allocation requests:  1312
Total bytes currently in use:             103617170
Total bytes in Free Memory Pool:          50138240
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        59.17 %
Percentage of memory allocation requests
  satisfied from the Operating System:    40.83 %


============================================================
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=r and reg=h5
ccdRow=1 ccdCol=3 frame=16
Information from the infoChain:
lambda     -2.59288889
eta        0
time       50893.3400198
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=i and reg=h8
ccdRow=2 ccdCol=3 frame=18
Information from the infoChain:
lambda     -2.59288889
eta        0
time       50893.3400198
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=u and reg=h11
ccdRow=3 ccdCol=3 frame=20
Information from the infoChain:
lambda     -2.59288889
eta        0
time       50893.3400198
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=z and reg=h14
ccdRow=4 ccdCol=3 frame=22
Information from the infoChain:
lambda     -2.59288889
eta        0
time       50893.3400198
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=g and reg=h17
ccdRow=5 ccdCol=3 frame=24
Information from the infoChain:
lambda     -2.59288889
eta        0
time       50893.3400198
-----------> Free up the memory allocated during iField=1
{ELAPSED 77.152} {CPU {{OVERALL 58.760} {UTIME 49.790} {STIME 8.970} {CUTIME 0.000} {CSTIME 0.000}}}
call nextFieldGet for iField = 2
--- bufferFill: thisFile=15 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000015.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=16 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000016.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=17 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000017.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=18 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000018.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=19 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000019.fits
--- nextEOF: No EOF yet
============================================================
MemStatsPrint after nextFieldGet:

Number of memory allocation requests:     2804
Number of memory de-allocation requests:  2132
Total bytes currently in use:             103617170
Total bytes in Free Memory Pool:          50138240
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        71.11 %
Percentage of memory allocation requests
  satisfied from the Operating System:    28.89 %


============================================================
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=r and reg=h5
ccdRow=1 ccdCol=3 frame=17
Information from the infoChain:
lambda     -2.292
eta        0
time       50893.3408556
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=i and reg=h8
ccdRow=2 ccdCol=3 frame=19
Information from the infoChain:
lambda     -2.292
eta        0
time       50893.3408556
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=u and reg=h11
ccdRow=3 ccdCol=3 frame=21
Information from the infoChain:
lambda     -2.292
eta        0
time       50893.3408556
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=z and reg=h14
ccdRow=4 ccdCol=3 frame=23
Information from the infoChain:
lambda     -2.292
eta        0
time       50893.3408556
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=g and reg=h17
ccdRow=5 ccdCol=3 frame=25
Information from the infoChain:
lambda     -2.292
eta        0
time       50893.3408556
-----------> Free up the memory allocated during iField=2
{ELAPSED 123.107} {CPU {{OVERALL 86.720} {UTIME 74.190} {STIME 12.530} {CUTIME 0.000} {CSTIME 0.000}}}
call nextFieldGet for iField = 3
--- bufferFill: thisFile=20 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000020.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=21 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000021.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=22 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000022.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=23 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000023.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=24 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000024.fits
--- nextEOF: No EOF yet
============================================================
MemStatsPrint after nextFieldGet:

Number of memory allocation requests:     3624
Number of memory de-allocation requests:  2952
Total bytes currently in use:             103617170
Total bytes in Free Memory Pool:          50138240
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        77.65 %
Percentage of memory allocation requests
  satisfied from the Operating System:    22.35 %


============================================================
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=r and reg=h5
ccdRow=1 ccdCol=3 frame=18
Information from the infoChain:
lambda     -1.99111111
eta        0
time       50893.3416914
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=i and reg=h8
ccdRow=2 ccdCol=3 frame=20
Information from the infoChain:
lambda     -1.99111111
eta        0
time       50893.3416914
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=u and reg=h11
ccdRow=3 ccdCol=3 frame=22
Information from the infoChain:
lambda     -1.99111111
eta        0
time       50893.3416914
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=z and reg=h14
ccdRow=4 ccdCol=3 frame=24
Information from the infoChain:
lambda     -1.99111111
eta        0
time       50893.3416914
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=g and reg=h17
ccdRow=5 ccdCol=3 frame=26
Information from the infoChain:
lambda     -1.99111111
eta        0
time       50893.3416914
-----------> Free up the memory allocated during iField=3
{ELAPSED 172.125} {CPU {{OVERALL 114.380} {UTIME 98.570} {STIME 15.810} {CUTIME 0.000} {CSTIME 0.000}}}
call nextFieldGet for iField = 4
--- bufferFill: thisFile=25 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000025.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=26 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000026.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=27 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000027.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=28 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000028.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=29 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000029.fits
--- nextEOF: No EOF yet
============================================================
MemStatsPrint after nextFieldGet:

Number of memory allocation requests:     4444
Number of memory de-allocation requests:  3772
Total bytes currently in use:             103617170
Total bytes in Free Memory Pool:          50138240
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        81.77 %
Percentage of memory allocation requests
  satisfied from the Operating System:    18.23 %


============================================================
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=r and reg=h5
ccdRow=1 ccdCol=3 frame=19
Information from the infoChain:
lambda     -1.69022222
eta        0
time       50893.3425272
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=i and reg=h8
ccdRow=2 ccdCol=3 frame=21
Information from the infoChain:
lambda     -1.69022222
eta        0
time       50893.3425272
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=u and reg=h11
ccdRow=3 ccdCol=3 frame=23
Information from the infoChain:
lambda     -1.69022222
eta        0
time       50893.3425272
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=z and reg=h14
ccdRow=4 ccdCol=3 frame=25
Information from the infoChain:
lambda     -1.69022222
eta        0
time       50893.3425272
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=g and reg=h17
ccdRow=5 ccdCol=3 frame=27
Information from the infoChain:
lambda     -1.69022222
eta        0
time       50893.3425272
-----------> Free up the memory allocated during iField=4
{ELAPSED 201.786} {CPU {{OVERALL 141.560} {UTIME 122.850} {STIME 18.710} {CUTIME 0.000} {CSTIME 0.000}}}
call nextFieldGet for iField = 5
--- bufferFill: thisFile=30 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000030.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=31 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000031.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=32 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000032.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=33 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000033.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=34 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000034.fits
--- nextEOF: No EOF yet
============================================================
MemStatsPrint after nextFieldGet:

Number of memory allocation requests:     5264
Number of memory de-allocation requests:  4592
Total bytes currently in use:             103617170
Total bytes in Free Memory Pool:          50138240
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        84.61 %
Percentage of memory allocation requests
  satisfied from the Operating System:    15.39 %


============================================================
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=r and reg=h5
ccdRow=1 ccdCol=3 frame=20
Information from the infoChain:
lambda     -2.74333334
eta        0
time       50893.3396019
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=i and reg=h8
ccdRow=2 ccdCol=3 frame=22
Information from the infoChain:
lambda     -2.74333334
eta        0
time       50893.3396019
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=u and reg=h11
ccdRow=3 ccdCol=3 frame=24
Information from the infoChain:
lambda     -2.74333334
eta        0
time       50893.3396019
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=z and reg=h14
ccdRow=4 ccdCol=3 frame=26
Information from the infoChain:
lambda     -2.74333334
eta        0
time       50893.3396019
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
for filter=g and reg=h17
ccdRow=5 ccdCol=3 frame=28
Information from the infoChain:
lambda     -2.74333334
eta        0
time       50893.3396019
-----------> Free up the memory allocated during iField=5
{ELAPSED 228.413} {CPU {{OVERALL 167.250} {UTIME 146.940} {STIME 20.310} {CUTIME 0.000} {CSTIME 0.000}}}
call nextFieldGet for iField = 6
--- bufferFill: thisFile=35 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000035.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=36 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000036.fits
--- nextEOF: No EOF yet
--- bufferFill: thisFile=37 endFile=1000
--- bufferFill: Reading /data/dp1.3/astroline/run/601Input/run000601-c3-000037.fits
--- nextEOF: No EOF yet