Pan-STARRS PS-1 IPP
Critical Design Review
January 24-25, 2006

IPP : The Pan-STARRS Image Processing Pipeline

The IPP Team

IfA Team

Eugene Magnier

Paul Price

Josh Hoblitt

Nick Kaiser

Robert Lupton

Dave Monet

Additional IfA Support

Trent Dupuy

Brandon Swift

Mark Pitts

Joe Masiero

MHPCC Team

Bruce Duncan

Mike Berning

Robert DeSonia

George Gusciora

David Robbins

Ross Harman

Eric VanAlst

Also known as PanPipes...

PS-1 Design Reference Mission

5-σ limit (AB)

Total int. (min)

Mode

y

z

i

r

g

w

Cad.

Area

19.7

65s

21.5

65s

22.6

185s

23.3

65s

23.7

65s

14d

3π

P&A

MVP

IVP

25.1

840

h/d/m

11000

24.2

2700

25.4

1333

25.9

533

27.0

2100

27.6

2100

4d

90

P&A Survey will visit each field 19 times:

5s + 2x30s in grzy

5s + 6x30s in i

P&A Survey goals are aggressive (but within reason):

10 milliarcsec relative astrometry

100 milliarcsec absolute astrometry

10 millimag internal photometry

IPP in Pan-STARRS

Preferred

Science

Clients

OTIS

PSPS

MOPS

Science Client

IPP

World

Users

Solar System

Community

static sky images

other data

metadata,

detections

raw

images

metadata,

detections

metadata

metadata,

detections

IDs

IDs

orbits

identifications

filtered detections &

metadata

Cameras

Telescopes

static sky images

PS

Subsystem

Legend

pixel data

meta & object data

External

System

commands

photons

Preferred

Sci Client

Primary Driving Requirements

Throughput

maximum rate: 45s per GPC image

max stellar densities (~300k / deg2)

Precision (AP Survey goals)

10 milli-magnitudes internal scatter

10 milli-arcseconds relative astrometry

100 milli-arcseconds absolute astrometry

Total Data Volume : ~1.2 PB over 3 years

Flexibility

Configurability

Reliability

Note that PS-1 system-level requirements have not been well defined to date!

Design Goals and Constraints

Reliability

IPP Infrastructure design

IPP Cluster design

Robust Analysis Scripts

Flexibility

IPP Infrastructure separate from Analysis

Analysis Scripts configuration

Traceability

PSLib log / error functions

Extensive Metadata

Testability

Modular Architecture

Modular Analysis Scripts

Throughput

Parallelized processing

Cluster design

Analysis Precision

Analysis Script steps

Photometry and Astrometry calibration terms

Analysis Strategies

raw

stacked

image

static

sky

image

difference

image

-

cleaned

stacked

image

+

The Static Sky

PS-1 : N sequentially

PS-4 : 4 simultaneous

Image Combine

&

Image Difference

Observing Strategies

AP Survey will distribute 16 images over 8 epochs

first 6 months will strongly bias to AP Survey

none in the last year

MOPS and IVP ramp up starting in mid-2007.

MOPS and IVP require their static sky pixels

AP Survey does not build static sky until ~end of survey

IVP is initial a demonstration: Phase 4 analysis techniques may iterate in the first rounds.

IPP Pipeline Architecture

Image

Server

Metadata DB

AP DB

IPP Controller

Camera

Client Science

Pipelines

PSPS

IPP Scheduler

IPP

Process

External

System

Legend

pixel data

meta & object data

commands & messages

Analysis Tasks

OTIS

images

static sky images

metadata

detections

filtered

detections

metadata

metadata

metadata

commands

commands

images

detections

metadata, Q/A

IPP

Data

publishing process

observing operations / processing boundary

ippTools

config

Note: Camera and Otis send data

using the Data Store interface

Basic Data Products

Object Measurements (Tables)

From Individual Science Images (P2)

From Cleaned / Stacked Images (P4Σ)

From Difference Images (P4Δ)

From Cumulative Static Sky

Image Data (Pixels)

Postage stamps for bright objects

Cumulative Static Sky images + exposure maps

Recent (~1 month) source and difference images

Might only be available to preferred science clients

Metadata Stream

Camera and Telescope logging data

Observing & Environmental Information

Processing inputs and outputs

IPP computing : distributed processing

classical parallel (eg MPI) vs distributed processing

increase total MHz

increase total I/O rate

'targeted' processing

OTA Nodes

Sky Nodes

Big Switch

connection to

observatory system

metadata db

server

pantasks

server

DVO

server

Software Architecture

image combination

image subtraction

astrometry calibration

photometry calibration

psModules

system configuration

camera representation

detrend construction

detrend application

object analysis

psLib

memory

error handling

data containers

images

vectors

FFTs

pcontrol (IPP Controller)

pantasks (IPP Scheduler)

Nebulous (Image Server)

Metadata DB Tools

DVO (AP DB)

statistics

fits / minimization

FITS & XML I/O

DB I/F

time

earth orientation

Legend:

concept

prototype

alpha code

beta code

Support Tools

ippTools

(dettools)

ippMonitor

Analysis Programs

psphot

psastro

ppImage

ppMerge

stac

pois

psLib and psModules version 0.9 released Dec 6.

Analysis Programs

PSPhot

object detection, PSF model, photometry, extended sources

PSAstro

single chip and mosaic image astrometry

ppImage

multi-format image load + convert, detrending, (PSPhot + PSAstro)

ppMerge

multi-image stack with scaling and outlier rejection

Stac / Warp

geometric warping and optimal science image combinations

POISub

image differencing using fitted PSF difference kernel

Current Schedule

January : CDR

February : finalize Phase 1,2,3 issues

basic system available for skyprobe

hand-built detrend images initially

March : initial detrend creation phase implementation

April-May : Focus on Testing P1-3, Detrend

Megacam / Suprime testing

TC-3 for communications tests

June : ready for TC-3

July-Sept : commissioning on TC-3

Oct : ready for GPC-1

Oct - Dec : commissioning test for GPC-1

Dec 2006 : ready for the survey

Phase 4 development proceeds in parallel, led by Paul Price

Phase 4 final decisions will require TC-3 input, astronomer debate...

Pan-STARRS Image Processing Pipeline Critical Design Review

PSLib and PSModules

PSLib

Provides a common set of data analysis tools

Consistent semantics and naming scheme

Encourages extensive code re-use

Facilitates testing through isolation

External library wrappers

System Tools

Data Containers

Math Constructs

Statistics and Data manipulation

I/O wrappers

Time tools

Earth orientation tools

PSLib : System Tools

Consistent Memory Management and Policies:

a type Foo is always provided a psFooAlloc()

psFree(foo) releases contained data

reference counting allows multiple views

leak and corruption checks built in

data types can be retrieved

psError / psLogMsg / psTrace

rich facilities for embedding different types of messages

an extensible error stack

convenient argument parsing, library initialization

PSLib : Data Containers

psList : doubly-linked list with iteration cursors

psHash : hash tables for key-based searches

psArray : serial container for void* elements

self-documenting

data container

representation of

a FITS header

may contain other

psMetadata folders

convenient I/O

tools

SIMPLE

T

BLANK

-32768

DATE-OBS

'2004-6-16'

COMMENT

DATA

COMMENT

PARAMS

EXPTIME

1.05

COMMENT

FOO

psListElem

psList

psListElem

psListElem

psListElem

psListElem

psListElem

psListElem

psHashBucket

psHash

psHashBucket

psHashBucket

psHashBucket

psHashBucket

COMMENT

psList

psListElem

psListElem

psListElem

LIST

BOOL

S32

STR

F32

psMetadata

CELL.00

METADATA

STR

STR

STR

psMetadata

EXTNAME

STR

CCD00

BIASNAME

STR

BSEC-00

CHIP

S32

CHIP.00

CELL.01

METADATA

psMetadata

psHashBucket

psHashBucket

psListElem

psListElem

EXTNAME

STR

CCD01

BIASNAME

STR

BSEC-01

CHIP

S32

CHIP.00

PSLib : the Metadata container

rich, human-readable, typed configuration files

# Specify the cell data

CELLS METADATA

left METADATA # Left amplifier

CELL.BIASSEC.SOURCE STR HEADER

CELL.TRIMSEC.SOURCE STR HEADER

CELL.BIASSEC STR BIASSEC

CELL.TRIMSEC STR DATASEC

CELL.XPARITY S32 1 # DEFAULT direction

CELL.X0 S32 1

CELL.Y0 S32 1

END

right METADATA # Right amplifier

CELL.BIASSEC.SOURCE STR HEADER

CELL.TRIMSEC.SOURCE STR HEADER

CELL.BIASSEC STR BIASSEC

CELL.TRIMSEC STR DATASEC

CELL.XPARITY S32 -1 # This cell is inverted

CELL.X0 S32 2048

CELL.Y0 S32 1

END

END

PSLib : psMetadata Config Files

PSLib : Math Structures

psVector

support for 8-64 bit int, 32 + 64 bit floats, chars, complex

psImage

support for 8-64 bit int, 32 + 64 bit floats, chars, complex

psHistogram

psPolynomial (1D,2D,3D,4D)

psSpline

'deterministic' random numbers (access to key)

extensive masking support

PSLib : Data Manipulation

FFTW wrappers for psVector, psImage

Polynomial fitting

Levenberg-Marqardt minimization (uses gradient)

Powell's Minimization algorithm (calculates gradient)

many analysis tools provided:

subsets, extractions, clipping, other range manipulations

vector and image statistics

general Unary and Binary math operations:

a = psBinaryOp (NULL, b, '+', c);

b = psUnaryOp (NULL, x, '^', psScalar(2.0))

a,b,c,x may be vector or image

vector sorting

matrix operations and linear math

PSLib : Time and Space

psTime conversion functions

psTime has second and nanosecond resolution

UTC / TAI / UT1 / TT

note: IPP database tables use TAI!

IERS Bulletin B interpolation is provided

Tidal correcitons for UT1 are also supplied

leap-second access is easy: psTimeLeapSecondDelta (t1,t2)

conversions to/from human-readable format

psPlane vs psSphere

linear transformations (2xN polynomials)

spherical rotations

Euler angles

Quartenions

2D / 3D coord conversions

various projections available

PSLib : Earth Orientation

Complete Transformations from ICRS to local Alt/Az coordinates

Based on IERS routines and SOFA package from IAU

psLib implementation is well-documented

APIs are clean and understandable:

psEOC_GetPolarMotion

psEOC_NutationCorr

psSphereRot_ITRStoTEO

High-Quality Refraction algorithm supplied by Ken Chambers

GCRS

Celestial, Geocentric

ITRS

Terrestial

Az/Alt

Local

CIP/CEO

Intermediate

CIP/TEO

Intermediate

Precession,

Nutation

Earth Rotation

Polar Motion

Topocentric,

Refraction

PSModules : Complex Library Functions

Configuration Data : site, camera, recipes

Focal Plane Tools : camera structure hierarchy

Astrometry Tools

Focal Plane Navigation

WCS / Astrometry File conversions

Matching / Fitting routines

Object Detection / Photometry

Detrending

Detrend Creation

Image Combination

Image Subtraction

PSModules : Camera Hierarchy

flexible camera hierarchy to represent wide range of camera

for example, a single chip detector (eg, skyprobe):

pmFPA

pmChip

pmCell

pmReadout

psImage

data

camera

concepts

cell

concepts

metadata

header

metadata

header

metadata

header

analysis

results

Pan-STARRS Image Processing Pipeline Critical Design Review

Analysis Programs

Analysis Programs

configuration / parameters

psphot

psastro

ppImage

ppMerge

POISub

Stac / warp

Software Configuration and Parameters

Site Definition

database hostname(s) (Metadata DB, Nebulous, DVO)

file system data as needed

known cameras

logging, tracing, etc

Camera definition files

camera layout

rules for recognition

camera-based recipes

Recipe files

analysis options

analysis input parameters

Configuration files checked into CVS and Tagged

psphotReadout ()

PSPhot

stand-alone program for testing, external uses

library operates on pmReadout

expects image, weight, mask (will build if needed)

image stats

fix background

find peaks

source moments

rough classes

basic deblend

choose PSF

fit objects

parameters

output

'Blend' fitting sequence

initial subtraction

try PSF model

try DBL model

try EXT model

assess and subtract

replace failures

aperture correction

linear ensemble fit

PSPhot : PSF modeling

pmModel vs pmPSF

'model' describes a specific object

(x,y), flux, local sky, + shape parameters

'psf' describes collection of objects on image

shape parameters vs position (2D polynomials, robust fitting)

curve-of-growth model

aperture correction model (may include dx,dy model)

statistics of image PSF analysis

many model classes available

pure Gaussian

polynomial approx (a la dophot)

Waussian (approx of polynomial approx)

power-laws of different types

'trails'

add your own!

extendable to pixel-based models

PSPhot : Galaxy Models and other extended sources

In 'Phase 2 & 4' mode, extended source models are for:

classification tests

subtraction to handle neighbors

quick & dirty information

Choice of extended object model is flexible

currently only one-at-a-time

Power-law models are fast

Sersic models may yield better classification

'trailed source' model (fast asteroid) in testing

Additional simple parameters to be added

PSPhot : sample results : simple star field

PSPhot : sample results (simple star field)

PSPhot : sample results : galaxy field

PSPhot : sample results : galaxy field

PSPhot : sample results : crowded field

PSPhot : basic PSF fitting

PSPhot : pre-filter with fixed-position PSFs

PSPhot : add in multi-source fit for blends

PSPhot : add in double / extended test

PSPhot : Aperture Corrections

fitting radius expands with star flux (out to (S/N)/pix limit)

aperture magnitudes also use scaled aperture

curve-of-growth is measured from PSF model

aperture magnitudes corrected to fixed aperture

ap-fit modeled as function of:

x,y (up to 2nd order polynomial)

1/flux

fluxmeas = fluxreal - dsky*Npix

dflux/flux = -dsky*Npix/flux

flux (~ saturation roll-over)

sample of 1800 images (power-law PSF)

ap-fit: 13 mmag +/- 5mmag

σap-fit : 6.7mmag

measured from stars with S/N > 100

PSPhot : Sextractor comparison

better saturation

cleaner low S/N

clean ap-fit

fast roll-off

PSPhot : other issues

Output formats include FITS tables

Output selections are user-configured

Summary data added to output headers

Library call saves output on pmReadout.analysis metadata

PSPHOT.SOURCES, PSPHOT.STATS

Processing may include / exclude certain steps

PSF-models may be loaded from prior analysis

Fixed source positions may be loaded

Tests are showing ~2-3 ms per source (for full fitting steps)

Path forward to the Static Sky

include models convolved with the PSF

simultaneous, multi-image fitting

requires pmSource links parallel to blends

varient on simultaneous, multi-source fitting (exists)

more detailed galaxies models

PSastro

Stand-alone program for testing, external uses

Library expects pmChip or pmFPA

Four Major Modes

sequence of readouts, relative to first readout (low-order)

per-chip analysis: linear fits (+ static projection + distortion)

mosaic tweak: boresite (RA,DEC,Theta,Scale : Phase 1)

full-mosaic analysis: distortion + chips, high-order

understands: camera layout, WCS, P-S astrometry table

input is an array of pmSources from PSPhot

stand-alone program reads and writes FITS tables

functions exist to convert to/from other source structures

PSastro : chip fitting

Based on the Elixir `gastro' program

Reads data from a DVO database

HST GSC, USNO A or B, 2MASS, Tycho, UCAC

Native read of these are possible as well (slower)

Steps include

rough match (2pt correlation / triangle is option)

shrinking radius match

robust outlier rejection and fit

Results go to pmFPA structures

Conversions to WCS and/or PS astrometry table

IPP implementation partially complete, minimal tests

Elixir version has 95-98% success rate per chip

>500,000 test cases!

PSastro : Full-Mosaic Fitting

re-register outlier chips

simultaneously fit all chips

single optical distortion model (3rd order)

per-chip 1st, 2nd, 3rd order models

break chip-position / distortion degeneracy with local distortion gradient

clip outliers, robust fitting

PSastro : Example from MegaPrime

2004 Mar

vs

USNO

PSastro : MegaPrime vs MegaPrime

2004 Mar

vs

2004 Dec

PSastro : a closer view

~20 mas res.

PSastro : chip order matters (what is best?)

coherent wiggles imply fits are inconsistent

results from offset fields reinforces this conclusion

higher order terms help

better outlier rejection?

is overlap solution required?

Pan-STARRS Image Processing Pipeline Critical Design Review

Camera Hierarchy

Requirements

Hierarchical approach:

Must adequately represent the Pan-STARRS camera

Focal plane array consists of multiple chips

Chips have multiple cells

Cells may have multiple readouts

Allow processing to occur at different levels (e.g.: cell by cell; entire FPA)

Desirable to adequately represent other astronomical mosaic cameras (testing)

esp. MegaCam (raw, splice, split)

PS camera is more general

Must be able to import/export camera metadata

FITS headers, database, manually configured

Must make camera metadata available at the appropriate level

Design

pmFPA

pmChip

pmCell

pmReadout

psImage

data

camera

concepts

cell

concepts

analysis

metadata

analysis

metadata

analysis

metadata

analysis

metadata

file info

file info

file info

chip

concepts

pixels, mask, weight

psList

psImage

overscans

x N

x N

x N

hdu info

hdu info

hdu info

Example: PS-GPC

pmFPA

pmChip

pmCell

pmReadout

psImage

data

camera

concepts

cell

concepts

analysis

metadata

analysis

metadata

analysis

metadata

analysis

metadata

file info

chip

concepts

pixels, mask, weight

psList

psImage

overscans

x 64

x 64

x N

hdu info

Example: MegaCam raw

pmFPA

pmChip

pmCell

pmReadout

psImage

data

camera

concepts

cell

concepts

analysis

metadata

analysis

metadata

analysis

metadata

analysis

metadata

file info

chip

concepts

pixels, mask, weight

psList

psImage

overscans

x 36

x 2

x 1

hdu info

Example: MegaCam splice

pmFPA

pmChip

pmCell

pmReadout

psImage

data

camera

concepts

cell

concepts

analysis

metadata

analysis

metadata

analysis

metadata

analysis

metadata

file info

chip

concepts

pixels, mask, weight

psList

psImage

overscans

x 36

x 2

x 1

hdu info

“Concepts”

“Concepts” = values that have some meaning within astronomical image processing

e.g., gain, exposure time, RA, filter

Needs a better name, but this is all I can think of....

A “concept” will apply to particular level in the camera hierarchy

e.g., cell, cell, fpa, fpa

Need a wrapper around the “concepts” to insulate ourselves from the peculiarities of each camera system

e.g., the gain of a particular detector:

Sometimes stored in a FITS header

Might be catalogued in a database

Maybe it has to be defined manually

“Concepts” details

Specified by name, comment, type, default value, and functions to read and write

Default functions for straightforward values (e.g., gain, airmass)

Some require more detail to parse an intermediate result (e.g., RA, time)

Standard:

FPA.

NAME, AIRMASS, FILTER, POSANGLE, RA, DEC, RADECSYS

CHIP.

NAME

CELL.

GAIN, READNOISE, SATURATION, BAD, EXPOSURE, DARKTIME, TRIMSEC, BIASSEC, XBIN, YBIN, X0, Y0, XPARITY, YPARITY, READDIR, TIME, TIMESYS

New “concepts” may be registered by the user

Camera configuration file

How to identify from a FITS PHU

What does the PHU represent?

What does each extension represent?

What is in each extension?

Definition of the types of cells

Camera configuration file

Translation of FITS headers to “concepts”

Default values for “concepts”

Accessing “concepts” from a database

Specify formats for some “concepts”

Recipes to use in processing

How the mask and weight images are stored

Camera hierarchy operations

Generate camera hierarchy from configuration file

Blank concepts

Read nominated hierarchy components

Read pixels from FITS

Ingest concepts from FITS headers, database or defaults (configuration file)

Read mask and weight images

Write nominated hierarchy components

Write pixels to FITS

Outgest concepts to FITS headers, database, or check defaults

Write mask and weight images

Facility to define and register additional concepts

Splice cells within a chip to form a single cell

Pan-STARRS Image Processing Pipeline Critical Design Review

ppImage:

Pan Pipes Image (processor)

Scott Gustafson

Requirements

Standard astronomical image processing (“Phase 2”)

Cell operations:

Overscan+bias+dark subtraction

Flat-fielding

Non-linearity correction

Mask bad pixels

Fringe subtraction

Chip operations:

Photometry

Astrometry

Design

Design of ppImageLoop()

Loop over FPA

Read fpa data if desired

Loop over chips

Read chip data if desired

Loop over cells

Read cell data if desired

Detrend:

Overscan+bias+dark subtraction

Non-linearity correction

Flat-field

Mosaic component cells

Photometry

Astrometry

Write chip

Status

Major work was defining and implementing the camera hierarchy and “concepts”

Will also form the basis for other programs (e.g., ppMerge)

Working prototype

Overscan+bias subtraction, flat-field, weight generation (required for psphot)

Dark subtraction, masking, non-linearity correction coded, but untested

Photometry and astrometry not plugged in yet

Library code exists (EAM)

Memory leaks currently prohibit processing entire MegaCam frame in one go

Can cheat by executing 36 times and appending extensions to output FITS file

Plugging the leaks is at the top of my “to do” list following CDR

5.5 sec per MegaCam chip (Xeon 2.4 GHz) including I/O --> 13 sec per OTA

Once all components have been completed, can compare with Elixir processing

ppMerge : detrend stacking

performs scaling and stacking

used for

bias and darks

raw flats

fringe frames

also used to renormalize the chips

very similar structure to ppImage

pixel loading may be performed by FPA, Chip, or Cell

Pan-STARRS Image Processing Pipeline Critical Design Review

stac

Simultaneous Telescope Array Combination

The problem

Given multiple images of a patch of sky, want to

Warp images to a common coordinate frame; and

Combine images to increase S/N and reject artifacts

Shifting is straightforward

Combination is a standard astronomical function, with several oft-used implementations

IRAF imcombine, JT autoclean, STScI multidrizzle

Most implementations act on shifted images, but shifting smears out CRs over multiple pixels

Makes CRs harder to identify

After finding CRs, need to examine nearby pixels

Would like to mask pixels in the unshifted image

Simultaneous Telescope Array Combination (stac)

Given input images and known transformation to a common frame

Warp images to common frame

One pixel in original frame corresponds to multiple pixels in warped frame, and vice versa

Compare pixels in common frame

Discordant pixels have the source pixels in the original frame flagged with their contribution to the discordant pixel

Pixels in the source frame with a flag value greater than a defined limit are masked

CRs that are spread over multiple warped pixels should get flagged two or three time, which should trigger the limit

CRs that contribute primarily to a single warped pixel should be easily identified

Warp masked original images and combine

Status

Prototype implementation: stac

Built on pslib

Quality (0.01) appears to be better than autoclean (0.05) and imcombine (0.19)

Application: Joe Masiero (asteroids); Trent Dupuy (TGBN); PAP (M31)

Design in PS Modules SDRS

Implementation by MHPCC team currently untested

Identified problem when combining images taken with differing PSFs

Stars get pixels rejected, but galaxies don't, so resultant PSF is function of object extent

Status

Positive identifications: Combine 4x5122 blank images with 2000 CRs each, count pixels above threshold

stac: 0.01%

autoclean: 0.05%

imcombine: 0.19%

False identifications: Combine 4 fake star fields with no CRs, count number of CRs identified

stac: 0.03%

autoclean: low

imcombine: 0.5%

Speed: Combine four 4k square images on 2.2 GHz Opteron

stac: 60 sec (includes warping for 37 sec)

autoclean: 20 sec (not including warping)

To do

Test MHPCC implementation

Solution to PSF problem: convolve to common PSF before combining

may involve using poiSub to match PSFs

Warp to specified sky cell instead of common coordinate frame

Accomodate “magic”

Phase 4a: warp, Q&D subtract static sky from each image

“Magic”: zap satellites

Phase 4b: combine warps, subtract static sky properly

More testing: IfA Deep survey, POMME, SuperMACHO, ESSENCE

Quantify CR contamination by tracing CRs through combination + subtraction + find objects

Important for, e.g., high z QSOs, faint transients (“one shot wonders”)

Pan-STARRS Image Processing Pipeline Critical Design Review

poiSub

Pan-STARRS Optimal Image Subtraction

The problem

Given an image, want to identify new or variable sources in the image domain

Prefer image domain over catalogue domain because

Want to find sources in crowded fields, and on galaxies

Work with a single catalogue

Often done in SN and microlensing searches

Legacy algorithm:

Alard & Lupton 1998, “A method for optimal image subtraction”

Alard 2000, “Image subtraction using a space-varying kernel”

Implementations:

ISIS (Alard; http://www2.iap.fr/users/alard/package.html)

hotpants (Andy Becker, LSST, SuperMACHO, etc)

PSF-matched subtraction

Given two images with different PSFs, want to solve for the convolution kernel that matches the PSFs between the two images

Define the convolution kernel as a linear combination of basis functions

Alard chose Gausians modified by polynomials, Gi(u,v)ujvk

An alternative is to use a delta function basis set (convolution is fast)

Choose reference points (“stamps”) that are assumed not to change

Solve for the kernel

Generate least-squares matrix+vector and invert

Apply solution to stamps: reject, replace, and repeat

Convolve entire image

Subtract

Status

Prototype implementation with delta function basis: pois

Built on pslib

Quality appears comparable to ISIS

Scientific result: Hjorth et al., “The optical afterglow of the short g-ray burst GRB 050709”, 2005, Nature, 437, 859)

Design in PS Modules SDRS

Both ISIS and pois kernels

Implementation by MHPCC team currently untested

To do

Test MHPCC implementation

Careful attention to masking (Becker)

Extend to convolving weight images (Becker)

Extend to perform KL decomposition on convolution kernel solution

Link with psphot

Careful attention to PSF for photometry (Stubbs)

How do we recognise bad subtractions?

An individual bad subtraction (e.g., a cosmic ray)

An entire frame didn't subtract well (catastrophic failure)

Apply to test data

IfA Deep survey, POMME, SuperMACHO, ESSENCE

Need to talk to Becker, Stubbs some more; observe SuperMACHO operation

Infrastructure Components

Nebulous : the IPP Distribute Image/File Server

Metadata DB

PanTasks : the IPP Scheduler / Parallel controller system

DVO : the Desktop Virtual Observatory (IPP Object DB)

Nebulous

User-space distributed object storage system

Can store many redundant copies of an object (implementation limit is 264)

Simple client/server design

All client <-> server IPC is done via SOAP over HTTP

Depends on local filesystems for data storage

Depends on localhost to provide a network file transport (nfs, sshfs, etc.)

Provides advisory locking

Very light weight

Yet another wheel

Distributed in the “wrong” sense

AFS (http://openafs.org/)

Coda (http://www.coda.cs.cmu.edu/)

Distributed/parallel in the “correct” sense

Intermezzo (http://www.inter-mezzo.org/)

Lustre (http://lustre.org/)

OCFS (http://oss.oracle.com/projects/ocfs/)

OCFS2 (http://oss.oracle.com/projects/ocfs2/)

PVFS (http://www.parl.clemson.edu/pvfs/)

PVFS2 (http://www.pvfs.org/pvfs2/)

GFS (http://www.redhat.com/en_us/USA/home/solutions/gfs/)

GPFS (http://www-03.ibm.com/servers/eserver/clusters/software/gpfs.html)

ZoneSRB* (http://www.sdsc.edu/srb/)

Honeycomb** (vaporware - http://sunflash.sun.com/articles/83/1/storage/13982)

EMC Centera** (http://www.emc.com/products/systems/centera.jsp)

Many, Many more...

All but one identifiable solution requires expensive hardware solutions for redundancy

* Can store redundant copies **Selectable Reed-Solomon coding (http://en.wikipedia.org/wiki/Reed-Solomon_error_correction)

Yet another wheel (Cont.)

Nebulous must be able to meet the software requirements for the IPP Image Server:

Select the volume that a specific copy of a storage object (file) will be stored on

Store redundant copies of the same file transparently

Map a symbolic name to the on disk storage file name

Provide a seekable standard C filehandle (PSDC-430-005 section 3.4.1.3)

A lot of consideration went into this requirement

Store 2 x 108 storage objects (PSDC-430-005 section 3.4.1.5)

None of the investigated solutions can meet all of the requirements

ZoneSRB is probably the closest contender out of the lot and appears to have improved since the first evaluation 2 years ago. The licensing restrictions may be problematic for Pan-STARRS' partners

Almost all of the systems investigated require complex setup procedures

Almost all of the systems lock you into specific, non-standard, versions of software (e.g. Linux kernel)

Nebulous can use almost any of these solutions as a data storage and/or transport

Nebulous' symbolic file name mapping will still be required

The client API would be relatively easy to implement as a wrapper around another technology if need be

Server makes use of Apache2 for HTTP handling

Server is implemented as a set of stand alone Perl modules with a separate mod_perl2 interface module

Design is modular so that other protocols would be easy to implement.

Makes heavy use of MySQL 4.1 features (transactions, etc.)

Multiple Servers are possible via MySQL database replication

File locations are specified as URIs

Data structures are modeled directly on the Linux 2.6 implementation of ext3

[Symbolic|Hard] links and other special files are possible (but unimplemented)

Server Implementation

Client Implementation

Native C and Perl clients

Perl SOAP API is hardwired (easy to modify)

C SOAP API is machine generated from a WSDL file by gSOAP (http://www.cs.fsu.edu/~engelen/soap.html)

C client (nebclient) library is “autotooled” and only depends on libc so it is independently distributable

Client Code Example

#include <stdio.h>

#include <stdlib.h>

#include <unistd.h>

#include <nebclient.h>

#define NEB_SERVER "http://example.org:80/nebulous"

#define NEB_KEY "myNewFile.fits"

int main (int argc, char **argv) {

nebServer *server = NULL;

char *key = NEB_KEY;

server = nebServerAlloc(NEB_SERVER);

if (!server) {

printf("nebServerAlloc() failed\n");

exit(EXIT_FAILURE);

}

int fh = nebOpenCreate(server, key, 0, NULL, NULL, NULL);

if (fh < 0) {

printf("nebCreate() failed: %s\n", nebErr(server));

exit(EXIT_FAILURE);

}

close(fh);

if (!nebReplicate(server, key, NULL, NULL)) {

printf("nebReplicate() failed: %s\n", nebErr(server));

exit(EXIT_FAILURE);

}

exit(EXIT_SUCCESS);

}

#!/usr/bin/env perl

use strict;

use warnings FATAL => qw( all );

use Nebulous::Client;

my $neb = Nebulous::Client->new(

proxy => 'http://example.org:80/nebulous'

);

my $key = 'myNewFile.fits';

my $fh = $neb->open_create($key);

die "can't create file $key " unless $fh;

close($fh);

if (!$neb->replicate($key)) {

die "can't replicate object $key";

}

Regression Testing

Testing via Apace::TestMB module

A complete Apache2/mod_perl2 configuration is generated and an httpd process is started on an ephemeral port

Server & Client are tested together

Apache::TestMB has been subclassed and taught to understand the how to build/test “autotooled” packages so both the Perl & C clients are tested

Not using a mock database object / A running MySQL 4.1 instance is required

Future Improvements

Volume watermarking / load rebalancing (needed for production use)

VFS integration via FUSE (http://fuse.sourceforge.net/)

“Hotspot” optimization

Special file support

Zeroconf or SLP for automatic server discovery

Authentication

Shell utilities

Optimization

xattrs

psDB

pslib

psDB

Relational database abstraction facility

Core psLib facility

Data I/O with relational databases from C is tedious & error prone

It requires the programmer to have a high level of working knowledge about the SQL dialect the database speaks

Most interfaces are exceedingly verbose. For example, inserting data with a prepared statements requires the programmer to:

input/generate a SQL string

prepare the statement

pack the input data into a buffer

bind the statement parameters

execute the statement

check the result

commit the transaction

close the prepared statement

For many operations, it's a highly repetitive process. I.e. Lots of cut 'n pasted code.

Rational

Design

Currently only support for MySQL is implemented. Support for other backends should be relativity easy to add.

Some similarities with the Perl/Python DBI interfaces but with a higher level of abstraction.

Relational features are not exposed. Represents the database as a simple persistence mechanism. This is a 80/20 solution. The other 20% required specialized code.

All access to the database, including connection setup and tear down is mediated by psDB* functions

SQL is dynamically generated from data input as a psLib psMetadata type

Most data I/O is via psMetadata or arrays of psMetadata types

Spec'd in psLib SDRS (PSDC-430-007 section 3.6.4)

Example Code

psDB *dbh = psDBInit(host, user, passwd, dbname);

psDBCreateTable(dbh, "mytable", mdTableDesc);

arrayOfMd = psDBSelectRows(dbh, "mytable", mdWhereDesc, limit);

psDBDeleteRows(dbh, "mytable", mdWhereDesc, limit);

psDBDropTable(dbh, "mytable");

psDBCleanup(dbh);

etc.

psMetadata *where = psMetadataAlloc();

psMetadataAddS32(where, PS_LIST_TAIL, "foo", PS_META_REPLACE, "==", 32);

psMetadataAddStr(where, PS_LIST_TAIL, "bar", PS_META_REPLACE, NULL, NULL);

psMetadataAddStr(where, PS_LIST_TAIL, "baz", PS_META_REPLACE, NULL, "%Red%")

Would be translated into the SQL snippet:

WHERE foo=32 AND bar IS NULL AND baz LIKE '%Red%'

GlueForge & the Metadata DB

pslib

psDB

metadataDB

Table Specification

GlueForge

GlueForge Code Generation Utility

Tables described in psMetadataConfig language (spec'd in psLib SDRS PSDC-430-007 section3.6.4)

Code generation is done via a series of templates (Template Toolkit - http://template-toolkit.org)

Code is emitted as a complete “Autotooled” package capable of building and installing shared/statics libs. Also includes Doxygen generated man pages, a scheme specific regression test suite is also generated, and pkgconfig support.

Provides tables specific datatypes (struct)

Usage Example

Valid psMetadataConfig formatted file

Multiple tables specified in the same configuration file

Table indexes may be optionally applied

A “hidden” index field is added to all tables

There is also a 'simple' format for describing a single table

glueforge METADATA

pkg_name STR foodb

pkg_namespace STR foo

END

foo METADATA

foo STR 60 # Primary Key

bar S32 0

baz F32 0.0

boing F64 0.0

zot BOOL t # Key

END

bar METADATA

zot BOOL t # Key

boing F64 0.0

baz F32 0.0

bar S32 0

foo STR 60 # Primary Key

END

Generated Package

foodb/:

Doxyfile.in Makefile.am autogen.sh configure.ac foodb.pc.in src tests

foodb/src:

Makefile.am foodb.c foodb.h

foodb/tests:

Makefile.am dbsetup.c insertobject.c popobject.c

alloc.c droptable.c metadatafromobject.c selectrowsfits.c

cleanup.c init.c objectfrommetadata.c testsuite.at

createtable.c insert.c pop.c

dbcleanup.c insertfits.c popfits.c

Generated Code

fooRow *fooRowAlloc(const char *foo, psS32 bar, psF32 baz, psF64 boing, bool zot)

{

fooRow *object;

object = psAlloc(sizeof(fooRow));

psMemSetDeallocator(object, (psFreeFunc)fooRowFree);

object->foo = psStringCopy(foo);

object->bar = bar;

object->baz = baz;

object->boing = boing;

object->zot = zot;

return object;

}

static void fooRowFree(fooRow *object)

{

psFree(object->foo);

}

Metadata DB

100% generated by GlueForge

Translates 274 lines of description into ~18K lines of code including whitespace (find . -name "*.[ch]" -exec cat {} \; | wc -l)

Currently about a dozen and a half tables are defined

Table configuration is generated by a simple m4 script processing a collection of psMetadataConfig files (.md)

Is completely redistributable without any dependency on GlueForge

DVO

Based on DVO running at CFHT

Used there for Standard Star analysis and archiving

DVO needed upgrades to meet Pan-STARRS requirements

higher precision fields in the tables

more tables / fields

support for different types of measurements

support for proper-motion & parallax

richer metadata

chip and mosaic astrometry

more flexible table definitions

structural improvements

memory management

code under CVS with tags

higher throughput capabilities

DVO : Data Tables

Tables are autocode-defined

Images

Photcodes

Transparency

Cameras

Astrometry

Main Observational Data

SkyRegions

Filters

Static Objects	Average

Detections	Measure

Object Data distributed on sky

Static Objects	SecFilt

Detections	Extend

Non-Detections	Missing

Detections	Moving

Moving Objects	Orbits

ImageRegions

Other Data

DVO : data organization

some tables are partitioned by sky region

sky regions are RA,DEC bounded

sky regions completely defined by source table

hierarchical table grouping (eg: fullsky, dec bands, ra segments...)

table associated with a host (using an abstracted name)

skyregions

images

images

objects

objects

objects

detections

detections

detections

objects

detections

objects 1

objects 2

objects 3

objects 4

images 1

images 2

DVO Data Formats and Modes

multiple disk modes for the layout of the data are available

Raw Binary

FITS Tables (SPLIT or MEF)

MySQL

multiple external representations supported

automatic table translations

Internal Struct

External Struct

A

convert

External Struct

B

External Format

fits

FITS

RAW

SQL

External Format

raw

External Format

mysql

table defs

DVO : addstar

a stand-alone program for tests

library version not yet implemented (in progress)

loads data per chip or mosaic in one pass

optionally loads external data (reference stars)

can read from 2MASS, USNO A,B, Tycho, HST-GSC

single program or client/server

matching criteria:

nearest neighbor

all nearest neighbors (not recommended!)

match radius may scale by positional sigma

DVO : addstar client / server

server has two threads

thread A accepts incoming data

thread B load the data

client sends data and exits

multiple server model

server hosts spatially partitioned tables

loads own tables

sends other servers data

negotiation of overlaps is a separate process

database

addstar

server

addstar

client

database

addstar

server

database

addstar

server

addstar

client

addstar

client

addstar load tests

defined 4 artificial dataset:

103, 104, 105, 105.5 stars / degree2

45 x 45 image array, (15 x 15 degrees)

10 iterations each

Images added to DB

Populated Object Tables

addstar load test - pt2

blue: read database

black : match detections

red line: write database

1000 det/deg2 yields ~4000 det/sec

10,000 det/deg2 yields ~19,000 det/sec

caveat: plots have a fraction of double-counted detections

addstar load test - pt3

100,000 det/deg2 yields ~86,000 det/sec

330,000 det/deg2 yields ~53,000 det/sec

note: last run used 4x4 table re-gridding

Required Throughput: ~50,000 det / sec!

DVO : relphot

determines calibrations from overlaps

solve for Mcal: Mcal + Mrel = Msys

given image (i), filter (j), referenc (r):

Minst = -2.5*log[counts]

Mcat = Minst + 2.5*log(exptime) + C(j) + K(j)(sec z - 1)

Msys = Mcat + f[A(j),(color - <color>)]

Mrel = Msys - Mcal(i)

Mave = mean (Mrel)

Mref = Mave + C(r) + f[A_i(r),color(r)]

exclusions:

area (>20 pix from edge)

saturation (Minst > -15)

type (dophot == 1)

cloudy images / nights

dMcal > 0.015

abs(Mcal) > 0.1

dM > 0.02

M-Mrel > 3sigma

star chisq > 10

Photflat Correction

Flat-fields correction based on stellar photometry

Will use higher density dithers than megacam

Will build high-density photometric reference field

Will enable photflat construction from single observations

Megacam photflat dither pattern

DVO : uniphot

performs regularization of zero-points for disconnected regions

assumes a consistency time-scale

outlier rejections

operates on image table

updates object and detection calibrations

minimize Mcal scatter for contiguous

time blocks to solve for region offsets

DVO shell : visualization

rich data language for data visualization

Region 0.0 25.0 90.0 sin

style -c red; cgrid

style -c black; images

plot-landolt

plot-sdss

region -25.5 -12.8 6.0

style -c blue; pcat -all

style -c black; images

DVO shell : data extraction

color-color diagram: ccd i - J_2MASS : J_2MASS - K_2MASS

CMDs

light curves

other data vectors

image parameters

etc, etc

PanTasks

Regularly-schedule task evaluation

Tasks (potentially) spawn jobs

Jobs may be local or parallel

Jobs may be targetted to specific machines

task datalist

command ls /data/foo

periods -exec 5.0

periods -timeout 50.0

periods -poll 1.0

task.exit 0

queueprint stdout

queuedelete stdout

end

task.exit 1

queuepush failure "task failed"

end

end

task datalist

periods -exec 5.0

periods -timeout 50.0

periods -poll 1.0

task.exec

$file = `next.file`

if ($file == "none")

break

end

command cp /data/foo/$file /data/bar

end

task.exit 0

queueprint stdout

queuedelete stdout

queuepush copied $file

end

task.exit 1

queuepush failure $file

end

end

pcontrol server

CheckControl

CheckTasks

CheckChild

task queue

job queue

user cmds

readline

PanTasks : Process Loop / User Interface

User issues commands, loads scripts via readline

Client / Server Model designed, not yet implemented

run task prep

construct job cmd

submit job

check task timer

next task

check job timer

check job status

return job results

next job

PanTasks : Task Loop vs Job Loop

Limited number of tasks / jobs per interrupt cycle

maximum of one evaluation per cycle

task / job queues are continuously cycled

idle

Pcontrol Host Queue & States

new

down

done

off

busy

delete

USER: host -on (name)

USER: host -delete (name)

USER: host -off (name)

USER: host -off (name)

USER: host (name)

USER: host -off (name)

LOOP: StartJob

PanTasks : Pcontrol Host States

Pcontrol monitors hosts

hosts may be added / deleted

'down' hosts are automatically re-attached

ssh communication to the hosts

busy

new

pending

done

exit

crash

delete

U1

U2

U2

P1

T1

T1

T2

hung

PanTasks : Pcontrol Job States

Pcontrol moves jobs and hosts in parallel

Users may delete pending jobs, kill running jobs, or harvest crash/exit jobs.

PanTasks scheduler is normal Pcontrol 'user'

command src

(scheduler)

pcontrol client

readline

CheckChild

Pcontrol Server

CheckChild

CheckChild

pcontrol client

pcontrol client

PanTasks : Pcontrol Process Loop / User Interface

User issues commands, loads scripts via readline

very similar to PanTasks scheduler loop (same code)

command src

(pcontrol)

child process

readline

CheckChild

Pclient

PanTasks : Pclient Process Loop / User Interface

User issues commands

Pclient launches, monitors background child process

reports stdout, stderr, exit status

very simple command set

PanTasks : loading tests

Pclient : demonstrated job rates of ~500 per second

Pcontrol : tests to manage 62 nodes, ~150 jobs per second total rate

PanTasks : schedule / harvest ~50 jobs per second

Requirement : << 640 jobs / 45 seconds (14 per second)

Pipelines

Metadata Database Tables track state

only final states are saved in the MDDB

intermediate / ephemeral processing states resid in PanTasks

ippTools

user-space programs to access the pipeline MDDB tables

the 'glue' to allow PanTasks to connect analysis steps together

programmed using glueforge to simplify development

sample commands:

p2search -pending

p4search -define -time 2006/11/1 2006/12/1 -region 0 0 10 10

table copy

select list of available tables

select list of uncopied tables

distribute copy jobs across cluster

(Discussion of the DataStore deferred to Wednesday)

pending tables

url, tableID, type, state

copy table

metadata tables

table data

data store : tables

url, tableID, type

get table

list

get table

pending

Pipelines : image copy

select exposures

select files

distribute copy jobs to cluster

target nodes by chip ID (class ID)

new imfiles

url, exp ID, camera,

class, class ID

copy

image file

pending imfiles

url, exp ID, camera,

class, class ID

new exp

exp ID, type, camera,

class, Nfiles

data store : exp (camera)

expID, type

get exp

list

get exp

pending

data store : imfiles (expID)

url, class, classID

pending exp

url, exp ID, type

camera

summit exp

url, exp ID, type

camera, state

get imfile

list

get imfile

pending

done copy

Pipelines : phase 0

register new images

summary statistics

only examine headers

P1/P2 pending exp selected by camera recipe

raw science exp

exp ID, camera,

class, Nfiles, stats

new exp

exp ID, type, camera,

class, Nfiles

P1 pending exp

exp ID, camera, version,

recipe, stats, Nfiles

mosaic

new imfiles

url, exp ID, camera,

class, class ID

raw imfiles

url, exp ID, camera, type,

class, class ID, stats

single

p0search

pending

p0search

update

P2 pending imfiles

exp ID, class, class ID,

url, version, recipe, stats

P2 pending exp

exp ID, camera, version,

recipe, stats, Nfiles

raw detrend exp

exp ID, camera,

class, Nfiles, stats

science

detrend

Pipelines : phase 1

automated vs manual insertion into pipeline

each submission of an exposure gets a new version number

ppImage stats to write to MDDB tables via external ipptools client

ppImage is not linked against the ippdb library

pipeline architecture(s) remain independent of core analysis programs

p1search

pending

P1 pending exp

exp ID, version,

recipe, stats, urls

p1search

define

raw SCIENCE exp

exp ID, camera, version

class, Nclass, stats

from P0

raw imfiles

url, exp ID, camera, type,

class, class ID, stats

P2 pending imfiles

exp ID, class, class ID,

url in, urls out, version,

recipe, stats

P2 pending exp

exp ID, Nclass

P1 & P2 versions

P1 done exp

exp ID, version,

recipe, stats, urls

ppImage

load chips

psphot

psastro

p1search

update

Pipelines : phase 2

manual and automated insertion

P1&P2 versions are tracked

like P1, core tools are not linked against pipeline db library.

we ran the example implementation this past weekend!

p2search

pending

p2search

define

P3 pending exp

exp ID,

P2 & P3 versions

from P1 / P0

raw SCIENCE exp

exp ID, camera,

class, Nclass, stats

raw imfiles

url, exp ID, camera, type,

class, class ID, stats

master detrend frames

det ID, version, label,

type, class, Nclass, filter,

exptime, etc

master detrend imfiles

det ID, class, class ID,

url

P2 pending imfiles

exp ID, class, class ID,

url in, urls out, versions,

recipe, stats

P2 pending exp

exp ID, Nclass

P1 & P2 versions

P2 done imfiles

exp ID, class, class ID,

url in, urls out, versions,

recipe, stats

P2 done exp

exp ID, Nclass, Ndone

P1 & P2 versions

p2search

done

ppImage

load chips

detrend

psphot

psastro

p2search

update

det-

search

Pipelines : phase 3

similar structures to P1, P2

p3search

pending

p3search

define

P3 pending exp

exp ID,

P2 & P3 versions,

urls out, recipe, state

raw SCIENCE exp

exp ID, camera,

class, Nclass, stats

P2 done imfiles

exp ID, class, class ID,

url in, urls out, versions,

recipe, stats, state

P2 done exp

exp ID, Nclass, Ndone

P1 & P2 versions, state

P3 done exp

exp ID,

P2 & P3 versions,

urls out, recipe, state

psastro

from P2

p3search

update

Pipelines : phase 4a

'Magic' is the Boeing Satelite detection algorithm

'Magic' will be managed by PanTasks

Requires 3-stage process

per-image warp/delta

magic

mask and stack

raw, warped

image (P4W)

static sky

image (P4R)

difference

image (P4Δ')

-

=

The Static Sky

poisub

(10%)

magic

destreaked

image & mask

(P4ΔC' + P4M)

Pipelines : phase 4b

After Magic, masks are applied

warped images are stacked

high-quality difference is calculated

Static sky is / may be updated (validation tests are required but details are not hashed out)

raw

stacked

image

static

sky

image

difference

image

-

cleaned

stacked

image

+

The Static Sky

P4W(i) &

P4M(i)

poisub

(1%)

Pipelines : magic summary

Run on the individual P4Δ' images

Performs a hierarchical Huff transform on clipped, binarized images:

individual cells are transformed

small-scale search for short streaks

Huff-space images are grouped 2x2

repeat, go to 4x4, then 8x8

Huff-Transform peaks are used to trigger a line search

cross-track shape is used to minimize false-positives

identified streaks are masked (end-to-end line, jittered position)

only masked images can be distributed

header times are course until masked

total processing time ~120 seconds per FPA on one CPU core

excised pixels << 1% of total.

mkdetrend

build the master detrend images

broken down into several stages

raw exposures

exp ID, stats, state

master detrend run

det ID, type, version

criteria, iteration, state

input detrend exp

det ID, version,

exp ID, camera, state

resid imfiles

det ID, version, type,

camera, class, class ID,

iteration, stats

master detrend imfiles

det ID, version, type,

camera, class, class ID,

iteration, stats

raw imfiles

url, exp ID, camera, type,

class, class ID, stats

groups

group ID, group type,

Nexp

group exposures

group ID, exp ID

dettools

define

master detrend frames

det ID, type, camera,

version, iteration, class

Ninput, Nfiles, state

input detrend imfiles

det ID, version, type

camera, class, class ID,

raw.url, proc.url, state

ppImage

dettools

pend. raw

process

dettools

update

raw

ppMerge

dettools

pend. stack

stack

ppImage

dettools

pend. resid

resid

dettools

update

resid

dettools

assess

resid exp

det ID, version, exp ID,

camera, stats, state

normalize

dettools

pend. norm

normalize

mkdetrend : process

detrend run ID corresponds to selection criteria

re-runs get new version

trials within run get iteration numbers

raw images are pre-processed according to target detrend image types

input detrend exp

det ID, version,

exp ID, camera, state

input detrend imfiles

det ID, version, type

camera, class, class ID,

raw.url, proc.url, state

dettools

pending

raw

ppImage

dettools

update

raw

dettools

define

master detrend imfiles

det ID, class, class ID,

url

mkdetrend : stac/norm

processed images are combined

resulting master images are renormalized (required to combined multiple-chip masters into a common framework)

dettools

pending

stack

ppMerge

dettools

pending

norm

dettools

assess

input detrend imfiles

det ID, version, type

camera, class, class ID,

raw.url, proc.url, state

master detrend imfiles

det ID, version, type,

camera, class, class ID,

iteration, state, stats

master detrend frames

det ID, type, camera,

version, iteration, class

Ninput, Nfiles, state

normalize

dettools

define

mkdetrend : resid

master images are applied to input images

residual stats per chip are combined to judge quality

notes

pre-existing detrend masters can be used to judge input images

choice of running a new master may depend on first-pass tests

master frames may trigger new iteration or may fail complete (insufficient valid inputs)

resid exp

det ID, version, iteration

exp ID, camera, state

resid imfiles

det ID, version, type

camera, class, class ID,

raw.url, proc.url, state

dettools

pending

resid

ppImage

dettools

update

resid

dettools

assess

master detrend imfiles

det ID, class, class ID,

url

input detrend imfiles

det ID, version, type

camera, class, class ID,

raw.url, proc.url, state

dettools

define

master detrend frames

det ID, type, camera,

version, iteration, class

Ninput, Nfiles, state

master detrend run

det ID, type, version

criteria, iteration, state

Pipelines : ippMonitor

ippMonitor

examine statistics of processing

view pending / done tables / subsets

view jpeg images and thumbnails

view detrend residual images and thumbnails

Interfaces : IPP/DataStore

DataStore provides file listing/serving

Otis & Camera place files (images and tables) in DataStore

IPP queries DataStore for new entries by source

GPC-1, SkyProbe, TC-3, Otis Tables, etc

query is http request

first level query returns list of available groups (eg, exposures)

second level query returns list of files in group

URL for second-level query supplied in first level

response includes URL for individual files

IPP requests DataStore entries by URL

responses identify data types (eg, OBJECT, BIAS, table by name)

responses identify class ID (chip00)

copy of DataStore software provides IPP to Summit connection

Data from Summit

Camera images : ICD documents header keywords, layout

Metadata Tables : ICD lists tables, identifies columns

Camera / Otis ICD specifies primary source for duplicate data

in general, camera header data is supplementary

but: some keywords are primary source because of Camera / Otis I/F lags

IPP / MOPS Interface

Quantities to be exchanged are defined

5σ P4Δ detections, not overlapping existing objects

all 3σ P4Δ detections

image metadata to associate detection with image

see IPP SCD & SRS and MOPS SRS

Transport layer is defined: FITS tables

Transport mechanism is not identified (DataStore is likely)

exchange timescale proposed (monthly)

IRS / ICD not yet started

IPP / PSPS Interface

Quantities to be exchanged are defined

see IPP SCD & SRS and PSPS SRS for lists

Transport layer is defined: FITS tables

Transport mechanism is not identified (DataStore is likely)

exchange timescale proposed (monthly)

IRS draft started

IPP / Science Clients interfaces

guidelines are not formalized, but MOPS provides example

max allowed I/O rate constrained by switching fabric

location of Science Clients not specified : impacts available network bandwidth

Software Development Tools

CVS tree @ MHPCC (psLib, psModules)

CVS tree @ CFHT (Ohana)

move to shared CVS tree at IfA (own host)

CVS tree @ IfA

mirrors the MHPCC and CFHT elements

when MHPCC winds down at end of 2006, hosting will move to IfA

Bugzilla

Mail Lists

Better developer communication tools? Wiki?

IPP web pages are improving as software examples and documentation are released

recent web reorganization makes the software hierarchy clearer

Software Build Tools

autoconf for most / configure scripts for all packages

jhbuild allows checkout with dependency tracking

psconfig : simple script to allow multi-library installations

Doxygen is used for auto-generated documentation

Software Configuration Management

psLib / psModules have tagged releases on development cycle

IfA development has followed branch after releases

fixes, improvements pushed into branch for merge with head

during development, high-level packages are more organically released

small developer set

will use scheduled releases as package stabilize

Configuration Control Board defined

charter should be revised in light of changes in project organization

IPP design details will be placed under CCB as they are released for production

parameter files, system configurations will enter CCD after commissioning, as a result of the Operational Readiness Review

psLib API Reports

Determine if the API of the code being produced conforms to the specification

Synchronizing the code with the specification and vice versa

The psLib SDRS that MHPCC is targeting changes approximately every two months

Since the specification is a moving target you can't simply 'check-off' function prototypes and data types as they are completed

This was burning valuable effort within the IPP group and at MHPCC

It turns out humans were doing a pretty bad job

The Problem

The Solution

Specification is written in LaTeX – A markup language

Add custom verbatim blocks ('prototype' & 'datatype') around the relevant parts of the specification with the 'fancyvrb' package

This isn't strictly necessary but very helpful

Develop a utility to list arbitrary blocks in a LaTeX document (texblock.pl)

Compares the psLib API defined in the psLib SDRS (PSDC-430-007) against the actual code in the MHPCC CVS tree

Develop a utility to find and compare the function prototypes and typedefs in two text files of regular C code (apidelta.pl)

Has to be able to tolerate illegal syntax

Develop a utility (in shell) to automatically checkout the needed utilities from CVS, the code from CVS, put the prototype & datatype blocks into a tmp file, put the content of all of the header files into a tmp file, and run apidelta.pl on them

Develop a simple shell script to run this process twice, once for plain text & ANSI color codes, and put the output into files named with the date

This actually works

Fast enough to be run while editing a document – less then 10s on my laptop

Example Output

As of 2006-01-26 at 21:21 (Pacific/Honolulu)

IPP Hardware

Test / Dev Environment

'po' cluster

35 nodes

2 x 2.8GHz Xeons

2GB RAM

3 x 73GB disks / RAID5

IPP dev machines

alala/alae/poma/nkfb0

2 x 2.2GHz Opterons (248s)

2-4GB RAM

3-4TB disk

IPP Prototype node

2 x 2.4GHz Dual-Core Opterons (280s)

4GB RAM

8.5TB

Large assortment of desktop hardware and platforms (x86/Linux, PPC/OSX, etc.)

Very important for robust software

Production Environment

Initially in the PS-1 support building

Transitioning to MHPCC MRTC-B facility as soon as feasible

IFA will be configuring the production environment at MHPCC and then passing control to MHPCC staff

Summit IPP Hardware

IPP 'prototype' node will move from the IFA to the PS-1 Support building computer room in the near future

Additional nodes purchased on an 'as needed' basis

Planning on 5 IPP nodes at the summit for a total of 42.5TBs by Sept. 2006

Planned* Summit computer room thermal envelope can support at least a dozen IPP nodes

IPP presence at the summit is temporary

Kihei IPP Hardware

The complete PS-1 IPP system will consist of 80 nodes

The planned ramp up schedule is different then what was outlined at the PDR

Nodes will be purchased with a complete set of disks and in smaller lots

The total capacity is the same but the time/capacity graph will be much smoother

IPP Node Design

1x Tyan S2892G3NR (S2892-D)

2x AMD Opteron 280

1x Areca ARC-1170

4x 1GB registered 400MHZ DDR DIMMS

1x Chenbro RM519 slimline cdrom & w/ 1350w power supply

25 x Western Digital WD4000KD (1 boot + 24 installed)

Storage Model

Distributed storage

Aggregate bandwidth scales with the number of nodes

Monolithic systems scale up in capacity but not bandwidth

Cost scaling is [almost] linear with the number nodes

Data availability even with storage device failure

Reduced TCO

2:1 data compression, allows 2 copies in the same space

Network Topology

Completely Flat

IPP Cluster is small enough to fit on a single large switch

1 hop between all nodes

Eliminates the need for “closest” neighbor optimization

Almost all IPC is in the form of data

I/O rates allow unified storage and messaging networks

Option of either 2-channel bonded Ethernet or two independent networks

Current COTS equipment can support this model with up to 240 1Gb/s “line rate” ports or 1000+ 1Gb/s “oversubscribed” ports

Switch

Cisco 6513

720 Gb/s backplane

13 slots total

5 slots are 40Gb/s (duplex)

48-port 1Gb/s modules

“Jumbo” frames

Distributed L2 forwarding

RJ-45/Cat6 wiring

Firewall module

Has been received

Kihei Cluster Configuration

Cluster is divided between 'self' booting and netbooting nodes

Control, Database, and Master nodes will be self booting

All other nodes will be netbooting from one of the master nodes

There will be more then one master for performance and redundancy reasons

Failover between masters will require a reboot*

There will be couple of pools of netboot nodes (AP/generic)

Kihei Cluster Management Plan

3-phase life cycle Commissioning/Production/EOL

Commissioning to be done by the IFA

MHPCC will manage the Production phase

EOL phase is not well defined as it is dependent on PS-4 design issues

At a minimum access to the static sky data will be preserved during the transitional period

Will all the bits be transferred to the PS-4 IPP?

Maintenance

Software support will be provided by the IPP group

Hardware will be covered by manufactures warranty

Cisco switch will be covered by a maintenance contract

No catastrophic loss mitigation

Summit <-> MHPCC connection

600Mb/s leased line from HawaiianTel

D-mark interface is GigE but transport is PoS

MTUs > 1500 bytes may or may not be possible

Impact is < 5%

Schedule

1st IPP node at summit in March

nodes at summit ramping through Aug./Sept. 2006

Transitioning to MHPCC as soon as is feasible

4.5 month lead time on the HawaiianTel leased line

IFA Cluster management hand off in Oct./Nov. 2006

Continuous ramp up of nodes at MHPCC through 2007

Testing : Unit Tests

psLib and psModules have extensive unit tests in code tree

psTest structure allows uniform test definitions

standard test program layout

supporting script executes and evaluates test functions

psLib : 40k lines of test code (for 35k lines of src code)

psModules : 7k lines of test code vs 11k lines src

Nebulous & GlueForge have extensive, well-defined tests

DVO and PanTasks need more unit tests defined

Analysis Tests

Test data samples used to test high-level programs

MegaPrime / CFH12K images

data selected to stress specific aspects

Test hardware : po cluster, alala, ipp000 (first production node)

Additional test cluster : fruit bowl at IfA / Hilo

Test data suite being set up under user ipp on clusters

Test infrastructure / tools need development work

Test sequences to be defined under PanTasks (one-off jobs)

Commissioning Plan

Commissioning Plan is Staged

SkyProbe on sky : specific, single-chip tests

TC-3 in lab

data transfer

data load test

bias, dark, non-linear correction construction tests

TC-3 on sky

full Phase 1-3, detrend creation tests

intermediate acceptance criteria less strict than GPC-1

we will not devote resources to TC-3 oddities

GPC-1 in lab

GPC-1 on sky

detrend creation tests

Phase 1-3 tests

Astrometry and Photometry dithers : residuals

Phase 4 tests on dithers

Schedule : 2006

Schedule : 2006

Schedule : 2006

Risks

Development

Phase 4 completion exceeds expected effort

Remaining development schedule slips drastically

Analysis Algorithms

Complete Analysis System fails to meet throughput

OTA mode analysis proves more subtle

PSF from PS-1 is difficult to model

Input Data Quality Issues

CTE, glow, non-linearity, cross-talk, etc...

Scattered light too large / too unstable

Flexure in Telescope or Camera excessive

Image Quality variations excessive

Haleakala conditions worse than expected

Hardware support effort higher than expected

Software support effort higher than expected

2006-01-19T11:00:47