Despeckling

These are taken from Vyacheslav’s instructions, and adapted where necessary.

  1. Prepare data for despeckling
    1. Create a directory on one of sr20, sr30 hosts
    2. Create subdirectories
    3. Copy raw camera images to the cluster
    4. Copy and edit create_fdraw.pro
    5. Alignment
    6. Configure variables for each camera
    7. Set up main config
  2. Running the reconstruction
    1. Log in to srXX
    2. Start despeckling
    3. Watch results
  3. Error checking
  4. After despeckling
  5. Check the reconstructed images
    1. Copy images to dotdb
    2. Add new data to search engines
  6. Viewing raw data

Prepare data for despeckling

Create a directory on one of sr20, sr30 hosts

First, log in to on of sr20 or sr30.

ssh -X tusys@srXX

Then make a directory in which you will despeckle. We usually use the date of the observation as the name of the directory. For example:

mkdir /cluster/sr20/2009-06-22

Create subdirectories

cd /cluster/sr20/2009-06-22
mkdir RAW
cd RAW
mkdir Align C1 C2 C3 C4 C5 C6 C7 flat

The seven directories below RAW/ correspond to the seven cameras, Align/ is for Align data, flat/ is for raw flats and darks.

Copy raw camera images to the cluster

Decide for which cameras and which fits files you want to despeckle. There are several utilities available to help selecting files. Then, for each camera, cd to the corresponding C# directory (where # is the number of the camera), and run the combine_fits routine. An example:

cd C1
idl
combine_fits, '/camera/scott/2009-06-22/Sunprobe_v4*.fits', NB, RESBASE='Sunprobe_v4_'
exit

cd ../C2
idl
combine_fits, '/camera/virgil/2009-06-22/Sunprobe_v4*.fits', NB, RESBASE='Sunprobe_v4_'
exit

...

NB is a number of subbursts in one cycle. RESBASE is a keyword specifying the prefix for the output (copied) files. If omitted, RESBASE = ‘Sunprobe_v4_’; set it to ‘flat_’ or ‘dark_’ when copying flats or darks. (This means that for copying data you can omit RESBASE (which is recommended since it avoids tipos, but for flat and dark data you have to specify RESBASE always). Do not forget the underscore at the end fo RESBASE. The Align files should be simply copied (without using combine_fits.pro) to the Align/ directory. You need only files from Barium, BaCont, Halpha, RedCont cameras for alignment.

Copy and edit create_fdraw.pro

Copy /cluster/master/Speckle/create_fdraw.pro to the RAW/ directory. Edit (with vi or nano for example) and adjust create_fdraw.pro (see comments in the file): specify the ids for the cameras that you wish to reconstruct, camera computer names, camera types, etc. Run create_fdraw.pro, as a result it will create /cluster/sr20/2009-06-22/raw directory and put there flats and darks, readable by de-speckling procedures.

Alignment

Alignment is a procedure to calculate relative shift, rotation and scaling between the two pairs of cameras for which the indirect (KvdL) reconstruction is applicable. These ara Barium (ba) and BaCont (bk), Halpha (ha) and RedCont (rc). Copy alig_ba.pro and alig_ha.pro to the Align/ directory. You must edit these files and specify the timestamps for Align data, flat and dark (see align_ba.pro and align_ha.pro) bursts. Run alig_ba and alig_ha in IDL. The alignment images are shown in blinking mode. Press and then the computation is continued and shows two graphs. The alignment results are correct if the two parabolas that are plotted by clearly show minima. If not, note down the estimated minima (x-axis) and change the initial values in the align_ba or align_ha.pro file (mentioned in last part of code), quit IDL, restart and run align_ha or align_ba again. The results are very sensitive to the initial values. Create directory ‘/cluster/srXX/yyyy-mm-dd/Par’ Copy ‘/cluster/master/Speckle/Par_new/allvars_ha.dat’ and ‘…ha1.dat’ into ‘…/Par’ Pick up output values: shift & shc, and put them as I_sx & I_sy parameters in Par/allvars_ha1.dat and Par/allvars_ba1.dat with the opposite sign! Pick up output values: scaling and rotation, and put them as I_Rot and I_Scale parameters in Par/allvars_ha1.dat and Par/allvars_ba1.dat without changing the sign!

Configure variables for each camera

Par/allvars_##.dat, where ## is camera id (see camparams.pro). Note that there are different config files for KvdL reconstruction: allvars_ba1.dat and allvars_ha1.dat. They contain additional data like alignment, number of wavelengths, etc. In principle, you only need to specify the following variables:

N_Image   =  100                         ;;; # of images per burst
N_Flat    =  100                         ;;; # of flats in burst
N_Dark    =  100                         ;;; # of darks in burst

In addition, for kvdl reconstruction, specify also:

I_Rot     = -0.2029                      ;;; Image rotation, given by align_ba.pro/align_ha.pro
I_Scale   = 0.9971                       ;;; Scaling, given by align_ba.pro/align_ha.pro
I_sx      = 109                          ;;; Image X shift given by align procedure
I_sy      =  72                          ;;; Image Y shift given by align procedure
I_wsteps  = 6                            ;;; Number of wavelengths in a scan
I_lams    = [0,0.075,-0.075,0,0.04,0.04] ;;; Wavelengths of the scan

Set up main config

Copy from ‘/cluster/master/Speckle/’ the main config file doall_full.cfg. Comments see inside. In principle, you have to specify only the following parameters:

base = 'Sunspot' -- prefix for the output file
position = {lat: 10, lon: 10, mu: 0.1}      -- position of the target on the disk
Cams = ['C1','C2','C5','C4','C6','C7']      -- array of cameras (correspond to the directories in RAW subdirectory). Note, that 'C5' -- BaCont is put before 'C4' -- Barium, because for kvdl the continuum must be reconstructed first.
Id   = ['ca','gb','bk','ba','rc','ha']      -- camera id's
camtyp=[4, 4, 4, 4, 5, 5]                   -- camera types (see camparams.pro)
safe = [2, 2, 2, 2, 2, 2]                   -- don't know, don't change it, just keep the array of the same size as all others
pext = ['','','','1','','1']                -- Extension of the allvars file, Ex.: allvars_ba1, allvars_ba. Must be '1' for kvdl
kvdl = [0, 0, 0, 1, 0, 1]                   -- =1 for kvdl reconstrustion, 0 otherwise
lam = ['-0000','-0000','-0000','-scan','-0000','-scan']  -- output filename suffix
gamm = [[.09,340],[.0,0],[.0,0],[.0,0],[0.,0],[0.,0]]    -- linearity, makes sense only for Ca
filt = [1, 1, 1, 1, 0, 0]                   -- Specifies, whether to filter the raw data against the electronic interference. Not required for Ha and RC cameras.

Running the reconstruction

“Speckle reconstructing” cluster consists of:

At the moment it is not necessary to log in to dspmaster or the computing nodes. One can directly log in to a “blade” and start a despeckling session from there.

Log in to srXX

From jeff, log in to one of the “blades”, and start a screen session:

The screen program allows you to disconnect from the blade without stopping the processes you have started inside the session. Documentation for this program can be found here: http://aperiodic.net/screen/.

Start despeckling

On sr20 start IDL and run doall_full in /cluster/sr20/2009-06-22/

cd /cluster/sr20/2009-06-22
idl
.r doall_full

After a few minutes the program STOPs. Create a new screen window (presses CTRL+A, then c). In this new window, go to the Reko directory and start the rekowatch program:

cd /cluster/sr20/2009-06-22/Reko
/cluster/master/Speckle/rekowatch

Then, use CTRL+A 0 to go back to the initial screen window where IDL is running, and type .continue.

Watch results

To check new FITS files available, you can create a third screen window and run these commands:

cd /cluster/sr20/2009-06-22/FITS
watch -n 120 'ls -lt *fits|head -10'

You can see how far the current burst is being despeckled by going to the window where rekowatch is running.

Error checking

Sometimes errors happen during despeckling.

where the is the next to the one which was processed last, i.e., skip the ‘bad’ burst. Then .continue IDL.

After despeckling

Check the reconstructed images

The reconstructed images are stored in the /cluster/sr20/2009-06-22/FITS directory. Pit has two routines for this: at /home/pit/idl/procs/nchk.pro and nchck_ha.pro. Please, see the source code to find out how to use them. It’s quite easy!

Copy images to dotdb

Connect as a root user to jeff, sftp dotdb.

Add new data to search engines

Viewing raw data

Raw data can be viewed within IDL. For Redlake camera’s:

    IDL>openr, 1, 'Filenamexxxxx.fits'
    IDL>pic=assoc(1, { img: intarr(1024,1024) }, 2880 )
    IDL>p=pic(0)
    IDL>show, p.img

FITS files can be viewed with ds9 (to quit ds9 press control-C)