Despeckling
These are taken from Vyacheslav’s instructions, and adapted where necessary.
Prepare data for despeckling
Create a directory on one of sr20, sr30 hosts
First, log in to on of sr20 or sr30.
ssh -X tusys@srXX
Then make a directory in which you will despeckle. We usually use the date of the observation as the name of the directory. For example:
mkdir /cluster/sr20/2009-06-22
Create subdirectories
cd /cluster/sr20/2009-06-22
mkdir RAW
cd RAW
mkdir Align C1 C2 C3 C4 C5 C6 C7 flat
The seven directories below RAW/ correspond to the seven cameras, Align/ is for Align data, flat/ is for raw flats and darks.
Copy raw camera images to the cluster
Decide for which cameras and which fits files you want to despeckle.
There are several utilities available to help selecting files.
Then, for each camera, cd to the corresponding C# directory (where # is the number of the camera),
and run the combine_fits routine. An example:
cd C1
idl
combine_fits, '/camera/scott/2009-06-22/Sunprobe_v4*.fits', NB, RESBASE='Sunprobe_v4_'
exit
cd ../C2
idl
combine_fits, '/camera/virgil/2009-06-22/Sunprobe_v4*.fits', NB, RESBASE='Sunprobe_v4_'
exit
...
NB is a number of subbursts in one cycle.
RESBASE is a keyword specifying the prefix for the output (copied) files.
If omitted, RESBASE = ‘Sunprobe_v4_’; set it to ‘flat_’ or ‘dark_’ when copying flats or darks. (This means that for copying data you can omit RESBASE (which is recommended since it avoids tipos, but for flat and dark data you have to specify RESBASE always).
Do not forget the underscore at the end fo RESBASE.
The Align files should be simply copied (without using combine_fits.pro) to the Align/ directory.
You need only files from Barium, BaCont, Halpha, RedCont cameras for alignment.
Copy and edit create_fdraw.pro
Copy /cluster/master/Speckle/create_fdraw.pro to the RAW/ directory.
Edit (with vi or nano for example) and adjust create_fdraw.pro (see comments in the file):
specify the ids for the cameras that you wish to reconstruct, camera computer names, camera types, etc.
Run create_fdraw.pro, as a result it will create /cluster/sr20/2009-06-22/raw directory and put there flats and darks, readable by de-speckling procedures.
Alignment
Alignment is a procedure to calculate relative shift, rotation and scaling between the two pairs of cameras for which the indirect (KvdL) reconstruction is applicable.
These ara Barium (ba) and BaCont (bk), Halpha (ha) and RedCont (rc).
Copy alig_ba.pro and alig_ha.pro to the Align/ directory.
You must edit these files and specify the timestamps for Align data, flat and dark (see align_ba.pro and align_ha.pro) bursts.
Run alig_ba and alig_ha in IDL. The alignment images are shown in blinking mode. Press and then the computation is continued and shows two graphs. The alignment results are correct if the two parabolas that are plotted by clearly show minima. If not, note down the estimated minima (x-axis) and change the initial values in the align_ba or align_ha.pro file (mentioned in last part of code), quit IDL, restart and run align_ha or align_ba again. The results are very sensitive to the initial values.
Create directory ‘/cluster/srXX/yyyy-mm-dd/Par’
Copy ‘/cluster/master/Speckle/Par_new/allvars_ha.dat’ and ‘…ha1.dat’ into ‘…/Par’
Pick up output values: shift & shc, and put them as I_sx & I_sy parameters in Par/allvars_ha1.dat and Par/allvars_ba1.dat with the opposite sign!
Pick up output values: scaling and rotation, and put them as I_Rot and I_Scale parameters in Par/allvars_ha1.dat and Par/allvars_ba1.dat without changing the sign!
Configure variables for each camera
Par/allvars_##.dat, where ## is camera id (see camparams.pro).
Note that there are different config files for KvdL reconstruction: allvars_ba1.dat and allvars_ha1.dat.
They contain additional data like alignment, number of wavelengths, etc.
In principle, you only need to specify the following variables:
N_Image = 100 ;;; # of images per burst
N_Flat = 100 ;;; # of flats in burst
N_Dark = 100 ;;; # of darks in burst
In addition, for kvdl reconstruction, specify also:
I_Rot = -0.2029 ;;; Image rotation, given by align_ba.pro/align_ha.pro
I_Scale = 0.9971 ;;; Scaling, given by align_ba.pro/align_ha.pro
I_sx = 109 ;;; Image X shift given by align procedure
I_sy = 72 ;;; Image Y shift given by align procedure
I_wsteps = 6 ;;; Number of wavelengths in a scan
I_lams = [0,0.075,-0.075,0,0.04,0.04] ;;; Wavelengths of the scan
Set up main config
Copy from ‘/cluster/master/Speckle/’ the main config file doall_full.cfg. Comments see inside.
In principle, you have to specify only the following parameters:
base = 'Sunspot' -- prefix for the output file
position = {lat: 10, lon: 10, mu: 0.1} -- position of the target on the disk
Cams = ['C1','C2','C5','C4','C6','C7'] -- array of cameras (correspond to the directories in RAW subdirectory). Note, that 'C5' -- BaCont is put before 'C4' -- Barium, because for kvdl the continuum must be reconstructed first.
Id = ['ca','gb','bk','ba','rc','ha'] -- camera id's
camtyp=[4, 4, 4, 4, 5, 5] -- camera types (see camparams.pro)
safe = [2, 2, 2, 2, 2, 2] -- don't know, don't change it, just keep the array of the same size as all others
pext = ['','','','1','','1'] -- Extension of the allvars file, Ex.: allvars_ba1, allvars_ba. Must be '1' for kvdl
kvdl = [0, 0, 0, 1, 0, 1] -- =1 for kvdl reconstrustion, 0 otherwise
lam = ['-0000','-0000','-0000','-scan','-0000','-scan'] -- output filename suffix
gamm = [[.09,340],[.0,0],[.0,0],[.0,0],[0.,0],[0.,0]] -- linearity, makes sense only for Ca
filt = [1, 1, 1, 1, 0, 0] -- Specifies, whether to filter the raw data against the electronic interference. Not required for Ha and RC cameras.
Running the reconstruction
“Speckle reconstructing” cluster consists of:
- Master computer dspmaster
- Two “blades”: sr20, sr30
- Unknown number of computing nodes: no21, no22, …, no51, no52, …, no56 (not all of them may be available at all times)
At the moment it is not necessary to log in to dspmaster or the computing nodes. One can directly log in to a “blade” and start a despeckling session from there.
Log in to srXX
From jeff, log in to one of the “blades”, and start a screen session:
ssh tusys@sr20(log in)screen(start a screen session)
The screen program allows you to disconnect from the blade without stopping the processes you have started inside the session.
Documentation for this program can be found here: http://aperiodic.net/screen/.
Start despeckling
On sr20 start IDL and run doall_full in /cluster/sr20/2009-06-22/
cd /cluster/sr20/2009-06-22
idl
.r doall_full
After a few minutes the program STOPs.
Create a new screen window (presses CTRL+A, then c).
In this new window, go to the Reko directory and start the rekowatch program:
cd /cluster/sr20/2009-06-22/Reko
/cluster/master/Speckle/rekowatch
Then, use CTRL+A 0 to go back to the initial screen window where IDL is running,
and type .continue.
Watch results
To check new FITS files available, you can create a third screen window and run these commands:
cd /cluster/sr20/2009-06-22/FITS
watch -n 120 'ls -lt *fits|head -10'
You can see how far the current burst is being despeckled by going to the window where rekowatch is running.
Error checking
Sometimes errors happen during despeckling.
Carefully check all the peremeters in the configuration files.
Carefully check if you haven’t mixed flat/dark data.
Sometimes, it crashes during kvdl reconstruction of bad quality images. In such case:
- kill all idl processes running:
ssh tusys@dspmaster clex ‘invoke-rc.d cluster stop’ no## no## no## no## no## no## clex ‘invoke-rc.d cluster start’ no## no## no## no## no## no##
- exit from IDL in the first screen window, and remove all intermediate files:
rm -rf Reko dosp doraw tmpdir Tars
- start IDL again, run doall_full. After the program STOPS, type:
start =
where the is the next to the one which was processed last, i.e., skip the ‘bad’ burst.
Then .continue IDL.
After despeckling
Check the reconstructed images
The reconstructed images are stored in the /cluster/sr20/2009-06-22/FITS directory.
Pit has two routines for this: at /home/pit/idl/procs/nchk.pro and nchck_ha.pro.
Please, see the source code to find out how to use them. It’s quite easy!
Copy images to dotdb
Connect as a root user to jeff, sftp dotdb.
Add new data to search engines
Viewing raw data
Raw data can be viewed within IDL. For Redlake camera’s:
IDL>openr, 1, 'Filenamexxxxx.fits'
IDL>pic=assoc(1, { img: intarr(1024,1024) }, 2880 )
IDL>p=pic(0)
IDL>show, p.img
FITS files can be viewed with ds9 (to quit ds9 press control-C)