Astrodrizzle - Segmentation fault: 11

mageemagee Member Posts: 2
Hi,

I'm combining thousands of ACSWFC image in a large mosaic (50000x50000 pixels at 30mas/pix) and I'm getting a sigfault 11 with Astrodrizzle. I'm just doing the final drizzle combine pass since the images have already been aligned and cr's have been identified. This is an HST archival proposal project I'm working on where I'm combining nearly all the ACSWFC and WFCIR data taken over the GOODS-South field.

I've combined the same dataset at 60mas/pix (25000x25000 pixels) in Astrodrizzle without problems. In testing, Astrodrizzle fails if I give it a few hundred FLT files as input or just a few (10) using the reference image that is 50000x50000 pixels at 30mas/pix.

Interestingly, I tried using Multidrizzle to do the final image combination using the same 30mas/pix reference image (using 10 FLT image as input) and it generated the 50k x50k pixel mosaic just fine. It seems Astrodrizzle allocates much more memory than Multidrizzle to do the processing. Astrodrizzle estimates the final image size to be ~ 29GB while it is actually more like 9.4GB.

BTW, I'm running this on and 4.0Ghz iMac Retina with 32GB of ram.

Any ideas?

Cheers,
Dan


PS. Here's the python script:

#!/usr/bin/env python

from drizzlepac import astrodrizzle as ad
import os

filters = ['f435w', 'f606w', 'f775w', 'f814w', 'f850lp']
fltd = 'wide_%s' % fltr
os.chdir(fltd)
print '##################### Processing hlf_wide_30mas_%s ######################' % fltr
ad.AstroDrizzle('*flt.fits',
output='hlf_wide_30mas_%s' % fltr,
static=False,
skysub=False,
driz_separate=False,
median=False,
blot=False,
driz_cr=False,
driz_combine=True,
final_wcs=True,
final_refimage='/Users/magee/dr1/Data/REDUCED/hlf_wide_acswfc/align/hlf_final_30mas_wcs.fits',
final_pixfrac=0.75,
resetbits=None,
num_cores=1,
preserve=False,
context=False
)
os.chdir('..')

and here's the astrodrizzle output:

Setting up logfile : astrodrizzle.log
AstroDrizzle Version 1.1.16(30-Dec-2013) started at: 10:33:21.866 (12/02/2016)

==== Processing Step Initialization started at 10:33:21.867 (12/02/2016)
********************************************************************************
*
* Estimated memory usage: up to 28674 Mb.
* Output image size: 50000 X 50000 pixels.
* Output image file: ~ 28610 Mb.
* Cores used by task: 1
*
********************************************************************************
==== Processing Step Initialization finished at 10:33:27.092 (12/02/2016)
==== Processing Step Static Mask started at 10:33:27.093 (12/02/2016)
==== Processing Step Static Mask finished at 10:33:27.093 (12/02/2016)
==== Processing Step Subtract Sky started at 10:33:27.093 (12/02/2016)
==== Processing Step Subtract Sky finished at 10:33:27.83 (12/02/2016)
==== Processing Step Separate Drizzle started at 10:33:27.837 (12/02/2016)
==== Processing Step Separate Drizzle finished at 10:33:27.837 (12/02/2016)
==== Processing Step Create Median started at 10:33:27.838 (12/02/2016)
==== Processing Step Blot started at 10:33:27.838 (12/02/2016)
==== Processing Step Blot finished at 10:33:27.838 (12/02/2016)
==== Processing Step Driz_CR started at 10:33:27.839 (12/02/2016)
==== Processing Step Final Drizzle started at 10:33:27.839 (12/02/2016)
Segmentation fault: 11

Comments

  • mcaramcara Member Posts: 4
    edited March 2016
    I am trying to estimate memory usage for final step and I do not find anything wrong with AstroDrizzle estimation.

    A 50,000x50,000 image will have 50000.**2/2**30=2.3283 Gpix. Each floating point (32-bit) or integer pixel will use 4 bytes => a single image will use 9.3GB of memory. In the final stage astrodrizzle creates the following images: science, weight, and context. Thus, for 10 input images, AstroDrizzle will require 27.94GB. If you have "hundreds" of input images, AstroDrizzle will have to create additional images to store context data. I think a relatively accurate estimation of memory usage would be:

    totmem=9.313226GB*{1+1+ceilling(N/32)}, where N is the number of input images.

    In the second term (curly bracket), first "1" is for SCI image, second "1" is for WHT image, and ceilling(N/32) gives the number of image planes required to store context image. The reason your final drizzled image is 9GB versus the reported 29GB (as you mention in your question) is because you do not have "build=True" set in astrodrizzle which would produce just a single output FITS file with extensions for SCI, WHT, CTX images. When build=False, Astrodrizzle creates three distinct FITS files for SCI image, WHT image, and CTX (context) image. In this later case, the size of the SCI image will be just 9.3GB.

    One way to reduce memory usage is to turn off creation of the context image by setting context=False in AstroDrizzle.

  • mageemagee Member Posts: 2
    @mcara thanks for the clarification on AstroDrizzles memory estimation.

    I'm now processing the dataset by breaking up the data into a 2x2 (25k^2 pix) grid and will then stitch the it back together for the final mosaic.
Sign In or Register to comment.