Overheads with NOTCam. The latest public overhead values for NOTCam are the ones on the page http://www.not.iac.es/instruments/notcam/guide/observe.html#overheads which were based on measurements made in Jan-Feb 2007. Now I have made a detailed study of how the NOTCam overheads vary with time, and I have checked large data sets from the following dates: NCpa14 14 Jan 06 First night with MEF, new UIF, new comp. marissa (BIAS upgraded on 28/8-06) NCqb03 03 Feb 07 old data acquisition system + old AG status NCqd29 29 Apr 07 old data acquisition system + old AG status NCqe03 03 May 07 old data acquisition system + old AG status NCqe27 27 May 07 old data acquisition system + old AG status NCqe30 30 May 07 old data acquisition system + old AG status NCqh28 27 Aug 07 old data acquisition system + old AG status NCqk30 30 Sep 07 old data acquisition system + old AG status NCqk16 16 Nov 07 first tests of sequencer + new AG status NCql13 13 Dec 07 sequencer (+nice'd uif) + new AG status NCra16 16 Jan 08 sequencer (+nice'd uif) + new AG status NCra17 17 Jan 08 old acq. system + nice'd uif? + new AG status NCra20 20 Jan 08 old acq. system + nice'd uif? + new AG status NCtf22 22 Jun 10 sequencer on computer camilla NCub19 19 Feb 11 talker and obslog out of marissa NCxf15 15 Jun 14 elena moved from SB to dome NCxf16 16 Jun 14 storing files on marissa, not on elena I have looked at the BIAS log files. These specify the time spent on readout, the time spent on data storage, and the total time spent on the expose command. Then I have read the FITS header keywords TM_START, TM_END, and EXPTIME. For each image, I compare the overheads as given by BIAS with the overhead as they look from the time stamps in the FITS header. I also flag images accompanied by a PC-board error message. Total time (from BIAS log) spent on a command (seconds): ------------------------------------------------------------------------ Date | frame 4.2 10 | exp 5 | mdark 0 5 | frame 6 10 | | typical max | typical max | typcial max | typical max | ------------------------------------------------------------------------- 14 Jan 06 | 52 52 | 14 15 | ~ 10 13 | | 29 Jan 07 | 58 58 | 15 15 | ~ 11 14 | 71 71 | 03 Feb 07 | 58 59 | 16 16 | ~ 11 14 | | 29 Apr 07 | 96 96 | | 17 21 | | ! 27 May 07 | | 18 | | 97 158 | ! 30 May 07 | 96 135 | | 18 21 | | ! 28 Aug 07 | | 16 | | | 02 Sep 07 | 59 106 | 16 | 12 15 | | 30 Sep 07 | 58 93 | 16 19 | 12 15 | | 16 Nov 07 | 82 126 | | 16 21 | | 13 Dec 07 | 62 91 | 16 | 12 16 | 79 105 | 16 Jan 08 | 68 106 | 17 19 | 14 18 | | 17 Jan 08 | | 15-16 21 | | 73 111 | 20 Jan 08 | | | | 73 113 | 21 Jan 08 | | 17 | | | ------------------------------------------------------------------------- 22 Jun 10 | 59 59 | 14 15 | 10 14 | 76.5 77 | 19 Feb 11 | 52 52 | 14 15 | 10 11 | | 15 Jun 14 | 52 53 | 14 14 | 10 11 | 69 69.4 | 16 Jun 14 | 52 52 | 14 14 | 10 12 | 68 68 | ------------------------------------------------------------------------- Remarks on total time spent: 1) Since about April 2007 there are occasionally (and sometimes very frequently) much longer times than the typical ones (see max). These are typically not(!) correlated with BIAS error messages of the kind: "PC-board reading error m!!!" or "PCboard timeout!!" or "PC-board reading error BUFZ!!!". 2) Overheads have increased steadily with time. Best values are for the date 14/1-2006 which is when we introduced MEF headers, the new UIF, and transition from elizabeth to marissa computer. It makes no sense to look at older values, so this becomes the reference for comparison. Something happened around 29-Apr-2007, giving substantially larger overheads. This is not understood. According to Jacob no changes were made to the system around these dates. Also, we note from that after this date we have frequent events where much longer total times are needed, especially for the frame mode. 3) On Nov 16 we used the sequencer for the first time. Now even longer times were needed. After Jacob "nice'd" the notcam-interface process this improved a bit on the 13 Dec, but increased again a bit on the 16 Jan. The question is why the overheads are longer with the sequencer and whether "nicing" the interface earlier would have given even lower nominal overheads for the old system. Looking at data from 17-21 Jan (old acq. system used) it does not look like the overheads are any lower than before. (Jacob: I assume that you nice'd the interface for both systems?) Typical overheads to readout (seconds): --------------------------------------------------------------- Date | frame 4.2 10 | exp t | mdark t 5 | | bias head | bias head | bias head | ---------------------------------------------------------------- 14 Jan 2006 | 40 48 | 7-8 8-9 | 8-9 8-9 | 29 Jan 2007 | | 7-8 | 7-8 | 03 Feb 2007 | 40 | 7-8 | 7-8 | 29 Apr 2007 | 77 85-88 | 13-14 11-12 | 13-14 11-12 | ! 30 Apr 2007 | | 13-14 | | ! 03 May 2007 | | 13-14 | 13-14 | ! 27 May 2007 | | 13-14 | | ! 30 May 2007 | 77 | 13-14 | 13-14 | ! 28 Aug 2007 | | 7-8 | 7-8 | 02 Sep 2007 | 41 | 7-8 | 7-8 | 30 Sep 2007 | 40 50 | 7-8 8-9 | 7-8 8-9 | 16 Nov 2007 | 62 73 | | ~12 12-13 | 13 Dec 2007 | 43 53 | 7-8 8-10 | 7-8 8-10 | 16 Jan 2008 | n.a. 59 | 9-10 10-12 | 9-10 10-12 | ---------------------------------------------------------------- 22 Jun 2010 | 29.5 48 | 5.4 8-9 | 5.4 7-8 | ? BIAS output wrong? 19 Feb 2011 | 38 48 | 6-7 8-9 | 6-7 8-9 | 15 Jun 2014 | 38 48 | 6-7 8-8 | ~7 8-9 | 16 Jun 2014 | 38 48 | 6-7 8-9 | 6-7 8-9 | ---------------------------------------------------------------- Remarks on readout overheads: 1) BIAS logged values ("bias") are compared to header time stamps ("head" = TM_END - TM_START - EXPTIME). NB! Note that the numbers given for frame mode are total time spent reading out, which is done while integrating, and here "head" = TM_END - TM_START. 2) In general there is good agreement between "bias" and "head" values for the exp/dark/mexp/mdark commands, deviating by 1-2 seconds only. For the frame mode, however, the time stamps from headers give ~10 s more overhead than output from BIAS. The value "bias" includes the real readout time of the controller (i.e. 3.6s x 11). The value "head" includes also the time needed to read the PC board and do the regression analysis. Remember: the first readout (reset frame) is done while integrating in this mode, so only 3.6s extra should be spent on readout. But reading the PC-board is estimated to take 4.5 - 5 seconds, while reading the detector takes 3.6 s. Therefore there is some extra delay included in exposures "frame t N" with t < 5 sec (see NOTCam BIAS documentation for more info). 3) Readout takes longer time with the sequencer. However, after Jacob "nice'd" the notcam-interface, the readout overhead diminished again (on the 13/12), although seems to have increased again on 16/1 for the frame mode. Typical overheads to data storage (seconds): ----------------------------------------------------- Date | 4 MB file | 25 MB file | | bias head | bias head | ----------------------------------------------------- 14 Jan 2006 | ~1.5 2-3 | 3-4 4-5 | 29 Jan 2007 | ~ 2 | 7-8 | 03 Feb 2007 | ~ 2 | 7-8 | 29 Apr 2007 | 2-3 3-4 | 8-9 11-12 | 30 Apr 2007 | 2-3 | | 30 May 2007 | ~ 3 | 8-9 | 28 Aug 2007 | ~ 3 | 02 Sep 2007 | ~ 3 | 7-8 | 30 Sep 2007 | ~ 3 3-4 | ~ 8 11-12 | 16 Nov 2007 | ~ 3 4-5 | ~10 13-14 | 1st sequencer test 13 Dec 2007 | ~ 3 3-4 | 7-8 11-12 | 16 Jan 2008 | ~ 3 3-4 | 7-8 12-13 | ----------------------------------------------------- 22 Jun 2010 | ~ 3 ~4 | 9-10 13-14 | 19 Feb 2011 | ~1.5 ~2 | 3-4 7 | 15 Jun 2014 | ~1.5 ~2 | 3-4 6 | 16 Jun 2014 | ~1.2 2-3 | 2.6 4-5 | ----------------------------------------------------- Remarks on overheads to storage: 1) Whenever the expose commands are given consequtively (from a script) the difference between one TM_END and the next TM_START includes 1) the data storage time, 2) the telescope dither (if present) and 3) any other command in between (such as for instance clear-array). In the above values from the FITS header ("head") only consequtive commands without dithers have been used, and all clear-array commands are flagged and subtracted. The "head" values are compared to the "time to write to disk" output in the BIAS log ("bias"). 2) The data storage time is too long! And why has it increased so much since the 14 Jan 2006? 3) The header estimates give slightly larger overheads than the output from BIAS, typically 1-2 seconds more for 4MB files (exp/dark/mexp/mdark images) and typcially 3-4 seconds more for 25 MB files ("(d)frame 4.2 10" images). These offsets are quite systematic and constant since 2007 (they were much smaller in Jan 2006). On the 13/2 2006 Jacob "added some debugging code to keep a backup of the image in case 'not2mef' crashes and makes a zero image". This may be the cause. 4) It is clear that the NOTCam sequencer introduced 16/11 is slower than the old system also regarding data storage. However, after Jacob had "nice'd" the notcam-interface, these values went back to the typical values (see 13/12). Typical overheads to dither (seconds): --------------------------------------------------------------------------- Date | 10" ag-off | 10" ag-on | 15" ag-on | 240" ag-on | | min max | min max | min max | min max | --------------------------------------------------------------------------- 14 Jan 2006 | 2 6 | | 2 7 | | 03 Feb 2007 | | | | | 29 Apr 2007 | | | | | 30 Sep 2007 | 2 8 | 3 11 | | 6 14 | 16 Nov 2007 | 3 6 | | 5 15 | | 13 Dec 2007 | 3 6 | 5 15 | | 8 19 | 20 Jan 2008 | | 2 11 | | 8 17 | 19 Feb 2011 | 3 9 | 7 8 | 8 11 | | --------------------------------------------------------------------------- Remarks on dither overheads: 1) As above, when the commands are given consequtively (from a script) the difference between one TM_END and the next TM_START includes: 1) the data storage time, 2) the telescope dither (if present) and 3) any other command in between (such as for instance clear-array). In the above dither overheads no "clear-arrays" were used, and the time to store has been subtracted based on the previous average estimates of data storage time (file size dependent and therefore varies with the expmode). 2) After the modification of the TCS autoguider status on Nov 15, such that it waits until the star is within 0.5" (OR - if it does not get there - it waits 5 sec) before giving "ag-on" feedback to the instrument, it is clear that teloffset overheads with autoguiding increased by 2-4 seconds on the average for a 10-15" dither, and by 2-5 seconds for a 240" dither. (It is still not known whether this is good enough for very good seeing, so it may become slower even.) Conclusions, suggestions and questions: --------------------------------------- 1) All overheads are now quite large and have increased since Jan 2006. Our task is to find out why they increased and what we can do to reduce them. 2) Clearly, we need to add overhead checks in the quality control data analysis to keep track of the overheads to readout and storage. 3) In general there is a gradual increase over time, except for the anomalous behaviour on 29 Apr 07 and on the 16 Nov 07 (first night with the sequencer). What happened to the system around 29 Apr? Here it was only the readout overheads that incrased, not the data storage overhead. All data from 29 Apr to 30 May have longer readout overheads. 4) In the above table of total times we note that after 29 Apr 07 the excursions away from the typical values are large and frequent. Is this because of an increasingly problematic communication with the ISA PC-board? 5) Why is the data storage time so long, and why has it increased so much since Jan 2006? What can be done to speed it up? 6) I am concerned about the frame mode. The minimum time it takes to do a "frame t N" command is found to be: (t x N) + dt when t > 5 sec. The dt is the overhead where one would expect 3.6s to one readout (since the other readouts are interlaced with integrations), a given time for file storage (lately: typically 7-8s if N=10), and 1-2 s to regression analysis, i.e. dt=13s. On 14 Jan 2006 this dt was of the order of 9-10 seconds or less. Now it is minimum 13s and frequently as high as 30-50 seconds. Why? The cause of this must be searched for. 7) After gaining shorter dither overheads with the new TCS when it was realized that the guide probe could move faster, we have again increased the dither overheads on 15/11-07 with the improved (wait for centering on star) autoguider status feedback. But this is the price to pay for accurate pointing and optical quality in good seeing. So the only thing to do here is to finetune this procedure. 8) As pointed out before, the optimal thing is to do the telescope dither while reading and storing, which is what is done with other NIR cameras. How complicated would it really be for us to do this? I understand it means digging into BIAS and subdivide the expose commands into separate commands for integrating/reading/storing. 9) The NOTCam sequencer is clearly slower than the old system. (This has been balanced now by giving higher priority to BIAS process compared to the notcam-interface.) Is it possible somehow to speed up things more? 10) With a new controller we will speed up the readout time from 3.6 to < 1 second, which means effectively from 7.2 to < 2 seconds per exposure for exp/mexp/dark/mdark mode and even less for frame mode. Amanda, Feb 2008. P.S. Later measurements added in for comparison. Amanda 2011, 2014.