
	Weekly SDAST Chat # 70: Thursday 22 January 2003

Participants: Peter Kretschmar, Sami Maisala, Jerome Chenevez, Stefan Larsson,
	Silvia Martinez-Nunez, Carol Anne Oxborrow, Niels Jrgen Westergaard

Subjects discussed: Status of development and delivery of all pipeline 
			and scripts software
		New ways to check for dead anodes
		New ECAL-MOD tables: ECAL alerts
		Making background models
		`Flower' background in images
		Strange time ranges in binning and source extraction results
		BKG level in pipeline/scripts
		When to update ADD
		jemx.h: update and delivery
		Science Results: EXO2030 appearances and disappearances
			Cyg X-1 spectra
		
Action Items:
		
AI030124.1:	CAO	Feb 2003 Write SCREW and submit templates to extent
OPEN			j_prp_verify so it can produce dead-anode data
			by binning shadowgrams by rawX values

**** Logging Started : Thu Jan 23 14:58:02 MET 2003
<silvia> Hej Carol  Anne !
> Hello everyone, how are we today? END
<silvia> Perfect, it is spring day in th "V" city ! END
> It's NOT spring here! END
<njw> To Siliva: We are still discussing how much our presence at ISDC is 
required
<njw> Perhaps we will continue after March but with a smaller coverage. END
*** larsson (~larsson@tenma.dsri.dk) has joined channel #jemxadr
<silvia> Thanks for the info Niels Joergen ! END 
> Hello Stefan and Sami - it's good to have you both here, now we're just 
waiting
<larsson> hhhiiiiii there! END
<silvia> Hi Stefan and Sami ! END
> for Jerome - isn't he in your office peter? END
<silvia> Carol Anne, Peter is not here, yet ! END
<smaisala> Hi all! END
<silvia> Niels Joergen: have you read my e-amil regardind EXO 2030 ? END
> I wonder where Peter and Jerome have got to - not like them to be late. END
<njw> Yes I have, but unfortunately I've been very tied up with the imaging 
thing.
<njw> Hope to get to science very soon END
<silvia> Niels Joergen: do not worry ! It is just that I am very exciting with 
the science ! END
*** peter (~pkretsch@isdcul9.unige.ch) has joined channel #jemxadr
<peter> Hello, sorry for the delay - seems I'm running from meeting to meeting 
today ... END
*** jerome (~jerome@uhuru.dsri.dk) has joined channel #jemxadr
<jerome> Hello, Hej, bonjour...
> Hi guys! It's good to `see' you all again. Let's get on with the meeting! END
> Any more agenda points people want to add at this time? END
<peter> You think we don;t have enough ;-) ?
<larsson> no
<silvia> No . END
<njw> N
<jerome> N
> Okay, I'll kick off with a quick summary of where I've got to and what I'm
> doing. First data preparation:
> j_prp_status_table was redelivered last week and can handle most of what's
> thrown at it, but Tess sent me an email to say that she's had one set of
> SVT-F data where it missed a grey value somewhare - it sounds like a very 
> minor problem that shouldn't be time consuming, and isn't high priority.
> j_prp_verify has a couple of problems - too many `bad OBT' alerts when there 
are
> data gaps and too many dead-anode alerts. We're thinking of scrapping the
> dead-anode checking function in favour of an offline tool looking at all the
> slew data for a revolution instead, with some human checking thereafter.
> This will be a little time yet.
> j_calib_gain_fitting was redelivered 2 weeks agon
> but apparently isn't been used in operations, so the same old `can't fit these
> spectra' rubbish is currently being churned out. This is for ISDC to solve.
<peter> Unfortunately I have to confirm that j_calib_gain_fitting 6.2 seems 
still in use. A glitch here (the software was supposed to be updated this 
perigee passage).
<peter> It will be solved ASAP - which might mean next perigee. END
> Other than non-useage, I think j_calib_gain_fitting is all correct now and
> currently needs no work.
> j_calib_adc is working fine in its current version. The alerts from this
> morning are only due to using an obsolete -DEAD-MOD table, so I'm working
> on a new one. In fact the table I sent NJW before Christmas would not have
<peter> Tess just installed j_calib_gain_fitting-7.0, so most of Rev 34 should be fine. END
> produced these spurious alerts, however I'm making a table now with much
> wider tolerances. I'm also going to look into the time-variation in our
> ECAL results: I think we've got a steady drift of the offsets upwards, so I
> think the long-term behaviour should be looked at a little. Other than that
> no current work to be done on J_calib_adc. However, the instrument team are
> planning a change in the ECAL procedure which I BELIEVE can be handled smoothly
> by the software, but only time and experience will tell.
> For data correction: j-cor_position's just fine. I'm currently working on 
> j_cor_gain to implement the `decay' model of the gain time variation, and
> also the 16bit STATUS flag (see below). This will be several more days yet:
> I want it well tested before delivering which will add a day or so.
> Finally j_dead_time_calc I hope to start major re-working after delivering
> j_cor_gain: it's not clear how we're going to calculate deadtime now that
> Soren and Niels have found that we're getting quite a few double triggers
> in the hardware trigger counter. They're thinking about how to deal with
> this while I finish j_cor_gain.
> Basically there's a fair amount of work to do, but I feel I have control
> over it. Any questions? END
<njw> N
<larsson> N
<silvia> N
<jerome> Yes, 1, but I am waiting for my turn to speak about dead anodes. END
> Okay, next up to bat: Sami, on background handling automatic and otherwise. 
END
<smaisala> I'm still working on bkg models. The main problem is to separate 
instrument and diffuse bkg
<smaisala> There is quite lot of variations of spectral shapes in different 
parts of detector and I'm trying to find solution to select the 'best' spectras 
to fill the spectrum map.
<smaisala> I'm going next week to ISDC and I hope after this very productive 
week we will have the first bkg models and software to test. END
> Might I suggest that you make sure all the energy/spectral data you use for
<njw> Niels Lund is also trying to do that separation - in a different 
framework, though -
> this analysis come from J_calib_gain_fitting 7.0, else you're just looking at
> rubbish much of the time, which might explain large differences in spectral
<njw> so at some stage in the near future I think it will be a good idea to 
make a comparison END
> shape from one anode segment to another. END
<smaisala> CAO I'm using the version 7.0 and blank field observations from 
rev0024. END
> That's good to hear Sami - I'd hate to find all your work wasted by that
> appalling version 6.x! END
> Is that all from you Sami? Any questions? END
<smaisala> The idea of comparision of the results  could be really nice! 
<smaisala> I think that's all now. END
> Yes, I agree two independant appraisals would be very valuable. END
> Okay next at the plate: NJW and imaging END
<njw> I have now a running version of j_ima_refined (I have not checked the 
results
<njw> yet, that's what I'm doing at the moment besides talking to you). 
j_ima_refined
<njw> is the Niels Lund version of the imaging (IBAS reworked) with IROS i
ncluded.
<njw> I hope to be able to deliver a version early next week. END
<silvia> Sorry ! I have not helped you with that ! As I said, in our last SDAST 
Meeting ! Could I still help in someway ? END
<njw> Thanks Silvia, but I fear that it is not so easy, but I'll keep the offer 
in mind END
> Any more on imaging NJW? Any questions ? END
<silvia> You are welcome ! END
<njw> Not for the moment END
<silvia> Niels joergen,do you think that the flower structure is indeed the mask
 geometry ? END
<silvia> It seems IMA is not removing all the pattern ! END
<njw> The three-fold symmetry is certainly from the mask. Perhaps these flowers 
will
<njw> be absent in the new imaging END
<silvia> That is great ! END
<peter> OK, but what is the run time of the new imaging? Comparable or much 
longer?
> I'll miss the flowers......END
<silvia> Not me Carol Anne, not me ! END
<peter> We need an imaging tool here that works for QLA & SA and does not hide 
weak sources in very systematic structure.
<peter> Do I gather correctly that currently no one really knows where it comes
 from? END
<njw> I have invested my efforts in developing the new executable in stead of
<njw> digging into the old one. That might become necessary if there are 
problems with
<njw> execution times. Else yes, no one knows for sure END
<peter> So do you have  numbers for the execution times? 
<njw> Not yet END
<peter> Are you saying we should foresee the possibility of ditching 
j_ima_basic_recon and running j_ima_refined_recon instead?
<peter> Do I assume correctly that their outputs are compatible so source 
finding still works? END
<njw> To #1: yes
<peter> Sorry for being so pushy but acording to plan we would want to give 
software to users in <2 months! END
<njw> To #2: There is still some work needed to produce a proper JMXi_SRCL_RES 
END
> But off course we know where the flowers come from: if you rotate the results
<peter> Another question: when Carl made his iamges they looked much smoother. 
what did he do differently? END
> (images) from JEM-X1 you get an identical flower pattern to JEM-X2 - at worst
> we can remove all the systematic structure that's the same in the two
> units once rotated by 180 degrees. END
<peter> Fine, but:
<peter> (a) now we usually have just one :-)
> I'd forgotten that! END
<peter> (b) one can in theory take an (appropriately scaled) "empty field" 
image and subtract that, but then one has
<peter> to be very careful to subtract it correctly. In the sky frame different
 images have the structure rotated.
<peter> One could imagine subtracting such a noise map from the RSTI images 
before storing them and going to the sky images.
<peter> Have you considered that Niels Jrgen? END
<njw> I've considered subtracting an empty field shadowgram before doing the
<njw> image reconstruction - because of the linearity of the process this 
should be
<njw> equivalent to subtracting from the RSTI. END
<silvia> Niels Joergen: Peter and I we did that last Christmas time (to 
substract shadowgram from an empty field), and we are not sure is good way ! 
I will send you some results ! END
<peter> I'm not convinced as according to one test of mine even an almost 
perfectly smooth shadowgram returns a flower after deconvolution. END
<njw> Well, thinking a little bit more, that is perhaps not quite true, since 
the
<silvia> We do not see real differences ! END
<njw> systematic pattern is imposed by the reconstruction, so it is probably a 
better idea
<njw> to do the RSTI subtraction after all END
<peter> Though it would be a clutch, admittedly ... END
> When do you expect to have you program finished Niels Joergen? END
<njw> Some time next week I will have a version that can run and show the 
promises of this
<njw> method. It will take a little longer to get the image in celestial 
coordinates
<njw> and a source list in the correct format
* njw "a little" is probably one week more END
> Any more on imaging? Any questions? END
<silvia> N 
<peter> No
<jerome> N
<smaisala> N
<larsson> n
<peter> So is it my turn now? END
<peter> Taking silence as affirmation ...
<peter> The spectrum and lightcurve _binning_ (as I do not really extract 
anything) tools seem to work fine so far,
<peter> except that this morning when Jrme and I analyzed a possible GRB 
event cutting times with a user GTI
<peter> the output contained a larger timerange than that covered by the data. 
I'll look into this.
<peter> Stefan - by the way for j_src_lc it was even worse, it seems to set up 
the output at full length and then put the small selected time output at the 
beginning.
<peter> And SPR will be written, once I get around to check it out once more.END
<peter> Another problem is that at one point I need to validate the application
 of correction factors. To be done when I get a breath. END
<peter> Next orator please ... END
<larsson> OK Peter, I'll look at it when I have the details. END
<larsson> Unless there are questions to Peter...? END
<njw> N
> N
> Looks like it's your turn Stefan. END
<larsson> Status is: j_src_lc was delivered one and a half week
<larsson> ago. j_src_spectra should also be delivered to clean off
<larsson> the same SPR/SCREWs as _lc.
<larsson> The most important part which is missing right now is
<larsson> the capability of processing REST mode data. Except for
<larsson> bug fixes that is my next task.
<larsson> Last few days I have learnt how to use fitsio so I can read the
<larsson> lightcurve files directly into my timing analysis
<larsson> program. Good for science but also good for all the testing.
<peter> You know that you could link DAL independently from the rest of our 
libraries? 
<larsson> Questions?
<peter> I've done so for PIL once for a tool which had nothing to do with ISDC.
 END
<larsson> OK Peter, I suspected I could but it turned out quite easy to use
<peter> Fine.
<larsson> the fitsio too so I went for that. END
<larsson> (Thanks for the hint anyway.)END
> Just a quick note chaps: I've just had an exchange of emails with Tess, and
> j_calib_gain_fitting 7.0 is now running in the pipeline with the new version 
of
> the gain history table, and every seems to be okay. So we should be getting
> good gain history out automatically now, if not actual good energy values, 
yet.
> The current j_cor_gain uses the new gain history okay, but it doesn't ignore
> gain dips or crazy values at the beginning of the revolution. END
<peter> A note from my side there:
<peter> I've started creating gain history tables for the PV phase with 
j_calib_gain_fitting 7.0 and will make them available to everyone. END
<peter> Carole Anne: your turn on offline tools, I guess. END
> Okay, well, it doesn't look like we'll be needing j_ical_frss any more: all
> the data's there in the gain history table, so you can easily compare time
> intervals of your choice and gain between different anodes. What we do need
> is a tool to sum up the events for given X values to check for dead anodes,
> and this could be incorporated into the tool for finding the collimator 
footprint
> by summing up shadowgrams over all the slew data in a given revolution.
> I envisage two outputs therefore. It's possible that the summed shadowgrams
> for the checking the position corrections (droopy, warped collimator 
footprint)
> will also show up the dead anodes. These things are several weeks from being
> coded - pipeline stuff naturally takes priority. END
<jerome> I'd like to come back to the question I was preparing when you said 
<jerome> something about the offline toollooking at all the slew data
<jerome> The question is: how have you thought to get the necessary data?
<jerome> Are they not private?
<jerome> After speeking with Peter I suggested to get them in the same way 
<jerome> as we get engineering data, or something like that.
<jerome> Was it what you had in mind?
<peter> One could use GPS and GCDE data plus, naturally, the calibration 
observations. END
<peter> This would give you an anode check about every 12 days in normal 
operations. 
<peter> Another option with a different strategy is to create a tool that runs 
in the pipeline and dumps out binned data not usable for normal scientific 
analysis. 
> What about slew data between observations: who do those belong to? END
<peter> Also to the owner! END
<njw> The coming Crab calibration should give a hint if 12 days of interval
> Can we add a new pipeline tool, say in the revolution file pipeline? The 
single
> SCW approach leaves alot to be desired. END
<njw> for anode checking is appropriate. Else the extra tool option should be 
implemented END
<peter> The revolution _file_ 'pipeline' has its actions triggered by the 
arrival of certain files in the raw data.
<peter> It cannot access science data in the ScW pipeline => no chance there. 
END
<peter> But nothing keeps us from providing, e.g. one histogram over RAWX per 
ScW (or slew) and to combine these histograms
<peter> in an offline tool. I'd help in the implementation but need some clear 
ideas of what is needed. END
> In that case I can tweek j_prp_verify, so that is makes all these files and
> an offline tool to compare them. END
<peter> This sounds like a good idea. So we should soon:
<peter> - define what we want
<peter> - setup templates
<peter> - write SCREW(s) accordingly
<peter> and then go and implement those changes.
> Okay, I'll put this on my to-do list END
> Next: Sami how's offline tools working for you? END
<smaisala> I really haven't got a time to put my attention to offline tools. 
I have made earlier one tool to examine bkg database and modify it but it's 
still really early beta version. END
> How's performance monitoring going - does it tell us the things we need to 
> know? END
<peter> I do not see any specific performance monitoring done by anyone. 
<peter> This would be a task of the Instrument Representatives, but I guess 
one would first need to decide what should be checked beyond what can be done 
by OSM and looking at allowed sciencedata. END
<smaisala> I delivered the performane montor collection tool a long time ago. 
Peter send us an email where he said the table is readable with hk osm display 
tool. After that I haven't tested that. END
<smaisala> sorry,performane=performance. END
<njw> Soeren is analyzing the IPF files for this on an almost daily basis END
> Okay, any more on Sami's tools? If not let's move on to Silvia and the 
scripts.
> END
<silvia> Peter, do you want to talk about this ? ENN
<silvia> He delievered a new version yesterday . END
<silvia> delievered = delivered ! Sorry END
<peter> The only change this time is that the scripts now have a hidden 
parameter "instMod" to point directly to the IMOD file one wants,
> Are there any changes we should know about? END
<peter> thus avoiding having to mess with the IC data to test a new Instrument 
Model. END
<silvia> Anymor changes ! Next week, I will start to work again on the scripts 
to perform minor changes. I hope to have a new version in a couple of weeks ! 
END
<silvia> My worry is to introducce BKG step in the pipeline, we have not done 
yet ! END
<peter> Don't worry, i often forget to skip it and it runs fine.
<peter> Just that the results make no real sense yet until Sami has real-world 
models. END
<peter> Sorry for skipping back to a previous item:
<smaisala> That nice to know that the executables are working! END
<silvia> Since last SDAST meeting, we have delivered two versions, one on 12th 
of DEc. and the one fom yesterday. In the firts one, the main change was to 
introducce the new GUI interface ( I suppose you have seen already) and some 
minor changes ( parameter checking and nearly all) END
<silvia> Thanks Peter ! END
> Is that all for scripts? END
<silvia> I think so ! END
<peter> I think yes. We plan to improve various points (like actually having 
matching .txt files ...)
<peter> But in general things work fine. END
<jerome> Yes, I can confirm that! END
<silvia> Any questions ? END
> Now ADD updating: do we have a date set? I can't find one on my calender:
> is there anyone out there who knows when we're supposed to have it updated? 
END
<silvia> Sorry for being so chaotic in the scripts comments. Tomorrow the 
Science Minister is coming, and Victor wants him in our desk. The clen woman 
just came,complaint because she has to clean our office ! I am very sorry ! END
<silvia> clen= clean END
<njw> I have a faint notion that it should be done early February END
<jerome> I thougth it was January 20, 2003...
<jerome> END
> Well you can wipe that idea from your mind.....it simply ain't getting done
> then! I think March is a more realistic timeframe, because it always takes
> me at least a week to do all the cross checking with data structures and
> parameters and so on to remove all the obsolete ones. I think we should 
> leave it open then. END
<jerome> Ok. I agree with that, because I am still waiting for some inputs
 about fluxes (NJW, SL, PK :-)
<jerome> END
<silvia> I agree with Carol Anne. I have not yet improved the scripts help 
files ! I will do it soon Peter, I will do it ! END
> Okay, next point: jemx.h update.
> This file hasn't been updated in ages. I'm pretty sure it's got obsolete
> data structure names and stuff in it. Could we, around the time of the
> ADD update make an effort each to look through the current version and come
> with comments, additions and subtractions? Also, has the idea of delivering
> it separately been forgotten. It's currently static state would indicate
> it's ready for this kind of treatment. END
<peter> Not completely forgotten. Maybe we should really get at this. 
<peter> My proposal is:
<peter> - We define a new software component (jemx_lib) or so
<peter> - As a start, this component only contains jemx.h, we could add common 
functions later
<peter> - Developers have a choice between having their local jemx.h and 
mentioning jemx_lib as a dependency on delivery.
<peter> Does everyone agree. END
<silvia> Yes, I do ! I like the idea !END
<njw> Y
<smaisala> Y
<larsson> y
<peter> So when would aim to have a first delivery (I'd do it) - next week 
already? END
> Yes
> I think the file needs a bit of a clean out first. END
> but if you'd like to deliver it as it is.......it has been unchanged since
> June, which probably speaks best for it. END
> Okay, I'll send you the newest version Peter, and you can deliver it just
> to get the ball rolling, but I still say we need an update fairly soon. END
> Next business: science results. How's it going chaps and chapesses? END
<silvia> Trying to understand why EXO2030 appears and disappears from one 
pointing to the other ! END 
<jerome> Yes, good question. And also from one instrument to the other...
<jerome> END
<silvia> In my analysis of revolution 19, the source is detected in the same 
SCWs in both instruments ! END
<peter> A question: has anyone looked into why our Cyg X-1 spectra are so far 
below that of RXTE? END
<larsson> No. END
<jerome> To Sylvia: Not for me!!
<jerome> But I have also it in Scw 90 and 100 for JEMX2.
<jerome> END
<silvia> To Jerome: That is interesting ! I think we should compare our input 
parameters, what do you think ? END
<larsson> I have looked at Cyg X-3 in rev 23. It is detected in both JemX's 
over
<jerome> Yes. I have used 20 and then 10 as detLev. END
<larsson> 7 SCW's but one is REST and one is FULL im. mode. END
<larsson> Unfortunatly I have to leave. There is a small boy waiting for me 
<larsson> (his mother will go riding).
<larsson> END
> have a good weekend Stefan! END
<larsson> The same! END
<peter> Bye Stefan!
<silvia> Enjoy the afternoon and the weekend, Stefan ! END
> Perhaps the rest of us should call it a day too: we'll pick up with this
<jerome> The afternoon is left...
<peter> Just a short note from my side on science results:
> agenda, starting at `Science Results'. Is that amenable to everyone? END
*** Signoff: larsson ()
<jerome> YES. END
<smaisala> Y
<silvia> Y
<peter> In one ScW of the GPS of revolution 30, I'm quite convinced we see 
GX301-2 and halfway convinced that we also pickup 1145.1-5141.
<peter> So the list of sources seen is slowly growing ... END
<peter> Otherwise I agree to call it a day. END
> Is this normal-people-convinced or astronomer-convinced? END 
<peter> Which one is the more strict? END
<njw> I'll also leave you now, have a nice evening! END
<silvia> Lets try to finish ! END
*** njw has left channel #jemxadr : (njw)
<silvia> I have to discuss with the clean woman , what fun ? END
> Have a good weekend everyone - there will be a chat next week - by which 
<silvia> On Science Results, Jerome I' d like to know your input paramters for 
rev. 19 ! And to compare results ! END
> time the wonderful new super-smooth, gain-glitch-free j_cor_gain will be
<jerome> Ok, you 'll get a email. END
> running! END
<silvia> Thanks Jerome ! END
<silvia> Carol Anne has a nice evening and a nice weekend ! END
<jerome> Sounds good CAO. See you next week in our office. END
> Bye everyone! END
