The XMM-Newton Serendipitous Source Catalogue:
3XMM-DR4The XMM-Newton Serendipitous Source Catalogue:
3XMM-DR4| Release 1.4 | 15 September 2013 | Associated with Catalogue version 1.0 |
Prepared by the XMM-Newton Survey Science Centre Consortium (http://xmmssc-www.star.le.ac.uk/) |
||
This User Guide refers directly to the full FITS and plain-text formats of the catalogue. Users interested in the details of changes to the data processing since the 2XMMi-DR3 catalogue release, can refer directly to section 3. Information about the columns contained in the 3XMM-DR4 catalogue are presented in section 4. Brief summaries of some elements of the 3XMM-DR4 catalogue properties are provided in section 5 but a comprehensive evaluation of the catalogue will be presented in a 3XMM-DR4 catalogue paper, in due course.
3XMM-DR4 is the third generation catalogue of serendipitous X-ray sources from the European Space Agency's (ESA) XMM-Newton observatory, and has been created by the XMM-Newton Survey Science Centre (SSC) on behalf of ESA. The catalogue has 2474 more observations and about 178000 (50%) more detections than the preceding 2XMMi-DR3 catalogue, which was made public in April 2010. Significant changes have been made to the processing system to generate the 3XMM-DR4 catalogue and the changes are described in Section 3
The catalogue contains source detections drawn from a total of 7427 XMM-Newton EPIC observations made between 2000 February 3 and 2012 December 8; all datasets included were publicly available by 2012 December 31 but not all public observations are included in this catalogue. For net exposure time ≥ 1ksec, the total area of the catalogue fields is ~ 1397 deg2 but taking account of the substantial overlaps between observations, the net sky area covered independently is ~ 794 deg2.
The catalogue contains 531261 X-ray source detections above the processing likelihood threshold of 6. These X-ray source detections relate to 372728 unique X-ray sources, that is, a significant fraction of sources (66728) have more than one detection in the catalogue (up to 44 repeat observations in the most extreme case).
The catalogue distinguishes between extended emission and point-like detections. Parameters of detections of extended sources are only reliable up to the maximum extent measure of 80 arcseconds. There are 52168 detections of extended emission, of which 10915 are 'clean' (in the sense that they were not manually flagged) and 7698 comprise the 'cleanest' set where no flags are set and they are not in fields with high background levels
Due to intrinsic features of the instrumentation as well as some shortcomings of the source detection process some detections are considered to be spurious or their parameters are considered to be unreliable. It is recommended to use either a detection flag or an observation flag (and, possibly, a high background flag) as filters to obtain what can be considered a 'clean' sample. There are 432321 out of 531261 detections that are considered to be clean (i.e., summary flag < 3) and 3625 out of 7427 fields are considered to have no, or at most a couple, of spurious detections in them (observation class < 2 and not a high-background field).
For about 123860 detections, EPIC spectra and time series were automatically extracted during processing, and a χ2-variability test was applied to the time series. 4612 detections in the catalogue are considered variable, within the timespan of the specific observation, at a probability of 10-5 or less based on the null-hypothesis that the source is constant. Of these, 2580 have a summary flag < 3.
The median flux (in the total photon-energy band 0.2 - 12 keV) of the catalogue detections is ~ 2.4 × 10-14 erg/cm2/s; in the soft energy band (0.2 - 2 keV) the median flux is ~ 5.7 × 10-15, and in the hard band (2 - 12 keV) it is ~ 1.3 × 10-14. About 20% have fluxes below 1 × 10-14 erg/cm2/s. The flux values from the three EPIC cameras are, overall, in agreement to ~ 10% for most energy bands. The positional accuracy of the catalogue point source detections is generally < 3 arcseconds (90% confidence radius) and 90% of point sources have 1-sigma positional uncertainties < 2.4 arcseconds.
Pointed observations with the XMM-Newton Observatory detect significant numbers of previously unknown 'serendipitous' X-ray sources in addition to the proposed target. Combining the data from many observations thus yields a serendipitous source catalogue which, by virtue of the large field of view of XMM-Newton and its high sensitivity, represents a significant resource. The serendipitous source catalogue enhances our knowledge of the X-ray sky and has the potential for advancing our understanding of the nature of various Galactic and extragalactic source populations.
The 3XMM-DR4 catalogue is the sixth publicly released XMM-Newton X-ray source catalogue produced by the XMM-Newton Survey Science Centre (SSC) consortium. It follows the 1XMM (released in April 2003), 2XMMp (July 2006), 2XMM (August 2007), 2XMMi (August 2008) and 2XMMi-DR3 (April 2010) catalogues: 2XMMp was a preliminary version of 2XMM. 2XMMi and 2XMMi-DR3 are incremental versions of the 2XMM catalogue.
The 3XMM-DR4 catalogue is about 50% larger than the 2XMMi-DR3 catalogue, which it supersedes. This is principally because of the ~3.2-year longer baseline of observations included, though some sensitivity enhancements also contribute. In terms of the number of X-ray sources, the 3XMM-DR4 catalogue is the largest ever produced. 3XMM-DR4 complements deeper Chandra and XMM-Newton small area surveys, probing a large sky area at the flux limit where the bulk of the objects that contribute to the X-ray background lie. The 3XMM-DR4 catalogue provides a rich resource for generating large, well-defined samples for specific studies, utilizing the fact that X-ray selection is a highly efficient (arguably the most efficient) way of selecting certain types of object, notably active galaxies (AGN), clusters of galaxies, interacting compact binaries and active stellar coronae. The large sky area covered by the serendipitous survey, or equivalently the large size of the catalogue, also means that 3XMM-DR4 is a superb resource for exploring the variety of the X-ray source population and identifying rare source types.
The production of the 3XMM-DR4 catalogue has been undertaken by the XMM-Newton SSC consortium in fulfillment of one of its major responsibilities within the XMM-Newton project. The catalogue production process has been designed to exploit fully the capabilities of the XMM-Newton EPIC cameras and to ensure the integrity and quality of the resultant catalogue through rigorous screening of the data.
The 3XMM-DR4 catalogue is the result of a bulk reprocessing of all the available XMM-Newton data. It is based on a pipeline (cat9.0 - configuration 00000004_04_cat9.0_20121220.153800) that contains significant changes to the processing approach to enhance the quality of the catalogue. It makes use of the latest SAS version (12.0.1, but with a small number of specifically updated SAS tasks) and the latest calibration files available at the time of the bulk reprocessing. For the bulk reprocessing run, the pipeline, SAS and calibration components were static throughout the run.
Users of the 3XMM-DR4 catalogue should be aware that the DETID and SRCID values bear no relation to those in the previous 2XMM series of catalogues. However, a cross-matching is provided in 3XMM-DR4 via the DR3DETID and DR3SRCID columns.
The extensive User Guide (UG) for the 2XMM catalogue still describes many of the details of the data processing and compilation approach applicable to the 3XMM-DR4 catalogue. However, a significant number of changes to the processing have been implemented for 3XMM-DR4 and these are described in the following section. For convenience, Table 1, which gives the energy band definitions, is repeated here.
| Basic energy bands: | 1 | = | 0.2 - 0.5 keV | ||
| 2 | = | 0.5 - 1.0 keV | |||
| 3 | = | 1.0 - 2.0 keV | |||
| 4 | = | 2.0 - 4.5 keV | |||
| 5 | = | 4.5 - 12.0 keV | |||
| Broad energy bands: | 6 | = | 0.2 - 2.0 keV | soft band, no images made | |
| 7 | = | 2.0 - 12.0 keV | hard band, no images made | ||
| 8 | = | 0.2 - 12.0 keV | total band | ||
| 9 | = | 0.5 - 4.5 keV | XID band | ||
Following the release of the 2XMMi-DR3 catalogue, the opportunity was taken to review software and calibration upgrades and to explore developments that could enhance the scientific quality of next catalogue. Here we summarise the key changes that have been implemented in to the cat9.0 pipeline that was used to perform the bulk-reprocessing of all XMM-Newton EPIC data for the 3XMM-DR4 catalogue. Further details of these changes are provided in subsequent subsections, along with explanations of other, more general differences for this bulk reprocessing. Section 2 refers back to the 2XMM catalogue documentation, which is a useful reference for other elements of the process that have not changed, and provides a background context for those that have.
1. Use of empirical Point Spread Function (PSF) models (section3.3): Recent, empirical characterisations of the EPIC pn, MOS1 and MOS2 telescope PSFs (contained in the calibration data) have been exploited in the source detection process. These PSFs provide better representations of the spatial profile, polygonal core structures and the spoke patterns of the EPIC PSFs, which helps improve source parameterisation, reduces the detection of spurious objects in the wings of bright objects and contributes to improvements in the astrometric accuracy of sources.
2. Improved field rectification (section3.4.1): The SAS task, catcorr, is used to provide enhanced field rectification (i.e. field frame-shift corrections) against external absolute astrometric reference catalogues. This involves i) the use of 3 possible reference catalogues (USNO B1.0, 2MASS and SDSS (DR8)), ii) improvements to the process of matching XMM-Newton detections with reference catalogue objects and iii) better algorithms for determining the frame-shift and rotation corrections and errors. The result is more fields that can be successfully rectified, greater accuracy of the results and improved error analysis.
3. Exploitation of a time-dependent boresight (section3.4.2): Extensive analysis of the spatial displacements of a subset of 2XMMi-DR3 X-ray detections, with respect to counterparts in the SDSS spectroscopic and photometric quasar catalogues, revealed systematic time-dependent offsets with a semi-amplitude ~ 1 arcsecond. These variations have been used by the XMM-Newton calibration teams to create a time-dependent correction to the spacecraft boresight, which is applied to event coordinates during event processing.
4. Optimised global background flare filtering (section3.2): In previous pipeline processing, background flares were filtered out by deriving Good-Time-Intervals (GTIs) during which high-energy ( > 7 keV) background timeseries count rates, obtained from the whole imaged field-of-view, were below instrument-specific, constant cut thresholds. In the current pipeline, GTIs to exclude flares are derived from a 0.5-7.5 keV (in-band) background flare timeseries from the imaged field of view, from which prominent sources have been excluded. The new algorithm employs the SAS task, bkgoptrate, to determine an optimum cut threshold that seeks to maximise the signal-to-noise of sources in the field. This analysis is applied to each exposure for each instrument and is used to create the background flare GTIs for downstream processing .
5. Optimised extraction of spectra and timeseries (section3.6). Two new approaches are applied to the extraction of EPIC spectra and timeseries from suitably bright detections. Firstly, the SAS task, eregionanalyse, is used to establish the radius of a circular extraction aperture that maximises the total signal-to-noise of the extracted source data - previously a fixed aperture of 28 arcsecond radius was used. Furthermore, the count threshold above which spectra and timeseries are extracted, has been reduced to requiring > 100 'good' spectrum counts for the instrument/exposure (previously > 500 EPIC counts were required). Secondly, in previous pipelines, in some circumstances, especially objects located in the central window in MOS Small-Window mode observations, the annular background region used, which was co-centred with the source, could yield empty, or near-empty, background spectra, which would cause downstream problems for spectral fitting and timeseries analyses. In the current cat9.0 pipeline, the location of a circular background region is moved around the source in radial and azimuthal steps until an acceptable fraction ( > 70%) of usable background area is identified.
6. Automated flagging of detections affected by Out-of-Time and RGA scattered light features (section 3.7): A simple algorithm has been implemented in the SAS task, eootepileupmask>, to detect the presence of piled up sources and create a mask that indicates where associated Out-of-Time (OoT) events may be inadequately modelled, which can lead to spurious source detections. The same SAS task also attempts to identify the linear features in MOS images arising from X-rays from bright objects that are scattered by the RGAs. These masks are subsequently used by the SAS task, dpssflag, to set a flag (flag 10) for detections whose centres lie on such features.
7. Improved pn low energy noise filtering (section3.5): The SAS task, epreject is used to suppress, in particular, the detection of features in pn band 1 images that arise from Minimum Ionising Particles (MIPs)
8. Improved detection matching for unique sources (section3.8): A revised approach has been adopted for matching separate source detections in to unique sources on the sky. This employs Bayesian methodology.
9. Addition of fractional excess variance measures (section3.9): The fractional excess variance (and error) is computed by the SAS task, ekstest, for each extracted EPIC timeseries; the catalogue provides values, where this value was computable, for the most variable exposure available for each instrument. This provides a measure of the amplitude of variability above statistical fluctuations.
XMM-Newton observations considered for inclusion in the 3XMM-DR4 catalogue were those with ODFs available for processing up to 2012 December 08 and which had public release dates up to 2012 December 31. No observations taken after the suspected MOS1 event on 2012 December 11 (revolution 2382) have been included. After allowing for a small number ( < 100) of observations which failed in processing for a variety of reasons, 7427 observations were available to make the 3XMM-DR4 catalogue. Table 2.1 gives the list of the final 7427 observations which are included in the 3XMM-DR4 catalogue.
3XMM-DR4 contains 2505 observations that are not in 2XMMi-DR3, of which around 440 were public and available for inclusion in 2XMMi-DR3 but were not included in 2XMMi-DR3. Many of these latter cases are fields with high backgrounds, while others are cases that failed in previous processing but processed satisfactorily during the processing for 3XMM-DR4. There are 31 observations in 2XMMi-DR3 that did not make it in to 3XMM-DR4, mainly due to software/pipeline errors during processing. Typical examples of the latter problems are due to revised ODFs (e.g. with no useful time-correlation information), more sophisticated SAS software that identified issues hitherto not trapped, or issues with exposure corrections of background flare light curves and pn time-jumps.
In previous pipeline processing, high-energy ( > 7 keV) background light curves were created from the whole field of view and GTIs were then created that only included intervals where the count rate was below a pre-defined, constant, instrument-dependent threshold (see User Guide (UG) for the 2XMM catalogue (sections 3.1.1 a, b and c). For 3XMM-DR4, this process has been replaced by one which creates a 0.5-7.5 keV background timeseries (per instrument), from which prominent sources are removed, and then determines GTIs based on count-rate thresholds that maximise the signal-to-noise (S/N) of sources in the field. Since the changes involved in this optimised process are substantial, and, in-part, instrument-specific, the steps are detailed below (for MOS and pn separately):
1. A first pass constructs a high energy background lightcurve selecting events with the (XMMEA_22 = REJECT_BY_GATTI & XMMEA_EM = GOOD_MOS_EVENTS) flags set.
2. GTIs are made by filtering the high energy background lightcurve using a rate-cut level derived by an optimisation process (described below) using the SAS task bkgoptrate. These GTIs are used to define the time regions in which bad pixel searching occurs.
3. All GTIs with a duration of less than 100 seconds are excluded.
4. The SAS task embadpixfind is used to locate dark pixels in each MOS CCD (using events which have not been filtered through the flare GTIs).
5. If no flare GTIs were made, the SAS task embadpixfind is used to locate bright pixels.
6. If flare GTIs do exist the events are filtered through the flare GTIs and then embadpixfind is used to locate bright pixels.
7. The SAS task badpix is run on each CCD event file in order to add a bad pixel extension.
8. The intervals in the global GTI file are aligned with the event list and merged with the CCD GTIs.
9. Attitude correction is applied to the individual events to convert raw CCD pixel coordinates, through camera coordinates, to celestial coordinates.
10. Raw event pulse height values are converted to rectified event energies.
11. Unwanted events are filtered out before lists are merged.
12. The per-CCD event lists are merged into one per camera.
13. Good imaging events are filtered into final event lists.
14. The Calibration Index File (CIF) is copied into a separate extension in the event list.
15. A second pass high energy background lightcurve is made, selecting events as before, but now with the bad pixels excluded.
16. Flare GTIs are created using bkgoptrate from this background light curve.
17. An image is created in the 0.5 - 7.5 keV band using pattern ≤ 12 and ((FLAG & 0x766aa000)==0), with the flare background and attitude GTIs applied.
If the image contains at least 1000 counts, steps 18-21 are run
18. The SAS task, eexpmap is used to create vignetted and un-vignetted corresponding exposure maps. Also emask is run to generate a detector mask.
19. eboxdetect is run in local mode to detect sources in the image.
20. The ftool, fselect, is run to create separate lists of faint (count rate < 0.35) and bright (count rate > 0.35) sources.
21. The SAS task, region is used to generate regions for detections from each list (using a radius of 60 arcseconds for faint sources and 100 arcseconds for bright sources). Only sources with detection likelihoods > 50 are considered. These bright and faint region files are merged in to a single region file.
22. A 0.5 - 7.5 keV (in-band), 'source-free' flare background light curve is now generated, filtering with PATTERN ≤ 12, (FLAG & 0x762ba000)==0) and, if regions were created, excluding events from within the identified source regions. The bin width is 26s.
23. The light curve is exposure corrected via epiclccorr (without absolute corrections).
24. bkgoptrate is applied to the in-band, 'source free' flare background light curve to derive the optimised count-rate cut level.
25. tabgtigen is used, with the optimised count-rate cut level, to define the final flare background GTIs.
26. The event files are filtered through GTIs into final event files.
1. The SAS task badpixfind is run to create a mask of non-source pixels to be used in generating a high energy background lightcurve.
2. badpixfind is run on each CCD to locate bright and dead pixels.
3. epreject is run to suppress pn low energy detector noise
4. Attitude correction is applied to the individual events to convert raw CCD pixel coordinates, through camera coordinates, to celestial coordinates.
5. Raw event pulse height values are converted to rectified event energies.
6. Events are filtered by selecting events with the (XMMEA_EP = PN_GOOD_EVENTS) flags set.
7. The CCD event files are filtered on the HK GTIs and merged into one.
8. The CIF is copied into a separate extension in the event list.
9. A high energy background lightcurve is made using events with energies between 7 keV & 15 keV and excluding bad pixels by using the previously created pixel mask.
10. The SAS task, epiclccorr is used to exposure correct the high-energy background light curve.
11. The SAS task, bkgoptrate is run to derive the optimised count-rate cut.
12. Initial flare GTIs are derived using tabgtigen and the optimised cut level
13. An in-band (0.5 - 7.5 keV) image is created, using RAWY > 12, PATTERN ≤ 4, (FLAG & 0x2fa002c) == 0) (for 0.5 - 1.0 keV events) or (FLAG & 0x2fa0024) == 0 (for 1.0 - 7.5 keV events), together with the attitude and initial flare GTIs.
If the image contains at least 1000 counts, steps 14-17 are run
14. eexpmap is used to create vignetted and un-vignetted corresponding exposure maps. Also uses emask to generate a detector mask.
15. The SAS task, eboxdetect is run in local mode to detect sources in the image.
16. The ftool, fselect, is used to create separate lists of faint (count rate < 0.35) and bright (count rate > 0.35) sources.
17. The SAS task, region is used to generate regions for detections from each list (using a radius of 60 arcseconds for faint sources and 100 arcseconds for bright sources). Only sources with detection likelihoods > 50 are considered. These bright and faint region files are merged in to a single region file.
18. A 0.5 - 7.5 keV (in-band), 'source-free' flare background light curve is now generated, filtering with PATTERN ≤ 4, ((FLAG & 0xfffffef) == 0 and, where regions were created, excluding events from within the identified source regions. The bin width is 26s.
19. The light curve is exposure corrected via epiclccorr (without absolute corrections).
20. bkgoptrate is applied to the in-band, 'source free' flare background light curve to derive the optimised count-rate cut level.
21. tabgtigen is used, with the optimised count-rate cut level, to define the final flare background GTIs.
The optimisation process is based on the approach used by Pye et al., 1995 (their section 3.3) for ROSAT. The signal-to-noise (S/N) of an object with constant count rate, RATE_S, can be expressed as
S/N = ( RATE_S * Texp ) / SQRT [SUM (RATE_B(i) * Texp(i) ]
where Texp is the total exposure time remaining in the light curve at any particular stage in the process, RATE_B(i) is the background rate in time bin i and Texp(i) is the exposure within time bin i. The SUM is over all time bins that remain at the same point in the process. Although the S/N is a function of the source count rate, the shape of its dependence on the background, which is the important factor, is not, so RATE_S is dropped from the equation.
To determine the optimised count rate cut (threshold) level, the background light curve is sorted in to ascending count-rate order (no binning). In an iterative process, high background count-rate points are successively removed and the S/N computed for the remaining data at each step. This process is conducted until all but the last (lowest count-rate) point have been removed. The S/N is stored as a function of the current background rate threshold at each step. Once completed, the curve is inspected for the maximum S/N (though it must be a true maximum). This maximum thus reflects the background count-rate cut (threshold) that yields the maximum S/N.
The optimisation process is performed independently for each EPIC instrument. The S/N v background rate threshold curve is stored as an extension in the flare background light curve files, along with a keyword containing the optimised count-rate threshold.
To understand how this process works, conceptually, one can envisage a background flare with slowly rising/falling wings. In such a case, permitting a higher cut threshold might allow a substantial gain in exposure time from the intervals spanned by the slowly changing wings, with only a small increase in the background noise, thus improving the S/N of detections
The source-fitting stage of pipeline-processing of data used for the 2XMM series of catalogues employed the Medium accuracy Point Spread Functions (PSFs) for each EPIC instrument, PSF models derived from theoretical ray-tracing of the XMM-Newton mirror system. The source fitting steps are described in Watson et al. 2009 (Section 4.4.3). In recent years, the XMM-Newton EPIC calibration team has developed a set of empirical PSFs, created by stacking images of real XMM-Newton point sources and parameterising these with multi-component models (see Read et al. 2011). In particular, these empirically-based PSF models include characterisation of instrument-dependent, polygonal structures in the core of the PSF and parameterised representations of the complex spoke structures that are caused by the XMM-Newton mirror shell support vanes.
The empirical PSFs have been measured for a number of discrete energies and off-axis angles and the parameters are provided through the calibration system. In the current cat9.0 pipeline, during the source fitting stage, a PSF is constructed for the given instrument, source off-axis position and energy band by linear interpolation of the model parameters of PSFs at nearby grid points.
The use of the empirical PSF has several ramifications. Firstly, the improved representation of structures in the real PSF results in more accurate source parameterisation. Secondly, it helps reduce the number of spurious detections found in the wings of bright sources. This is because the previous Medium accuracy PSFs (which was taken to be the same for all 3 EPIC instruments) did not adequately model the core and spoke features, leaving residuals during fitting that were subsequently detected as spurious sources. With the empirical PSFs, fewer such spurious detections are found, especially for bright objects at larger off-axis angles. Thirdly, as a result of the work on the PSFs, the astrometric accuracy of XMM-Newton source positions has been substantially improved.
A number of significant changes have been made to the processing that affect the astrometry of sources in 3XMM-DR4.
Celestial coordinates from emldetect include a small systemic error arising from uncertainty in the telescope orientation during each observation. These may result in frame shifts, typically of the order of an arc-second in RA and DEC and a field rotation of the order of 0.1 degrees. To correct for these shifts, sources in the XMM-Newton field of view can be cross-correlated with objects from astrometric reference catalogues. X-ray sources with counterparts in the reference catalogue are used to derive the frame shifts and rotation. In all previous pipeline processing (and catalogues derived from those processings) these frame corrections were estimated using eposcorr, which used a single reference catalogue, USNO B1, and evalcorr, to determine the success and reliability of the outcome.
The 3XMM-DR4 catalogue uses results from a pipeline in which these tasks have been replaced by new task, catcorr, which incorporates an iterative fitting function and several other improvements. The cross-match between XMM and reference catalogue source positions is also carried out separately using three reference catalogues: (1) USNO-B1.0, (2) 2MASS and, where sky coverage permits, (3) the Sloan Digital Sky Survey DR8. When there is a good enough fit from at least one catalogue, the frame shift and rotation from the 'winning' reference catalogue are used to correct the positions. These corrected coordinates replace the original RA and DEC values; the original uncorrected coordinates are relegated to columns RA_UNC and DEC_UNC and the index of the catalogue used is provided in column REFCAT. If the best fit has parameter values (e.g. the number of matches used) that fall below specific thresholds then the original positions are retained and REFCAT has a negative value. Further details may be found in the documentation for catcorr. The position correction algorithm is successful in about 83% of observations which contain 89% of detections. Although SDSS (DR8) covers only 34% of the sky, its greater depth means that it has been used for about 45% of the successful position corrections.
It should be noted that a SAS task, catprep, is used to extract subsets of objects from the reference catalogues that lie within the EPIC field of view. However, it is not a public SAS task because it requires local access to the full reference catalogue data, which most users do not have. Nevertheless, each of the reference catalogue extracts (where available) are provided to users of XMM-Newton data products via the file-type=REFCAT product file, which is used by the task, catcorr.
The XMM-Newton SSC, EPIC calibration and SOC teams have analysed the spatial offsets of a subset of XMM-Newton sources from counterparts in the Sloan Digital Sky Survey (SDSS) spectroscopic and photometric quasar catalogues. These analyses revealed a time-dependent, oscillatory variation of the RA and DEC offsets (ultimately found to be due to variations in the Z axis of the instrument plane), with an amplitude of +/- 1.2 arcseconds, superposed on a longer-term trend. The variations, whose origins are uncertain, have been modelled as time-dependent excursions of the spacecraft boresight and characterised by simple functions in calibration.
During pipeline processing of an XMM-Newton observation, corrections for this time-dependent boresight movement are applied to individual event positions based on the observation epochs of the events.
During the late stages of testing of the pipeline used for the bulk reprocessing that fed into the 3XMM-DR4 catalogue, an analysis of 3XMM positions identified a small but significant position shift, primarily along the radial vector from the optical axis to the source, which is a function of off-axis angle. The effect grows from approximately zero arcseconds on axis to ~ 0.65 arc-seconds at off-axis angles of 15 arc-minutes, and is in the sense that the real position is closer to the axis than that measured by emldetect. This radial shift is due to the displacement between the true position of a source and the defined centroid of the empirical PSF, which grows as the PSF becomes increasingly distorted at high off-axis angles. It should be noted that identifying and measuring this effect has only been possible because of the corrections for other effects (see sections 3.3 and 3.4.2 that masked it, and because of the large number of sources available that provide sufficient statistics. In due course a correction for this effect will be applied directly to event positions, on a per-instrument basis, via the calibration system, but for the 3XMM-DR4 catalogue, to avoid a lengthy delay in its production, a solution was implemented within the catcorr task. The correction, computed via a third-order polynomial function, is applied to the initial coordinates of the source output by emldetect, i.e. prior to the field rectification step, based on the off-axis angle of the source as measured from the spacecraft boresight. The coordinates from emldetect, including this correction, are provided through the RA_UNC and DEC_UNC columns in the catalogue.
A second PSF-related problem that affected 2XMMi-DR3 positions was uncovered during early testing of the empirical PSF (see Read et al. 2011). This arose from a 0.5 pixel (equivalent to 0.5-arcsecond) error in the definition of the pixel coordinate system of the medium-accuracy PSF map. When transferred to the image frame during PSF fitting in emldetect, this error in the PSF map coordinate system manifested itself as an offset of up to 0.7 arcseconds in the RA/DEC of a source position, varying with azimuthal position within the field. The introduction of the empirical PSF removes this error.
Some XMM-Newton observations show enhanced noise in the pn instrument band 1 images. This noise manifests itself in two ways. Firstly, 'bright' columns can be seen close to the centre line of the pn detector. Secondly, Minimum Ionising Particle (MIP) events can give rise to clumps of bright pixels. These MIP features, in particular, are prone to being detected as sources, which, being band-1 only detections, can then be misinterpreted, for example, as super-soft-sources.
In the cat9.0 pipeline, the SAS task, epreject, is used during the event processing stage to suppress features caused by MIPs. This removes most of these pn band 1 features.
The pipeline processing automatically extracts spectra and timeseries (source-specific products, SSPs), from suitable exposures, for detections that meet certain brightness criteria.
In previous versions of the processing pipeline, extractions were attempted for any source which had at least 500 EPIC counts. In such cases, source data were extracted from a circular aperture of fixed radius (28 arcseconds), centred on the detection position, while background data were accumulated from a co-centred annular region with inner and outer radii of 60 and 180 arcseconds, respectively. Other sources that lay within or overlapped the background region were masked during the processing. In most cases this process worked well. However, in some cases, especially when extracting SSPs from sources within the small central window of MOS Small-Window mode observations, the background region could comprise very little usable background, with the bulk of the region lying in the gap between the central CCD and the peripheral ones. This resulted in very small (or even zero) areas for background rate scaling during background subtraction, often leading to incorrect background subtraction during the analysis of spectra in XSPEC.
For the bulk reprocessing leading to 3XMM-DR4, two new approaches have been adopted and implemented in to the pipeline.
1) The extraction of data for the source takes place from an aperture whose radius is chosen to maximise the signal-to-noise (S/N) of the source data. This is achieved by a curve-of-growth analysis, performed by the SAS task, eregionanalyse. This is especially useful for fainter sources where the background level is high.
2) To address the problem of locating an adequately filled background region for each source, the centre of a circular background aperture of radius, rb=168 arcseconds (comparable area to the previously used annulus) is stepped around the source along a circle centred on the source position. Up to 40 uniformly spaced azimuthal trials are tested along each circle. A suitable background region is found if, after masking out other contaminating sources and allowing for empty regions, a filling factor of at least 70% usable area remains. If no background trial along a given circle yields sufficient residual background area, the aperture is moved out to a circle of larger radius and the azimuthal trials are repeated. The smallest trial circle has a radius of rc=rb + 60 arcseconds so that the inner edge of the background region is at least 60 arcseconds from the source centre - for the case of MOS Small-Window mode, the smallest test circle for a source in the central CCD is set to a radius that already lies on the peripheral CCDs. Other than for the MOS Small-Window cases, a further constraint is that, ideally, the background region should lie on the same instrument CCD as the source.If no solution is found with at least a 70% filling factor, the background trial with the largest filling factor is adopted.
For the vast majority of detections where SSP extraction is attempted, this process obtains a solution in the first radial step, and a strong bias to early azimuthal steps, i.e. in most cases an acceptable solution is found very rapidly. For detections in the MOS instruments, about 1.7% lie in the central window in Small-Window mode and have a background region located on the peripheral CCDs. Importantly, in contrast to earlier pipelines, this process always yields a usable background spectrum for objects in the central window of MOS Small-Window mode observations.
In addition, the cat9.0 pipeline permits extraction of SSPs for fainter sources. Extraction is considered for any detection with at least 100 EPIC source counts (EP_8_CTS). Where this condition is met, a spectrum from the source aperture (i.e. source plus background) is extracted. If the number of 'good' (in the sense adopted by XSPEC) counts > 100, a spectrum and timeseries are extracted. The initial filter on EPIC counts is used to limit the processing time as, for dense fields, the above background location process can be slow.
A significant issue in terms of spurious detections in XMM-Newton data arises from detections associated with Out-of-Time (OoT) events. For sources that do not suffer significantly from pileup, the background map used by emldetect includes a component that models the OoT features. However, for sources where pile up is significant, the OoT modelling is inadequate. This can give rise to spurious sources being detected along OoT features. For the more piled up objects, the numbers of spurious detections along OoT features can become large (tens to hundreds).
Another feature arising from bright sources that affects the MOS instruments is scattered X-rays from the Reflection Grating Arrays (RGA). These manifest themselves as linear features in MOS images passing through the bright object, rather similar in appearance to OoT features. These features are not modelled at all in the background map.
In previous catalogues, spurious detections associated with OoT and RGA features have simply been masked during manual screening. In the cat9.0 pipeline, for the first time, an attempt is made to identify the presence of OoT and RGA features from piled up sources and to flag detections that are associated with them.
The SAS task, eootepileupmask, is used for this purpose. This task uses simple instrument (and mode) -dependent pre-defined thresholds to test pixels in an image for pile-up. Where it detects pixels that exceed the threshold, the column containing that pixel is flagged in a mask map for the instrument. The task attempts to identify and mask columns and rows associated with such pixels in OoT and RGA features.
Once the pile up masks are generated, the SAS task, dpssflag is used to set flag 10 (see PN_FLAG, M1_FLAG, M2_FLAG, EP_FLAG) for any detection whose centre lies on any masked column or row.
The 3XMM Catalogue is a list of individual detections. Since some fields have partial overlaps with others and some targets have been observed repeatedly, many sources were detected more than once (up to 44 times in the most extreme case). Individual detections have been resolved into unique sources on the sky using the procedure outlined here. It should be noted, however, that in a few cases, nearly all caused by the presence of extended emission, the association of detections to unique sources can be ambiguous. The process used in 3XMM-DR4 has changed from that used for 2XMMi-DR3 (see 2XMMi-DR3 UG section 3.2)
The matching process is divided into two stages. The first stage finds, for each detection, all other matching detections within 15 arc-seconds of it (except those within the same observation). The Bayesian matching algorithm of Budavari and Szalay (2008) was used. This list of pairs was then summed to find, for each detection, the total number of matches, NMATCH, and the sum of the Bayesian probabilities for all its matches, SUMPROB.
In the clustering stage, the list of pairs is sorted in decreasing order of NMATCH with SUMPROB as a tie-breaker, so that detections likely to be at the centre of a cluster, and with the highest total pair probability, come first. The source identifier, SRCID, of all detections is initially set to zero. Examining each pair in turn: if both detections have a SRCID of zero they are assumed to form a new source and an incremented SRCID is assigned to the two detections. If either one but not both detections in a pair already has a SRCID then the other detection is assigned the same SRCID. If both already have a SRCID assigned and it is the same then no action is needed; if they already have two different SRCID values assigned then the CONFUSED flag is set. This algorithm resolves all but a very small number of detections into unique sources. The column N_DETECTIONS is set to the total number of detections comprising each source, while the CONFUSED flag marks those (128 in total) where the detection has a non-zero probability of being a member of two apparently distinct sources.
An IAU identification, IAUNAME, has been assigned to each unique
source (SRCID) based upon the
IAU registered classification, 3XMM.
The form of the IAU names is "3XMM Jhhmmss.sSddmmss" where
hhmmss.s is taken from the right ascension coordinate given in the column
SC_RA and Sddmmss is the
declination taken from the column
SC_DEC of the respective
catalogue.
The correct nomenclature for references to detections in the catalogue
is the IAUNAME followed by a
colon and the detection identification number DETID (with six digits), that is:
"3XMM Jhhmmss.sSddmmss:detid".
New quantities are provided in the 3XMM-DR4 catalogue to provide measures of the magnitude of variability within instrument timeseries. This is done through the calculation of normalised (or fractional) excess variances, in the SAS task, ekstest (see Edelson et al. 2002 and Vaughan et al. 2003).
The variance, S2, of the timeseries, is
where N is the number of points in the timeseries,
xi is the count rate in bin i and <x> is the mean count
rate over the timeseries. The variance is a combination of intrinsic
variations in the source and measurement noise.
The fractional excess variance, Fvar, is computed as
where <σ²err> is taken as
the mean square error of all bins in the timeseries. The error on the fractional
excess variance is
The fractional excess variance is a measure of the amplitude of variations beyond that due to the measurement noise alone, expressed as a fraction of the mean signal.
A fractional excess variance and error are provided for each instrument (PN_FVAR, PN_FVARERR, M1_FVAR, M1_FVARERR, M2_FVAR, M2_FVARERR), where computable. Where a single timeseries is available for the instrument the values are for that timeseries. Where more than one timeseries is available for the instrument, it is provided for the one that has the largest probability of being variable (see PN_CHI2PROB, M1_CHI2PROB, M2_CHI2PROB).
A value is not always computed, even though a timeseries may exist. This may be due to insufficient numbers of points in the timeseries once only data within flare and attitude GTIs are accepted - although the timeseries are created without GTI filtering, GTI filtering is applied when computing the fractional excess variance. Also, for broadly constant, faint sources, the statistical noise can sometimes exceed the variance such that Fvar becomes the square root of a negative number. In such cases no value is computable. It should be noted that the timeseries on which ekstest is run are binned. The bins in a given timeseries are of fixed width but this width is subject to the requirement that, for pn, each bin contains at least 18 counts, whilst for MOS there should be at least 5 counts in each bin. In both cases a minimum width of 10s is imposed.
Summary html pages are provided for each detection. Links to the html summary pages of the other constituent detections of the unique source are embedded in the page. They can be accessed through LEDAS. The slimline catalogue lists a column with the LEDAS URL which can be activated from within applications such as topcat.
As for previous catalogues, every XMM-Newton observation in 3XMM-DR4 has been visually inspected with the purpose of identifying problematic areas where source detection or source characterisation are potentially suspect. The manual screening process generates mask files that define the problematic regions. These may be confined regions around individual suspect detections or larger areas enclosing multiple affected detections, up to the full area of the field where serious problems exist. Detections in such regions are subsequently assigned a manual flag (flag 11) in the flag columns (PN_FLAG, M1_FLAG, M2_FLAG, EP_FLAG). A detection with flag 11 set to (T)rue does not necessarily indicate that the detection is spurious.
One significant change to the screening approach adopted for 3XMM-DR4 relates to the flagging of bright sources and detections within a halo of suspect detections around the bright source. Previously, all detections in the halo region, including the primary detection of the bright source itself (where discernible), had flag 11 set to True (manual flag) but the primary detection of the bright object itself, also had flag 12 set. The meaning of flag 12 there was to signify that the bright object detection was not considered suspect. The use of flag 12 in this 'negative' context, compared to the other flags, was considered to be potentially confusing. For this reason, for 3XMM-DR4, we have dropped the use of flag 12 and simply ensured that, where the bright object detection is clearly identified, it is un-flagged (i.e. neither flag 11 or 12 are set). Flag 12 is effectively not used in 3XMM-DR4.
The masked area of each image is an indicator of the quality of the field as a whole. Large masked areas are typically associated with diffuse extended emission, very bright sources whose wings extend across much of the image, or problems such as arcs arising from scattering of X-rays from bright sources just outside the field of view. The fraction of the field of view that is masked is characterised by the observation class (OBS_CLASS) parameter. The distribution of the six observation classes has changed with respect to 2XMMi-DR3. The dominant change is in the split of fields assigned observation classes (OBS_CLASS) 0 and 1. The largest change with respect to 2XMMi-DR3 is in OBS_CLASS 1 cases; this is due, in part, to the fact that during visual screening for 3XMM-DR4, individual suspect objects in the wings of bright sources were individually masked rather than the whole region.
Table 4 lists the observation class, the fractional area of exclusion with respect to the total detection area, and the percentage of observations affected for the 3XMM-DR4 and 2XMMi-DR3 catalogues.
| Obs class | 'bad' area fraction | 3XMM-DR4 | 2XMMi-DR3 |
|---|---|---|---|
| 0 | 0% area | 27% | 35% |
| 1 | 0% < area < 0.1% | 22% | 12% |
| 2 | 0.1% <= area < 1% | 12% | 10% |
| 3 | 1% <= area < 10% | 24% | 28% |
| 4 | 10% <= area < 100% | 11% | 12% |
| 5 | 100% | 4% | 4% |
| Total number of observations | 7427 | 4953 | |
Most XMM-Newton observations are performed in pointing mode, where the spacecraft is locked on to a fixed position on the sky for the entire observation. Prior to XMM-Newton revolution 1812 (2009-Oct-30), a few special case observations were performed that involved attitude step changes during the observation, generally for the purposes of tracking a moving target. However, in the AO8 observing cycle, a specific mosaic observing mode was introduced in which the satellite pointing direction is stepped across the sky, taking snapshots at points (sub-pointings) on a user-specified grid. Data from dedicated mosaic mode or tracking (mosaic-like) observations are recorded into a single ODF for the observation.
In previous pipelines, the small number of mosaic-like observations sometimes produced products for a single sub-pointing only. This is because the pipeline filters data such that only events taken during an interval where the attitude is stable and centred on the nominal observation pointing direction (within a 3 arcmin tolerance), are accepted. Data from some, or all, of the other sub-pointings were thus sometimes excluded.
During 2012, the SOC devised a scheme whereby the parent ODF of a mosaic mode observation is split into separate ODFs, one for each mosaic sub-pointing. All relevant data are contained within each sub-pointing ODF and the nominal pointing direction is computed for the sub-pointing. This approach is applied to both formal mosaic mode observations and those mosaic-like/tracking observations executed before revolution 1812. For a mosaic mode observation, the first 8 digits of its 10-digit OBS_ID are common for the parent observation and its sub-pointings. However, while the last two digits of the parent observation OBS_ID almost always end in 01, for the sub-pointings they form a monotonic sequence, starting at 31. Mosaic mode sub-pointings are thus immediately recognisable in having OBS_ID values whose last two digits are ≥ 31.
To the pipeline, mosaic mode (and mosaic-like) observation sub-pointings are transparent. No special processing is applied. Each sub-pointing is treated as a distinct observation. Source detection is performed on each sub-pointing separately and no attempt is made to simultaneously fit common sources detected in overlapping regions of multiple sub-pointings. While simultaneous fitting is possible, this aspect had not been sufficiently explored or tested during the preparations for 3XMM-DR4.
There are 42 observations performed in the dedicated mosaic mode before the 3XMM-DR4 processing cut-off date of 2012-Dec-08, of which 40 had usable data for 3XMM-DR4. None of these was available for previous catalogues - the last revolution included in 2XMMi-DR3 was 1800. A further 11 observations prior to revolution 1812 used a mosaic-like or tracking strategy and were available for 3XMM-DR4. In total, there are 419 successfully processed mosaic sub-pointings in the 3XMM-DR4 catalogue.
This section summarises the organisation of the catalogue and gives details of all the columns. Known problems with parameters presented in the catalogue or with products associated with it are listed in Sec. 6.
There are 318 columns in the catalogue; they are grouped together and explained in the links below.
For each observation there are up to three cameras with one or more exposures which were merged when the filter and submodes were the same (2XMM UG, Sec. 2.2). The data in each exposure are accumulated in several distinct energy bands (Table 1). Camera-level measurements can further be combined into observation-level parameters. Consequently, the source parameters can refer to some or all of these levels: on the observation level there are the final mean parameters of the source (prefix 'EP'); on the camera level the data for each of the three cameras (where available) are given (prefix 'PN', 'M1', or 'M2'), and on the energy band level the energy-dependent details of the source parameters are given (indicated by a 'b' in the column name where b = 1,2,3,4,5,8,9). Finally, on a meta-level, some parameters of sources that were detected more than once (prefix 'SC') were combined, see 2XMM UG, Sec. 3.2.4.
The column name is given in capital letters, the FITS data format in brackets and the unit in square brackets. If the column originates from a SAS task, the name of the task is given to the right hand side and a link is set to the SAS package documentation with which the data in the 3XMM-DR4 catalogue was processed (see 3XMM UG, App. A.2 for more details). It should be pointed out that the SAS used for the bulk reprocessing was from manifest xmmsas_20121219_1645, which is based on SAS 12.0.1 but contains a number of SAS task upgrades that were required after the SAS 12.0.1 public release. Note that documentation for the latest public version of the SAS can be found here. A description of the column and possible cross-references follow.
Entries with NULL are given when no detection was made
with the respective camera, that is, ca_MASKFRAC < 0.15 or
NULL (i.e., a camera was not used in an observation).
| Part 1: | 9 columns: Identification of the source |
|---|---|
| This includes the basic static identifiers, IAU name, together with cross-references to the (spatially) nearest detection and source ID in the previous 2XMMi-DR3 catalogue, where relevant, including information about the spatial displacements and the number of 'nearby' matches. Five new columns are used to provide this information, replacing one column that was in 2XMMi-DR3 | |
| Part 2: | 11 columns: Details of the observation and exposures |
| Part 3: | 11 columns: Coordinates |
| The external equatorial and Galactic coordinates and the internal equatorial coordinates as derived from the SAS tasks catcorr and emldetect are given together with the error estimates. Two new columns are added to convey information about the absolute astrometric catalogue used for field rectification and whether the field was successfully rectified | |
| Part 4: | 225 columns: Source parameters |
| The parameters of the source detection as derived from the SAS tasks emldetect and srcmatch are given here. | |
| Part 5: | 8 columns: Detection flags |
| This part lists the flags to qualify the detections. The summary flag, which gives an overall assessment for the detection, is followed by particular flags for each camera. A flag each is given if there exists at least one time series or one spectrum for this source. One new column is added at the end of the catalogue to provide information about detections arising in fields with high background levels | |
| Part 6: | 13 columns: Detection variability |
| This part gives variability information for those detections for which time series were extracted. This includes six new columns compared to 2XMMi-DR3, which provide measures of the fractional variance of the timeseries | |
| Part 7: | 41 columns: Unique source parameters |
| This part lists the source parameters for the unique sources across all observations (using the prefix 'SC'); these are coordinates, fluxes, hardness ratios, likelihoods, extent information and a variability and a summary flag. The number of detections is given also. In 3XMM-DR4 six new columns are introduced in this section. Two of these relate to a fractional variance measure (and error). The other four provide information about the maximum and minimum measured fluxes from constituent detections (and errors). Two further columns relating to the epochs of the first and last observations contributing to a unique source, replace two columns in 2XMMi-DR3 |
Table 6 lists the 44 columns in the 3XMM-DR4 'slimline' version of the catalogue, all of which are explained in Part 1 or Part 7 of the 3XMM-DR4 column description, except the LEDAS_URL column which is described at the end of the table.
This section summarises the main properties of the catalogue but does not provide a detailed analysis. A comprehensive evaluation of the catalogue will be presented in the 3XMM catalogue paper which is currently being prepared for publication.
The catalogue contains source detections drawn from 7427 XMM-Newton EPIC observations made between 2000 February 3 and 2012 December 8 and which were publicly available by 2012 December 31. Net exposure times in these observations range from < 1000 up to ~ 130000 seconds (that is, a full orbit of the satellite). Figure 5.1 shows the distribution of the net exposure time of the fields included, whilst Figure 5.2 shows the distribution of fields on the sky.
The total sky area of the catalogue observations with effective exposure > 1 ks is ~ 1397 deg2 which translates to ~ 794 deg2 when corrected for field overlaps. Figure 5.3 shows the sky area as a function of net exposure time. For Figs. 5.1-5.3 the equivalent data for the previous 2XMMi-DR3 catalogue are shown for comparison.
The catalogue contains 531261 X-ray detections with total-band (0.2 -12 keV) likelihood values ≥ 6. These are detections of 372728 unique X-ray sources (Sec. 3.2), that is, 66728 X-ray sources have multiple detections in separate observations (up to 44 detections). Of the 531261 X-ray detections, 52168 are classified as extended with 10915 of these being in regions considered to be 'clean' (SUM_FLAG < 3).
As part of extensive quality evaluation for the catalogue, each field has been visually screened. Regions where there were obvious deficiencies with the automatic source detection and parameterization process were identified and all detections within those regions were flagged (cf. 2XMM UG, Sec. 3.2.6 but importantly, note Section 3.11). Such flagged detections include clearly spurious detections (many of which are classified as extended) as well as detections where the source parameters may be unreliable. Each XMM-Newton field is also evaluated to assess the fractional area of the observation that is affected by flagged detections, as reflected by the OBS_CLASS parameter. For most uses of the catalogue it is recommended to use either a detection flag (SUM_FLAG, EP_FLAG or SC_SUM_FLAG) or an observation flag (OBS_CLASS) as a filter to obtain what can be considered a 'clean' sample. There are 375273 detections (71%) that have not received any flag (i.e. no issues noted, SUM_FLAG =0), while 432321 detections (81%) can be considered to be reasonably 'clean', i.e. have SUM_FLAG < 3. In terms of the overall field properties, 6369 out of 7427 fields (86%) have only small fraction of flagged detections (i.e OBS_CLASS < 4).
Note that no attempt is made to flag spurious detections arising from statistical fluctuations in the background. An updated analysis of the false detection rate will be presented in the forthcoming 3XMM catalogue paper.
Figure 5.4 presents, for each of the three cameras, the distributions of flux for energy bands 1 to 5 and also for the combined (EPIC) data. These give an indication of the limiting flux available in the catalogues for each of the bands.
Comparison of the detection count rates and fluxes in the 3XMM-DR4 and previous 2XMMi-DR3 version shows good agreement between the two catalogues. A more detailed analysis of photometric issues will be presented in the forthcoming 3XMM catalogue paper.
As noted in section 3.4, the 3XMM-DR4 catalogue benefits from a number of improvements to the astrometry, several of which resulted from effects only discovered in the process of compiling the catalogue. The net effect for 3XMM-DR4 source positions is a small improvement in the statistical position errors, a reduction in the position error systematics and increased confidence in the reliability of the position errors. A more detailed analysis of these issues will again be presented in the forthcoming 3XMM catalogue paper.
Please read the Watchouts section of the 3XMM-DR4 catalogue page for the latest information on 3XMM-DR4 catalogue issues.
A significant number of observations have shown clear evidence of low energy noise affecting specific CCDs in the MOS cameras. Generally but not exclusively, it is CCD4 or CCD5 in MOS1 and CCD2 and CCD5 in MOS2 that are affected and the effect predominantly affects energies below 1keV (bands 1 and 2). Affected CCDs often stand out in the MOS images as having notably higher count levels compared to the adjacent CCDs. Of itself, this increased noise primarily leads to reduced sensitivity in the relevant CCD sky area.
However, a further significant impact arises in source detection because the computation of the background map (see 2XMM UG, Sec. 3.1.2d) does not adequately cope with the step transition in the brightness level between the noisy CCD and adjacent CCDs. The result can be an over- or under- representation of the background map in the vicinity of the CCD boundary and this can then lead to the detection of spurious (often extended) sources near the edges of the noisy CCD where it borders another CCD. These sources generally receive a manual flag from the visual screening process (see 2XMM UG, Sec. 3.2.6, and the changes discussed in Section 3.11 above) but users should be aware of the issue.
The optimised flare filtering process generally results in greater sensitivity to sources. However, in some circumstances, the reverse can occur. About 160 fields that are present in both 3XMM-DR4 and 2XMMi-DR3 show higher backgrounds in the EPIC images, and fewer detections, in the 3XMM-DR4 data. This arises because occasionally, one or more instruments can have a persistent high background while the other instruments have a lower background count rate. In previous processing the high background instruments were generally excluded from the source detection stage because, after applying the flare GTI filtering, less than 1ksec of data remained.
In the new processing for 3XMM-DR4, the optimised flare filtering process determines an optimum background threshold, even if the count rate is persistently high. This may then leave significant apparently usable exposure with a high count rate. As such, instruments showing a persistently high background can still be included in the source detection stage and, even when combined with lower background data from the other instruments, can lead to reduced, rather than increased sensitivity.
In previous catalogues, a few cases have been noted where the detection shows a variability that is due to incorrect handling of the data. Two reasons have been considered responsible:
EPIC timeseries are provided to the public as XMM-Newton pipeline-processed products (filetype SRCTSR), for detections in EPIC exposures that were used in source detection. In this process, the timeseries of the source region and background region (see section 3.6) are fed in to the SAS task, lccorr_pcms, to generate a background-subtracted, exposure-corrected timeseries. lccorr_pcms applies several corrections to take account of aspects such as exposure differences between CCDs. However, while the task can correct for the off-axis dependence of vignetting (e.g. see section 3.2.2.2 of the XMM-Newton User handbook), this correction is not applied to timeseries in the products from the bulk reprocessing that was used for the 3XMM-DR4 catalogue. This was also the case for the 2XMM series of catalogues.
A consequence of not applying this vignetting correction is that the absolute mean count-rate of the timeseries is generally not equivalent to that of the source being observed in the on-axis position. Thus if a user compares two or more timeseries of a common, constant object, observed at very different off-axis angles in separate observations, they will find discrepancies in the mean levels of the timeseries due to the lower effective area pertinent to detections at larger off-axis angles.
To make a more accurate comparison of such timeseries, they should be corrected to the on-axis position. To do this, one can compute an approximate constant scaling factor that can be applied to the pipeline product timeseries (filetype=SRCTSR), for example, via the ftool, farith, by obtaining a measure of the vignetting factor at the source position. A recipe for doing this will be provided here shortly.
A number of improvements in the calibration of the MOS and pn have occurred which lead to slight changes in the Energy Conversion Factors (ECFs) (see here for information on the EPIC response files) that are used for converting EPIC band count rates to fluxes. Of note is the fact that MOS redistribution matrices are now provided for 13 epochs and for three areas of the detector that reflect the so-called 'patch', 'wings-of-patch' and 'off-patch' locations.
For 3XMM-DR4 a simple approach has been adopted. ECFs were computed following the prescription of Mateos et al. (2009), for energy bands 1-5 and band 9, for full-frame mode, for each EPIC camera, for each of the Open, Thin, Medium and Thick filters. For pn, the ECFs are calculated at the on-axis position. The pn response is sufficiently stable that no temporal resolution is needed.
For MOS, to retain a direct connection between the ECFs and publicly available response files, the ECFs used are taken at epoch 13 and are for the 'off-patch' location. The latter choice was made because the large majority of detections in an XMM-Newton field lie outside the 'patch' and 'wings-of-patch' regions, which only relate to a region of radius ≤ 40 arcseconds, near the centre of the field. The use of a single epoch (epoch 13) was made to retain simplicity in the processing and because the response of the MOS cameras exhibits a step function change between epochs 5 and 6, with different but broadly constant values either side of the step. None of the 13 calibration epochs represent the average response and thus no response file exists to which average ECFs can be directly related. The step-function change in the responses for MOS is most marked in band 1 (0.2-0.5 keV) for the 'patch' location, where the maximum range in ECFs either side of the step amounts to 20%. Outside the 'patch' region, and for all other energy bands, the range of the ECF values with epoch is ≤ 5% and is ≤ 2.5% for the 'off-patch' region. Epoch 13 was chosen, somewhat arbitrarily, as being typical of epochs in the longer post-step time interval.
The ECFs, in units of 1011 cts cm2/erg, adopted for the bulk reprocessing of data used for 3XMM-DR4, are provided in Table 8, for each camera, energy band and filter. The camera rate and flux are related via
| Camera | Band | Open | Thin | Medium | Thick |
| PN | 1 | 16.9783 | 9.5248 | 8.3696 | 5.1065 |
| 2 | 10.0696 | 8.121 | 7.8681 | 6.0479 | |
| 3 | 6.1551 | 5.867 | 5.7673 | 4.9893 | |
| 4 | 1.9844 | 1.9527 | 1.929 | 1.8282 | |
| 5 | 0.5782 | 0.5774 | 0.5764 | 0.5698 | |
| 9 | 5.098 | 4.5586 | 4.4602 | 3.7636 | |
| MOS-1 | 1 | 3.0896 | 1.7345 | 1.5258 | 0.9977 |
| 2 | 2.1152 | 1.7461 | 1.6974 | 1.3792 | |
| 3 | 2.1387 | 2.0407 | 2.0058 | 1.7871 | |
| 4 | 0.7491 | 0.7375 | 0.7292 | 0.6991 | |
| 5 | 0.1452 | 0.145 | 0.1451 | 0.143 | |
| 9 | 1.5035 | 1.3843 | 1.3584 | 1.2035 | |
| MOS-2 | 1 | 3.1094 | 1.7336 | 1.5223 | 0.9907 |
| 2 | 2.1309 | 1.7572 | 1.7082 | 1.3869 | |
| 3 | 2.1407 | 2.0426 | 2.0079 | 1.7887 | |
| 4 | 0.7531 | 0.7418 | 0.7333 | 0.703 | |
| 5 | 0.1528 | 0.1528 | 0.1524 | 0.1502 | |
| 9 | 1.5097 | 1.3894 | 1.3634 | 1.2076 |
No epoch information is used when matching detections to construct unique sources. As a consequence, detections of high proper motion stars from multiple observations spanning a significant period of time may not have been matched into a single unique source in the catalogue. A good example is 61 Cyg whose proper motion (~ 5 arcseconds/year) between observations from the earliest XMM-Newton revolution (539) in 3XMM-DR4 to the latest (2083) translates in to a shift in position of more than 40 arcseconds between the first and last observations. The detections of the stellar component at higher declination are mapped to two distinct unique sources due to its movement (i.e. 3XMM J210655.9+384516 and 3XMM J210657.4+384527) - the component at lower declination is grouped in to 3 unique sources. However, the more relaxed criteria for recognising potential confusion result in it being flagged as CONFUSED, the confusion arising from positional overlaps with other detections of itself.
Budavari, T & Szalay, A 2009, Ap.J, 679, 301-309 Probabilistic Cross-Identification of Astronomical Sources
Edelson, R., et al. 2002, Ap.J, 568, 610-626 X-Ray Spectral Variability and Rapid Variability of the Soft X-Ray Spectrum Seyfert 1 Galaxies Arakelian 564 and Ton S180
Mateos, S., et al. 2009, A&A, 496, 879-889 Statistical evaluation of the flux cross-calibration of the XMM-Newton EPIC cameras
Pye, J., et al. 1995, MNRAS, 274, 1165-1193, The ROSAT Wide Field Camera all-sky survey of extreme-ultraviolet sources - II. The 2RE Source Catalogue
Read, A., et al. 2011, A&A, 534, A34 A new comprehensive 2D model of the point spread functions of the XMM-Newton EPIC telescopes: spurious source suppression and improved positional accuracy
Vaughan, S., et al. 2003, MNRAS, 345, 1271-1284 On characterizing the variability properties of X-ray light curves from active galaxies
Watson, M., et al. 2009, A&A, 493, 339-373 The XMM-Newton serendipitous survey. V. The Second XMM-Newton Serendipitous Source Catalogue
| Release No. | Release Date | Comments |
| 1.0 | 23 July 2013 | First release |
| 1.1 | 24 July 2013 | Added section 3.12 and fixed minor typographic corrections |
| 1.2 | 02 August 2013 | Section 6.1.4 added. Reference added. Some SAS task links ammended. Links to CCF and SAS tasks provided in A.2 |
| 1.3 | 30 August 2013 | Added link to watchouts in section 6. |
| 1.4 | 15 September 2013 | Added units and flux-rate conversion formula in section 6.2.1. |
List of observations ('fields').
The processing for 3XMM-DR4 was conducted during December 2012/January 2013. A static set of Current Calibration Files (CCFs) were used which are lsited here. The SAS used was similar to SAS 12.0.1 but includes some upgraded tasks required for the pipeline. The SAS manifest for tasks used in the cat9.0 pipeline for the bulk reprocessing is here. The perl modules that comprise the cat9.0 pipeline (configuration 00000004_04_cat9.0_20121220.153800) used for the bulk reprocessing will also be available from here shortly.