Incorrect EPIC band 4 fluxes in the 2XMM and 2XMMi catalogues

We have uncovered an error in the 2XMM and 2XMMi catalogues which needs to be brought to the attention of users of these catalogues.

Summary

  1. The error affects only the computation of the EPIC band 4 (2.0-4.5keV) flux values for detections (the EP_4_FLUX column) and the corresponding quantity (the SC_EP_4_FLUX column) for unique sources when one or more of the constituent detections is affected. The relevant (OBSMLI) source lists held in the XSA archive also contain the incorrect EP_4_FLUX value.

    None of the other flux columns, or related columns, in the catalogue are affected. There is no error in the individual camera fluxes or in any of the count rates or hardness ratios (which are based on count rates) quoted in the catalogue.

    The error only affects a subset of the entries in the catalogues (around 19% in 2XMM and 17% in 2XMMi) of the detections in existing versions of the 2XMM catalogue released before March 2010). The origin of the error is described in detail in Sec. 1 below.
  2. Statistically, the impact of the error is that the correct EPIC band 4 flux values are systematically larger than the erroneous values (typically by a few sigma) - the mean shift is about a factor 2. A fuller description of the analysis is given in Sec. 2 below. While the size of the correction is modest, we emphasise that the uncorrected values are simply wrong and that the impact on XMM science analyses that exploit the EPIC band 4 fluxes can be significant
  3. Tables which can be used to check which detections are affected by this error, together with the correct values for the EP_4_FLUX and SC_EP_4_FLUX columns are available - see Sec. 3 below.
  4. New full versions of the 2XMMi catalogue (v1.1) and the 2XMM catalogue (v1.2) are available and contain the correct columns.
  5. The new incremental version of the catalogue, 2XMMi-DR3, planned for release in late April 2010, contains the correct the EPIC band 4 fluxes.

1) Background to the problem

During the late stages in the preparation of the 2XMM catalogue in 2007, it was found that the EP_4_FLUX values were wrong due to a software bug in the srcmatch SAS task. The EP_4_FLUX value is the all-EPIC band-4 flux and is calculated as a weighted average of the band-4 fluxes from the available, individual cameras. The problem arose from the erroneous use of the PN band 1 flux value rather than the PN band 4 flux value when calculating this weighted average.

The problem was uncovered after the re-processing of all the available XMM-Newton data in 2006/7 had been completed. To avoid a further lengthy re-processing cycle to generate correct source lists, a correction was applied via an external script, run on all the existing observation summary source lists. These observation summary source lists are inputs to the software used to generate the final source catalogues.

While validation of a sample of source lists appeared to show the corrections had been successfully applied, a modest fraction (~ 17-19%) appear not to have been corrected. As none of these cases were in the validation sample, the problem was missed.

All observations that were re-processed in 2006/7 are potentially affected, but we have found that only observations between XMM-Newton revolutions 0369 (2001 Dec 13) and 1212 (2006 July 23) (and only a modest subset of those) are affected. Observations from revolutions beyond 1338 have been processed with a pipeline in which the srcmatch task bug had been fixed and are not affected in any way.

We also note that a new version of the catalogue, 2XMMi-DR3, which also contains a further ~64000 additional detections compared to 2XMMi, is planned to be released in late April 2010. This new version of the catalogue contains the correct EP_4_FLUX and SC_EP_4_FLUX columns.

However, the observation summary source lists that were delivered to the XMM-Newton Science Archive (XSA) in 2007 remain uncorrected. This means there will be discrepancies between the EP_4_FLUX column values in the new 2XMMi-DR3 catalogue and the observation summary source lists in the XSA and other sites that hold them. The SC_EP_4_FLUX values are specific additions to the catalogues and, as such, are not present in the observation summary source lists.

2) Statistical analysis

Figure 1 below summarises the overall statistical effect of this error in terms of both the flux ratio and flux difference normalised to the flux error. Key points are that the corrected fluxes are systematically higher than the incorrect values, but that in the majority of cases the difference is less than a factor of a few, typically amounting only to a few sigma shift. Nevertheless as the error produces values which are completely wrong and because it introduces a systematic shift to the fluxes, the effects of this error on data analysis projects based on the 2XMM or 2XMMi catalogues can be quite significant.

Smoothed density plots: EP_4_FLUX ratio (corrected/uncorrected) as a function of EP_8_FLUX, and, EP_4_FLUX difference (corrected-uncorrected) normalised by the corrected EP_4_FLUX_ERR value as a function of EP_8_FLUX
Figure 1

Left: Smoothed density plot of EP_4_FLUX ratio (corrected/uncorrected) as a function of EP_8_FLUX for detections affected by the coding error. The background density plot is on a linear scale; the contours enclose [99, 97, 90, 70, 50 & 5%] of the points. The mean ratio is ≈ 1.75; 90% of the points lie between ratios of ≈ 0.7 & 3.4.

Right: Smoothed density plot of EP_4_FLUX difference (corrected-uncorrected) normalised by the corrected EP_4_FLUX_ERR value as a function of EP_8_FLUX for detections affected by the coding error. The background density plot is on a linear scale; the contours enclose [99, 97, 90, 70, 50 & 5%] of the points. The mean difference is ≈ 0.7 σ whilst 90% of the points lie between -3 and +3.4 σ.

3) Corrected catalogue column entries for 2XMM and 2XMMi

New full (and shortly) slim versions of the 2XMM (v1.2) and 2XMMi (v1.1) catalogues are available.

However, for users who wish to check detections or sources affected by this problem and/or correct the values in versions of the 2XMM or 2XMMi catalogues they have previously downloaded, we provide auxiliary .fits and .csv files which contain the corrections. These consist of a small subset of the columns for all detections in each catalogue, i.e. they are provided for each detection regardless of whether the EP_4_FLUX or SC_EP_4_FLUX values were right or wrong in the original catalogues.

The reason that we provide separate files for the 2XMM and 2XMMi catalogues is that, although the correct value of the EP_4_FLUX column is unique to each separate detection, independent of which version of the catalogue it appears in, the correct value of the SC_EP_4_FLUX is catalogue-version-dependent. This is because the SC_EP_4_FLUX value is an average over all detections of the same unique source and thus can change between different catalogue versions as further observations and thus detections are added.

The 9 columns are:

The files can be accessed from

2XMMcat_v1.0_corrected_ep4_subcols.fits FITS binary 8.7 MB MD5: ec09c04f41a19627b5e80a46de68e2c6
2XMMcat_v1.0_corrected_ep4_subcols.csv.gz CSV 4.3 MB MD5: 8fa5eddde501c2060ff8003f2d4cea27
2XMMicat_v1.0_corrected_ep4_subcols.fits FITS binary 10.2 MB MD5: 3e346b5cd62d7b66b314030bec14864c
2XMMicat_v1.0_corrected_ep4_subcols.csv.gz CSV 4.9 MB MD5: fbf509ab4ad336546a712d7ff8c43d5f

SQL CREATE statements are provided to load the CSV format files into relational database management systems.

A list of the observation IDs (and corresponding revolution numbers) which contain catalogue detections affected by this error is available in Table 1.

EP_4_FLUX Error version: 2010-04-15