Error: can't decode messages - IEDX01

[Gaiag]

We are encountering an error while parsing a defined set of files.
For every file in the subset below we obtain the following (number can change of course) output:

### Message header : "0004593100\001\015\015\012000\015\015\012IEDX01\040EUMP\040260824\015\015\012"
### BUFR Edition : 4
### length : 45896
### Section 0
### length : 8
### Section 1
### length : 22
### BUFR master table : 0
### originating center : 254
### sub center : 0
### update sequence number : 0
### Data category : 3
### International sub category : 255
### Local sub category : 223
### master table version : 13
### local table version : 1
### Year : 2014
### Month : 9
### Day : 26
### Hour : 8
### Min : 24
### Sec : 0
### octet 8 : 0
### optional section : No
### Section 2
### length : 0
### Section 3
### length : 131
### datasubsets : 240
### octet 7 : 192
### compression : Yes
### observed data
### Section 4
### length : 45731
### Section 5
### length : 4
###
Error: can't decode messages

On the other side the BUFR/CREX format checker at ECMWF (http://old.ecmwf.int/products/data/d/check) reports no error and decodes correctly the bufr.

Message 1
Section 0
Length of Section 0: 8 byte(s)
Total length of BUFR message: 45896 byte(s)
BUFR edition number: 4
Section 1
Length of Section 1: 22 byte(s)
Originating subcentre: 0
Originating centre: 254
Update sequence number: 0
Flag (Presence of section 2): 0
Local table version number: 1
Data category: 3
Data subcategory: 255
Local data sub-category: 223
Year: 2014
Month: 9
Day: 26
Hour: 8
Minute: 24
Second: 0
BUFR master table: 0
BUFR master table version number: 13
Section 3
Length of Section 3: 131 byte(s)
Reserved: 0
Number of data subsets: 240
Data type/data compression: 192
Data descriptors (unexpanded)
1 001007
2 001031
3 002019
4 002020
5 004001
6 004002
7 004003
8 004004
9 004005
10 004006
11 005040
12 201133
13 005041
14 201000
15 005001
16 006001
17 005043
18 013038
19 008012
20 008013
21 002022
22 008065
23 040192
24 040193
25 040194
26 040195
27 040196
28 040197
29 040198
30 007024
31 005021
32 007025
33 005022
34 008003
35 010004
36 008049
37 012061
38 008049
39 202126
40 007001
41 202000
42 008003
43 106000
44 031001
45 202131
46 201138
47 007004
48 201000
49 202000
50 012101
51 110000
52 031001
53 202131
54 201138
55 007004
56 201000
57 202000
58 202129
59 201134
60 013002
61 201000
62 202000
Data descriptors (expanded)
1 001007 SATELLITE IDENTIFIER
2 001031 IDENTIFICATION OF ORIGINATING/GENERATING CENTRE (SEE NOTE 10)
3 002019 SATELLITE INSTRUMENTS
4 002020 SATELLITE CLASSIFICATION
5 004001 YEAR
6 004002 MONTH
7 004003 DAY
8 004004 HOUR
9 004005 MINUTE
10 004006 SECOND
11 005040 ORBIT NUMBER
12 005041 SCAN LINE NUMBER
13 005001 LATITUDE (HIGH ACCURACY)
14 006001 LONGITUDE (HIGH ACCURACY)
15 005043 FIELD OF VIEW NUMBER
16 013038 SUPERADIABATIC INDICATOR

[...]

Also the utilities that comes with Geo::BUFR Perl module decodes the file correctly.

If need we can provide some example files.

Names of the files for which we encouter the problem:

W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_C_EUMC_??????????????_?????_eps_o_twt_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_C_EUMP_??????????????_?????_eps_o_twt_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_C_EUMP_??????????????_?????_eps_o_clp_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_C_EUMC_??????????????_?????_eps_o_ozo_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_C_EUMP_??????????????_?????_eps_o_ozo_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_C_EUMC_??????????????_?????_eps_o_ems_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_C_EUMP_??????????????_?????_eps_o_ems_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPA+IASI_C_EUMC_??????????????_?????_eps_o_trg_l2.bin
W_XX-EUMETSAT-Darmstadt,SOUNDING+SATELLITE,METOPB+IASI_C_EUMP_??????????????_?????_eps_o_trg_l2.bin
W_XX-EUMETSAT-Darmstadt,SURFACE+SATELLITE,SARAL+OGDR_C_EUMS_??????????????_T_???_????_??????????????.bin
W_C_ISFX04_EGRR_??????????????_UK-MetOffice-Sferics-Africa-BUFR-15minute_concatenation.bin
S-O3M_GOME_O3-NO2_L2_??????????????_001_METOPA_?????_DLR_03.BUFR

[cpb (chris-beauregard)]

Yes, please attach at least a couple of the files it's having trouble with.

[Gaiag (sdk)]

Attachment of the example file with the problem

[cpb (chris-beauregard)]

What BUFR tables are you using to decode it?

The default tables with LibECBUFR don't know about local descriptors.

When you bufr_decoder, check for a DEBUG.decoder file in the local directory. In my case, I'm seeing:

cpb@delta:~/bufr$ more DEBUG.decoder
Descriptor 40192 ??
Error: unknown descriptor 40192
Descriptor 40193 ??
Error: unknown descriptor 40193
Descriptor 40194 ??
Error: unknown descriptor 40194
Descriptor 40195 ??
Error: unknown descriptor 40195
Descriptor 40196 ??
Error: unknown descriptor 40196
Descriptor 40197 ??
Error: unknown descriptor 40197
Descriptor 40198 ??
Error: unknown descriptor 40198
Error: Template definition contains error(s)
Error: Unable to create Template
Error: can't decode messages

Obviously, the stock code tables available to ECMWF and Geo::BUFR know about these local descriptors.

[Gaiag (sdk)]

You are right, we are using the stock tables for each of the products, so I think the problem can be connected to this missing descriptor.
I'd been probably misleading by the question https://answers.launchpad.net/libecbufr/+question/240912 in thinking that was only an "wait for an update problem" sure we were not able to understand it from the screen output. (but our fault because we didn't check the debug file)

Do you know if there's a way or a tool to import their table for use together with libecubufr?

[cpb (chris-beauregard)]

> Do you know if there's a way or a tool to import their
> table for use together with libecubufr?

Not that I'm aware of. I've written code to handle the ECMWF tables from LibECBUFR as a proof-of-concept (see the Examples/foreign_table.c in the source) and it wouldn't be difficult to incorporate into the decoder using, say, a bufr_load_ecmwf_tables() function, but generally speaking we've elected to leave the details of table management outside of the LibECBUFR core.

The master-table-version-checking blueprint sheds a bit more light on the whole issue, and Yves could probably shed even more if that doesn't scare you away.

[cpb (chris-beauregard)]

Indeed... in a pinch, the foreign_table.c example is a usable, is plain, BUFR decoder to use with ECMWF tables. I.e.

cpb@delta:~/src/libecbufr/Examples$ ./foreign_table -tableb B0000000000098013001.TXT -tableb B0000000000254010001.TXT -tabled D0000000000098013001.TXT ~/bufr/W_XX-EUMETSAT-Darmstadt\,SOUNDING+SATELLITE\,METOPB+IASI_C_EUMP_20140926082400_10494_eps_o_twt_l2.bin
cpb@delta:~/src/libecbufr/Examples$ more OUTPUT.TXT
BUFR_EDITION=4
HEADER_STRING="0004593100\001\015\015\012000\015\015\012IEDX01\040EUMP\040260824
\015\015\012"
BUFR_MASTER_TABLE=0
ORIG_CENTER=254
ORIG_SUB_CENTER=0
UPDATE_SEQUENCE=0
DATA_CATEGORY=3
INTERN_SUB_CATEGORY=255
LOCAL_SUB_CATEGORY=223
MASTER_TABLE_VERSION=13
LOCAL_TABLE_VERSION=1
YEAR=2014
MONTH=9
DAY=26
HOUR=8
MINUTE=24
SECOND=0
DATA_FLAG=192
COMPRESSED=1
DATASUBSET 1 : 1486 codes
001007 3
001031 254
--More--(0%)

[cpb (chris-beauregard)]

As you can see, with the correct tables LibECBUFR handles the message perfectly fine. As such (and no offense intended), I've changed the bug status to Invalid. Whether we should directly handle ECMWF tables is another topic entirely which is worthwhile debating in a separate feature request, I think.