The protocol version according to which the protocol message is interpreted by the library such as TLS1_3_VERSION, TLS1_2_VERSION etc. This is set to 0 for the SSL3_RT_HEADER pseudo content type (see NOTES below).
But we see version set to 769, then 771. And the code was correctly checking for version 0 AND some content_type, but the patch changes that to an OR.
So I now understand the OR change, just not why content_type is compared with UINT8_MAX. The docs say that pseudo content types have very specific values. Again from that manpage:
content_type
This is one of the content type values defined in the protocol specification (SSL3_RT_CHANGE_CIPHER_SPEC, SSL3_RT_ALERT, SSL3_RT_HANDSHAKE; but never SSL3_RT_APPLICATION_DATA because the callback will only be called for protocol messages). Alternatively it may be a "pseudo" content type. These pseudo content types are used to signal some other event in the processing of data (see NOTES below).
And
Pseudo content type values may be sent at various points during the processing of data. The following pseudo content types are currently defined:
SSL3_RT_HEADER
(...)
SSL3_RT_INNER_CONTENT_TYPE
All of these I found defined in /usr/include/openssl/ssl3.h:
$ grep -E "^#[[:blank:]]*define.*(SSL3_RT_CHANGE_CIPHER_SPEC|SSL3_RT_ALERT|SSL3_RT_HANDSHAKE|SSL3_RT_HEADER|SSL3_RT_INNER_CONTENT_TYPE)" -w /usr/include/openssl/ssl3.h
# define SSL3_RT_CHANGE_CIPHER_SPEC 20
# define SSL3_RT_ALERT 21
# define SSL3_RT_HANDSHAKE 22
# define SSL3_RT_HEADER 0x100
# define SSL3_RT_INNER_CONTENT_TYPE 0x101
While they are all less than UINT8_MAX, UINT8_MAX seems an arbitrary threshold, unless it's mentioned in some other documentation I didn't find yet.
> Debug: Ignoring cbtls_msg call with pseudo content type 256, version 769
These troubled me a bit. When there is pseudo content type, the docs say the version is set to 0. From https:/ /www.openssl. org/docs/ man3.0/ man3/SSL_ CTX_set_ msg_callback. html:
version
The protocol version according to which the protocol message is interpreted by the library such as TLS1_3_VERSION, TLS1_2_VERSION etc. This is set to 0 for the SSL3_RT_HEADER pseudo content type (see NOTES below).
But we see version set to 769, then 771. And the code was correctly checking for version 0 AND some content_type, but the patch changes that to an OR.
I then found this openssl bug, still open: https:/ /github. com/openssl/ openssl/ issues/ 17262 set_msg_ callback - cb function version argument in 3.0.0 does not match documentation"
"SSL_CTX_
So I now understand the OR change, just not why content_type is compared with UINT8_MAX. The docs say that pseudo content types have very specific values. Again from that manpage:
content_type
This is one of the content type values defined in the protocol specification (SSL3_RT_ CHANGE_ CIPHER_ SPEC, SSL3_RT_ALERT, SSL3_RT_HANDSHAKE; but never SSL3_RT_ APPLICATION_ DATA because the callback will only be called for protocol messages). Alternatively it may be a "pseudo" content type. These pseudo content types are used to signal some other event in the processing of data (see NOTES below).
And
Pseudo content type values may be sent at various points during the processing of data. The following pseudo content types are currently defined:
SSL3_RT_HEADER INNER_CONTENT_ TYPE
(...)
SSL3_RT_
All of these I found defined in /usr/include/ openssl/ ssl3.h: ]]*define. *(SSL3_ RT_CHANGE_ CIPHER_ SPEC|SSL3_ RT_ALERT| SSL3_RT_ HANDSHAKE| SSL3_RT_ HEADER| SSL3_RT_ INNER_CONTENT_ TYPE)" -w /usr/include/ openssl/ ssl3.h CHANGE_ CIPHER_ SPEC 20 INNER_CONTENT_ TYPE 0x101
$ grep -E "^#[[:blank:
# define SSL3_RT_
# define SSL3_RT_ALERT 21
# define SSL3_RT_HANDSHAKE 22
# define SSL3_RT_HEADER 0x100
# define SSL3_RT_
While they are all less than UINT8_MAX, UINT8_MAX seems an arbitrary threshold, unless it's mentioned in some other documentation I didn't find yet.