Improper handling of values in HuginBase::PTools::Transform::transform causes assertion error in libpano13

[Heewon Park]

Hi there

We just want to share that the latest version (2022.0.0) of pto_merge causes reaching assertion error, which is improper to the normal execution.

The stack execution from the function HuginBase::PTools::Transform::transform() checks NaN through the function erect_lambertazimuthal(), but it causes assertion has failed saying ‘pto_merge: math.c:846: erect_lambertazimuthal: Assertion `! isnan(x)' failed.'

Here is the output of gdb results.

### Bug Report

(gdb) r
Starting program: /home/ubuntu/targets/hugin-2022.0.0_original/build/src/tools/pto_merge ../AFLplusplus/hugin_ptomerge_jpg/default/crashes/id:000242,sig:06,src:001403,time:28753634,execs:1225769,op:havoc,rep:2 1.jpg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
pto_merge: math.c:846: erect_lambertazimuthal: Assertion `! isnan(x)' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff48e9859 in __GI_abort () at abort.c:79
#2 0x00007ffff48e9729 in __assert_fail_base (
    fmt=0x7ffff4a7f588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=0x7ffff5d93d92 "! isnan(x)", file=0x7ffff5d93d8b "math.c", line=846,
    function=<optimized out>) at assert.c:92
#3 0x00007ffff48fafd6 in __GI___assert_fail (assertion=0x7ffff5d93d92 "! isnan(x)",
    file=0x7ffff5d93d8b "math.c", line=846, function=0x7ffff5d93e60 "erect_lambertazimuthal")
    at assert.c:101
#4 0x00007ffff5d5ce68 in erect_lambertazimuthal () from /lib/x86_64-linux-gnu/libpano13.so.3
#5 0x00007ffff5d5b8d3 in execute_stack_new () from /lib/x86_64-linux-gnu/libpano13.so.3
#6 0x00007ffff733a368 in **HuginBase::PTools::Transform::transform** (this=this@entry=0x7fffffff3fc0,
    dest=..., src=...)
    at /home/ubuntu/targets/hugin-2022.0.0_original/src/hugin_base/panotools/PanoToolsInterface.cpp:250
#7 0x00007ffff724a33f in HuginBase::PanoramaOptions::getVFOV (this=<optimized out>)
    at /home/ubuntu/targets/hugin-2022.0.0_original/src/hugin_base/hugin_math/hugin_math.h:87
#8 0x00007ffff724c31e in HuginBase::PanoramaOptions::setProjectionParameters (this=0x7fffffffc530,
    params=std::vector of length 0, capacity 0)
    at /home/ubuntu/targets/hugin-2022.0.0_original/src/hugin_base/panodata/PanoramaOptions.cpp:190
#9 0x00007ffff724c859 in HuginBase::PanoramaOptions::resetProjectionParameters (this=<optimized out>)
    at /home/ubuntu/targets/hugin-2022.0.0_original/src/hugin_base/panodata/PanoramaOptions.cpp:200
#10 0x00007ffff71cd2ba in HuginBase::PanoramaMemento::loadPTScript (this=<optimized out>, i=...,
    ptoVersion=<optimized out>, prefix=...)
    at /home/ubuntu/targets/hugin-2022.0.0_original/src/hugin_base/panodata/Panorama.cpp:2492
#11 0x00007ffff71f7619 in HuginBase::Panorama::readData (this=<optimized out>, dataInput=...,
    documentType=...) at /usr/include/c++/9/bits/basic_string.h:267
#12 0x000055555555e976 in main (argc=<optimized out>, argv=0x7fffffffe4c8)
    at /usr/include/c++/9/ext/new_allocator.h:80

### Envionment
OS: Ubuntu 20.04.5 LTS x86_64
Release: hugin 2022.0.0
Program: pto_merge
libhuginbase: 2020.0.0 (retrieved and compiled from source code)
libpano13: 2.9.19
To reproduce the problem, we need to build hugin:
sudo cmake -DCMAKE_C_FLAGS="-g" -DCMAKE_CXX_FLAGS="-g" ..

### How to reproduce
$ pto_merge poc-file *.jpg
(*.jpg any name of jpg file including asterisk(*))
poc-file is attached.

[tmodes (tmodes)]

Fixed in repository.

[Heewon Park (3den5eo)]

Hi there. I am Heewon, and I am writing to you regarding the recent vulnerabilities that our security team identified in Hugin. I appreciate your prompt attention to these matters, and I am pleased that the vulnerabilities have been confirmed and successfully patched by your development team.

To provide a standardized reference for these vulnerabilities within the cybersecurity community, we would like to request the assignment of Common Vulnerabilities and Exposures (CVE) identifiers. These identifiers will help streamline communication and information sharing among security professionals.

Below is a brief summary of the vulnerabilities along with the relevant details:

### CVE-2023-XXX1: [Description of Vulnerability 1]

- Confirmation: Fixed in Hugin 2022.0.0
- Patch: 2023.0beta1 on 2023-06-29 by tmodes user
- url: https://bugs.launchpad.net/hugin/+bug/2025032

### CVE-2023-XXX2: [Description of Vulnerability 2]

- Confirmation: Fixed in Hugin 2022.0.0
- Patch: 2023.0beta1 on 2023-06-29 by tmodes user
- url: https://bugs.launchpad.net/hugin/+bug/2025035

### CVE-2023-XXX3: [Description of Vulnerability 3]

- Confirmation: Fixed in Hugin 2022.0.0
- Patch: 2023.0beta1 on 2023-06-29 by tmodes user
- url: https://bugs.launchpad.net/hugin/+bug/2025036

### CVE-2023-XXX4: [Description of Vulnerability 4]

- Confirmation: Fixed in Hugin 2022.0.0
- Patch: 2023.0beta1 on 2023-06-29 by tmodes user
- url: https://bugs.launchpad.net/hugin/+bug/2025037

### CVE-2023-XXX5: [Description of Vulnerability 5]

- Confirmation: Fixed in Hugin 2022.0.0
- Patch: 2023.0beta1 on 2023-06-29 by tmodes user
- url: https://bugs.launchpad.net/hugin/+bug/2025038

We kindly request that you forward this information to the appropriate party responsible for CVE assignments within your organization. If your organization has a designated CVE Numbering Authority (CNA), please let us know the preferred process for CVE assignment.

Additionally, we have submitted the same request to MITRE Corporation and CERT/CC, the primary CVE Numbering Authority, for their consideration. However, CERT/CC asked us to refer to you for CVE assignments. Please work on this case and let us know which steps to take.

Thank you for your cooperation and commitment to addressing security issues promptly. If you require any further information or clarification, please do not hesitate to reach out.

We look forward to continuing a collaborative approach to enhancing the security of Hugin and appreciate your ongoing dedication to the security and well-being of your users.