[upstream] Batch libreoffice --convert-to offers no way to wait for document completion

[Richard Elkins]

Command-line: libreoffice --convert-to odt:writer8 myfile.txt

The returned status code ($?) is zero. So far, so good. But, when I try to access the ODT file (E.g. copy it), it doesn't yet exist.

Adding `sync; sync; sync` right after the libreoffice batch execution did not help.
Adding --headless as an option did not help.

If I sleep for a few seconds immediately after the libreoffice batch execution, then the file finally gets completed before a subsequent command references the ODT file.

Sample script:

### Create somehow a file called myfile.txt
rm myfile.odt somewhere-else.odt
libreoffice --headless --convert-to odt:writer8 myfile.txt
RC=$?
if [ $RC -ne 0 ]; then
 echo '*** libreoffice conversion failed for myfile.txt'
 exit 86
fi
#sleep 3
ls *.odt

Result: ls: cannot access '*.odt': No such file or directory

If one changes the above script so that the sleep is executed, then the ODT file is available.

Observation: It appears that executing libreoffice in batch mode is somehow kicking off a separate process to finish the ODT file. The exit to the shell and the availability of the ODT file should be synchronized.

Before 18.04, I never saw this behavior.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: libreoffice 1:6.0.3-0ubuntu1
ProcVersionSignature: Ubuntu 4.15.0-23.25-generic 4.15.18
Uname: Linux 4.15.0-23-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
CurrentDesktop: XFCE
Date: Sat Jun 16 17:07:02 2018
InstallationDate: Installed on 2017-10-13 (246 days ago)
InstallationMedia: Xubuntu 17.10 "Artful Aardvark" - Alpha amd64 (20170926)
SourcePackage: libreoffice
UpgradeStatus: Upgraded to bionic on 2018-05-18 (29 days ago)

[Richard Elkins (texadactyl)]

Description:
Command-line: libreoffice --headless --convert-to odt:writer8 myfile.txt
The returned status code ($?) is zero. So far, so good.
But, when I try to use the ODT file (E.g. copy it), it doesn't exist yet.
Yes, I tried `sync; sync; sync` but it did not help.
If I add a `sleep 1` immediately after checking the status code, then this seems to allow enough time for some libreoffice subprocess (?) to finish.

Before version 6, I did not need the sleep step. Maybe, this is some sort of optimization? If so, please provide an option to indicate that libreoffice should hold up the process until the desired output is available.

I am using libreoffice 1:6.0.3-0ubuntu1 on Xubuntu 18.04.

`libreoffice --help` shows:
LibreOffice 6.0.3.2 00m0(Build:2)

Steps to Reproduce:
Linux batch script, starting with an existing text file called "myfile.txt":

 libreoffice --headless --convert-to odt:writer8 myfile.txt
 RC=$?
 if [ $RC -ne 0 ]; then
  echo '*** libreoffice conversion failed for myfile.txt'
  exit 86
 fi
        cp myfile.odt somewhere-else.odt

Actual Results:
cp: cannot stat 'myfile.odt': No such file or directory

Expected Results:
Copy completes as normal because myfile.odt is available.

Reproducible: Always

User Profile Reset: No

Additional Info:

User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0

[JBF (jbf-faure)]

Not reproducible for me under Ubuntu 16.04 x86-64 with LibreOffice 6.0.4 from Ubuntu PPA. The script completes as expected.

To be sure, does it works for you if you try only the command

libreoffice --headless --convert-to odt:writer8 myfile.txt

in a terminal ?

Do you have some non standard settings for you file system?

Status set to NEEDINFO, please set it back to UNCONFIRMED once requested
informations are provided.

Best regards. JBF

[Richard Elkins (texadactyl)]

Like you, when I ran LibreOffice under Xubuntu 16.04 and 17.10, there was no timing issue. This only has appeared for me in 18.04.

In a terminal window (I've done this before),
`libreoffice --headless --convert-to odt:writer8 myfile.txt; ls *.odt` result:
ls: cannot access '*.odt': No such file or directory

My LibreOffice packages installed:
ii libreoffice 1:6.0.3-0ubuntu1 amd64 office productivity suite (metapackage)
ii libreoffice-avmedia-backend-gstreamer 1:6.0.3-0ubuntu1 amd64 GStreamer backend for LibreOffice
ii libreoffice-base 1:6.0.3-0ubuntu1 amd64 office productivity suite -- database
ii libreoffice-base-core 1:6.0.3-0ubuntu1 amd64 office productivity suite -- shared library
ii libreoffice-base-drivers 1:6.0.3-0ubuntu1 amd64 Database connectivity drivers for LibreOffice
ii libreoffice-calc 1:6.0.3-0ubuntu1 amd64 office productivity suite -- spreadsheet
ii libreoffice-common 1:6.0.3-0ubuntu1 all office productivity suite -- arch-independent files
ii libreoffice-core 1:6.0.3-0ubuntu1 amd64 office productivity suite -- arch-dependent files
ii libreoffice-draw 1:6.0.3-0ubuntu1 amd64 office productivity suite -- drawing
ii libreoffice-gnome 1:6.0.3-0ubuntu1 amd64 office productivity suite -- GNOME integration
ii libreoffice-gtk 1:6.0.3-0ubuntu1 all transitional package for LibreOffice gtk2 backend
ii libreoffice-gtk2 1:6.0.3-0ubuntu1 amd64 office productivity suite -- GTK+ 2 integration
ii libreoffice-gtk3 1:6.0.3-0ubuntu1 amd64 office productivity suite -- GTK+ 3 integration
ii libreoffice-help-en-us 1:6.0.3-0ubuntu1 all office productivity suite -- English_american help
ii libreoffice-impress 1:6.0.3-0ubuntu1 amd64 office productivity suite -- presentation
ii libreoffice-java-common 1:6.0.3-0ubuntu1 all office productivity suite -- arch-independent Java support files
ii libreoffice-librelogo 1:6.0.3-0ubuntu1 all Logo-like progamming language for LibreOffice
ii libreoffice-math 1:6.0.3-0ubuntu1 amd64 office productivity suite -- equation editor
ii libreoffice-nlpsolver 0.9+LibO6.0.3-0ubuntu1 all "Solver for Nonlinear Programming" extension for LibreOffice
ii libreoffice-ogltrans 1:6.0.3-0ubuntu1 amd64 LibreOffice Impress extension for slide transitions using OpenGL
ii libreoffice-report-builder 1:6.0.3-0ubuntu1 all LibreOffice component for building database reports
ii libreoffice-report-builder-bin 1:6.0.3-0ubuntu1 amd64 LibreOffice component for building database reports -- libraries
ii libreoffice-script-pro...

[Richard Elkins (texadactyl)]

`libreoffice --version`:
LibreOffice 6.0.3.2 00m0(Build:2)

[Richard Elkins (texadactyl)]

You might be tempted to think that Ubuntu or my hardware is running a little slow. If you insert a `sync; sync; sync` right after the libreoffice step which flushes buffers, that does not help consistently. Something got spawned by or because of libreoffice. It finishes long after libreoffice exits. Bad idea, in my opinion.

When libreoffice exits, that ODT file should be immediately available.

[JBF (jbf-faure)]

You should report this behavior against Ubuntu 18.04 on https://bugs.launchpad.net/

Best regards. JBF

[Richard Elkins (texadactyl)]

Not urgent. I am not stuck. The sleep for a few seconds is a consistently successful work-around.

[Olivier Tilloy (osomon)]

I haven't found any mention in the documentation of the asynchronous nature of the --convert-to option, but it looks like it is asynchronous indeed. You should probably ask for confirmation at https://ask.libreoffice.org/. If this is confirmed then requesting an option to wait until the conversion completes would have to be done in the upstream bug tracker: https://bugs.documentfoundation.org/enter_bug.cgi?product=LibreOffice.

In any case, please share whatever relevant information you'll get from upstream here. Thanks!

[Qa-admin-q (qa-admin-q)]

Dear Bug Submitter,

This bug has been in NEEDINFO status with no change for at least
6 months. Please provide the requested information as soon as
possible and mark the bug as UNCONFIRMED. Due to regular bug
tracker maintenance, if the bug is still in NEEDINFO status with
no change in 30 days the QA team will close the bug as INSUFFICIENTDATA
due to lack of needed information.

For more information about our NEEDINFO policy please read the
wiki located here:
https://wiki.documentfoundation.org/QA/Bugzilla/Fields/Status/NEEDINFO

If you have already provided the requested information, please
mark the bug as UNCONFIRMED so that the QA team knows that the
bug is ready to be confirmed.

Thank you for helping us make LibreOffice even better for everyone!

Warm Regards,
QA Team

MassPing-NeedInfo-Ping-20190111

[Richard Elkins (texadactyl)]

Opened on Launchpad as #1777285 against libreoffice (Ubuntu).

[Richard Elkins (texadactyl)]

Moved to bugs.launchpad.net and confirmed:
https://bugs.launchpad.net/ubuntu/+source/libreoffice/+bug/1777285

[Richard Elkins (texadactyl)]

Done today: https://ask.libreoffice.org/en/question/198713/executed-from-linux-command-line-convert-to-is-unexpectedly-asynchronous/

[Richard Elkins (texadactyl)]

The file /usr/bin/libreoffice is a soft link to soffice, a Bourne shell script. Inside the script, the soffice.bin executable is invoked.

The source code indicates that soffice.bin is the one starting a background process that does not finish before soffice.bin exits.

See: https://github.com/LibreOffice/core/blob/master/shell/source/unix/exec/shellexec.cxx
Go to line 218:

    OString cmd =
#ifdef LINUX
        // avoid blocking (call it in background)
        "( " + aBuffer.makeStringAndClear() + " ) &";
#else
        aBuffer.makeStringAndClear();
#endif
    FILE *pLaunch = popen(cmd.getStr(), "w");
    if ( pLaunch != nullptr )
    {
        if ( 0 == pclose( pLaunch ) )
            return;

It would be interesting to understand why for only Linux, the request is executed in background. In my opinion, this is undesirable for command line execution in any O/S.

[Richard Elkins (texadactyl)]

The source code indicates that soffice.bin is the one starting a background process that does not finish before soffice.bin exits.

See: https://github.com/LibreOffice/core/blob/master/shell/source/unix/exec/shellexec.cxx
Go to line 218:

    OString cmd =
#ifdef LINUX
        // avoid blocking (call it in background)
        "( " + aBuffer.makeStringAndClear() + " ) &";
#else
        aBuffer.makeStringAndClear();
#endif
    FILE *pLaunch = popen(cmd.getStr(), "w");
    if ( pLaunch != nullptr )
    {
        if ( 0 == pclose( pLaunch ) )
            return;

It would be interesting to understand why execution is in background *only* for Linux. In my opinion, it is undesirable for command line execution in any O/S.

Other opinions? It is possible for me to be perfectly content with the artificial `sleep N` in my shell script.

[Richard Elkins (texadactyl)]

I reopened https://bugs.documentfoundation.org/show_bug.cgi?id=117731 since it seems to be an issue in the central Libreoffice code.

[Xiscofauli (xiscofauli)]

Could you please try to reproduce it with the latest version of LibreOffice
from https://www.libreoffice.org/download/libreoffice-fresh/ ?
I have set the bug's status to 'NEEDINFO'. Please change it back to
'UNCONFIRMED' if the bug is still present in the latest version.

[Richard Elkins (texadactyl)]

Yes, I have tried this with the latest download. This is an off-the-shelf Xubuntu 18.04.latest LTS desktop that I use primarily for Libreoffice, browsing, and Python development. I tend to stay with Ubuntu's packages including Libreoffice.

Please see comment #9.

The current source code has special #ifdef LINUX logic to fork the transaction into background and then immediately exit to the command line. It does not wait for the ODT file to be ready for use. Hence, any Linux/Unix bash command that references the ODT file will definitely fail with file not found. Workaround: sleep for 2/3 seconds before trying to access the ODT file.

For Mac and Windows, the transaction takes place in foreground so that when soffice.bin exits to the O/S, the ODT file is guaranteed to be available for use. This is the desirable effect for all O/Ses.

Why is there more concern with "blocking" in the Unix/Linux environments as opposed to Mac (Unix bash environment) and Windows?

IMO, this is a bug for the Linux environment. There is no reason for special #ifdef LINUX logic.

Has anyone else tried the above script yourself using a Linux desktop with the current or recent Libreoffice?

[Richard Elkins (texadactyl)]

Seriously, I am not trying to make work for anyone. We are all busy and I do not want to be a pest.

If the #ifdef LINUX for Linux and Unix in shellexec.cxx has a purpose, I'd like to know. I can live with my `sleep` logic.

[Richard Elkins (texadactyl)]

The command (OString cmd) is executed in background through a pipe no matter what. The difference is that in Mac and Windows, the foreground process waits (blocked) while in Linux and Unix, the foreground process continues to the exit.

Pardon my misuse of terminology.

[Xiscofauli (xiscofauli)]

(In reply to Richard Elkins from comment #9)
> The source code indicates that soffice.bin is the one starting a background
> process that does not finish before soffice.bin exits.
>
> See:
> https://github.com/LibreOffice/core/blob/master/shell/source/unix/exec/
> shellexec.cxx
> Go to line 218:
>
> OString cmd =
> #ifdef LINUX
> // avoid blocking (call it in background)
> "( " + aBuffer.makeStringAndClear() + " ) &";
> #else
> aBuffer.makeStringAndClear();
> #endif
> FILE *pLaunch = popen(cmd.getStr(), "w");
> if ( pLaunch != nullptr )
> {
> if ( 0 == pclose( pLaunch ) )
> return;
>
> It would be interesting to understand why execution is in background *only*
> for Linux. In my opinion, it is undesirable for command line execution in
> any O/S.
>
> Other opinions? It is possible for me to be perfectly content with the
> artificial `sleep N` in my shell script.

it was added in https://cgit.freedesktop.org/libreoffice/core/commit/?id=f5fe1aa1 to fix bug 32427

@Richard, can you confirm removing the mentioned ifdef condition would fix the issue? In that case we could submit a patch removing it. Most likely bug 32427 is obsolete nowadays...

[Richard Elkins (texadactyl)]

@Xisco Faulí - Are you asking me to download and build LibreOffice for that one source change? I wouldn't know where to begin. I am a developer but up to my eyeballs in Astrophysics projects at the moment.

Is there anyone on the development or testing team who already has the source downloaded and knows how to build LibreOffice? I am confident that such a person could easily reproduce this anomaly on Linux, Android, or Unix with the script I provided. Then, make the change I suggested. Re-run the script and see that the anomaly disappears. Also, after a change like this, wouldn't one need to run your standard regression test suite?

[Qa-admin-q (qa-admin-q)]

[Automated Action] NeedInfo-To-Unconfirmed

[Ilmari-lauhakangas (ilmari-lauhakangas)]

(In reply to Richard Elkins from comment #0)
> Steps to Reproduce:
> Linux batch script, starting with an existing text file called "myfile.txt":
>
> libreoffice --headless --convert-to odt:writer8 myfile.txt
> RC=$?
> if [ $RC -ne 0 ]; then
> echo '*** libreoffice conversion failed for myfile.txt'
> exit 86
> fi
> cp myfile.odt somewhere-else.odt
>
> Actual Results:
> cp: cannot stat 'myfile.odt': No such file or directory
>
> Expected Results:
> Copy completes as normal because myfile.odt is available.

I get the expected result. I guess you should bite the bullet, clone the source and try modifying it.

https://wiki.documentfoundation.org/Development/BuildingOnLinux
https://wiki.documentfoundation.org/Development/Linux_Build_Dependencies

Arch Linux 64-bit
Version: 7.0.0.0.alpha1+
Build ID: c73418d8d1258ea0a8c77c07672fd182e2b28b26
CPU threads: 8; OS: Linux 5.6; UI render: default; VCL: kf5;
Locale: fi-FI (fi_FI.UTF-8); UI: en-US
Calc: threaded
Built on 9 May 2020

[Xiscofauli (xiscofauli)]

(In reply to Buovjaga from comment #17)
> (In reply to Richard Elkins from comment #0)
> > Steps to Reproduce:
> > Linux batch script, starting with an existing text file called "myfile.txt":
> >
> > libreoffice --headless --convert-to odt:writer8 myfile.txt
> > RC=$?
> > if [ $RC -ne 0 ]; then
> > echo '*** libreoffice conversion failed for myfile.txt'
> > exit 86
> > fi
> > cp myfile.odt somewhere-else.odt
> >
> > Actual Results:
> > cp: cannot stat 'myfile.odt': No such file or directory
> >
> > Expected Results:
> > Copy completes as normal because myfile.odt is available.
>
> I get the expected result. I guess you should bite the bullet, clone the
> source and try modifying it.
>
> https://wiki.documentfoundation.org/Development/BuildingOnLinux
> https://wiki.documentfoundation.org/Development/Linux_Build_Dependencies
>
> Arch Linux 64-bit
> Version: 7.0.0.0.alpha1+
> Build ID: c73418d8d1258ea0a8c77c07672fd182e2b28b26
> CPU threads: 8; OS: Linux 5.6; UI render: default; VCL: kf5;
> Locale: fi-FI (fi_FI.UTF-8); UI: en-US
> Calc: threaded
> Built on 9 May 2020

@Richard Elkins, have you had time to try it ?

[Richard Elkins (texadactyl)]

@xiscofauli

In message #18, I did indicate that I tried the latest download at that time (2019-06-26). The source code still has the #ifdef LINUX logic which causes this timing issue. All one has to do is read the code and you can see the issue. Maybe that #ifdef made sense some time in the past but it does not anymore.

BUT if there is no will to fix it, my work-around still is fine.

[Richard Elkins (texadactyl)]

Since there clearly no interest or resources to fix this, I am withdrawing this bug report.

[Qa-admin-q (qa-admin-q)]

Dear Richard Elkins,

This bug has been in NEEDINFO status with no change for at least
6 months. Please provide the requested information as soon as
possible and mark the bug as UNCONFIRMED. Due to regular bug
tracker maintenance, if the bug is still in NEEDINFO status with
no change in 30 days the QA team will close the bug as INSUFFICIENTDATA
due to lack of needed information.

For more information about our NEEDINFO policy please read the
wiki located here:
https://wiki.documentfoundation.org/QA/Bugzilla/Fields/Status/NEEDINFO

If you have already provided the requested information, please
mark the bug as UNCONFIRMED so that the QA team knows that the
bug is ready to be confirmed.

Thank you for helping us make LibreOffice even better for everyone!

Warm Regards,
QA Team

MassPing-NeedInfo-Ping

[Richard Elkins (texadactyl)]

Since there seems to be no interest nor resource to fix the source code where I indicated in comment 9, just forget it. My work-around works fine.

I marked this report as RESOLVED and WONTFIX.