Losing Assets in filesystem when copying within a ZEO installation

[Flynt]

Happens within a ZEO installation with Silva-1.1 with assests on filesystem via
ExtFile and Samba share:

It seems that during cut and copy operations sometimes the assets get lost on
the filesystem and have to be restored from backup. The same happens sometimes
when changing Silva Folders to Silva Publications, probably because there is a
copy operation involved, according to the sysadmins report.

Problem does not happen in single Zope/Silva instances and it does happen on the
ZEO intermittendly. Happens with Silva-1.1.5, not tested yet with Silva-1.5 or
Silva-1.6.

Can you reproduce the error with a Silva-1.5 and ZEO installation ?

[Martijn Faassen (faassen)]

Hm, I can test Silva 1.5 directly first or test with Silva 1.1 initially. This
depends a bit on your priorities. When are you expecting an upgrade to Silva
1.5? Do you want this issue fixed in Silva 1.1 at all?

Another question: have you by any chance replicated this problem in a non-Samba
ZEO setup or is no information on this available? I am hoping I can exclude
Samba as a possible cause easily this way if you have seen this issue without
Samba. :)

[Flynt (flyntle)]

Best is - and after talking back to the operation's guys, to try to reproduce
the error already with an 1.1 installation (Silva-1.1.5 and ExtFile-1.4.2). As a
first point it would be important to see wether at all you can reproduce the
errors on the Silva-1.1.5 level with ZEO and Samba.

Samba for ZEO: All tests here were done with assets on a Samba share. The single
instance tests (Non-ZEO) where we do not observe the errors were without Samba.
No test were done with ZEO and NFS e.g.

I also upload the Config.py of the ExtFile product.

[Martijn Faassen (faassen)]

Thanks, I'll try to reproduce in a Silva 1.1 context then.

As to Samba: I'll attempt to reproduce this on a single system with ZEO and
without Samba first. If that doesn't reproduce the issue, I'll pull Samba into
the mix.

[Martijn Faassen (faassen)]

I've now set up a Silva 1.1 with ZEO & ExtFile. No samba yet, just two ZEO
clients talking to the same filesystem storage on a single machine. I'm trying
to reproduce the issue.

Some more questions, hope you can answer some at least:

* how often does this happen? How often during the day, what percentage of copy
and paste events does this occur would you say?

* is this reliably demonstratable or does this happen once every while?

* do you have indications that the machine is loaded heavily when this happens?
Many conflict errors, say?

* does file size matter? I.e. does this happen more or less often with large
files, for instance?

* does the size of the copy matter? I.e. if you're copying a lot (say, a whole
tree of folders with assets in there) does it happen more often than if you just
copy one file?

* you've seen this happen both with cut/paste and copy/paste?

I'm asking these questions just to get an idea on how to best reproduce this, so
in general if you have a sequence of steps to make this more likely to happen,
please let me know. Or possibly in reverse: are there circumstances or ways to
make the problem *less* likely to occur?

[Martijn Faassen (faassen)]

Is there an error message associated to this asset loss, or does the operation
complete successfully and are the assets just broken?

[Martijn Faassen (faassen)]

To expand that question: how do you actually find out that something is wrong
with the assets? The "restore from backup" solution implies you find out about
brokenness
quickly after things get broken somehow. Do you actually go to the filesystem
and notice the file has disappeared?

[Martijn Faassen (faassen)]

At some points after various actions (in particular folder -> publication
conversions and back), attempting to copy & paste a file subsequently breaks
with an attribute error.

* Is this something you also see?

* Is it related to the problem of disappearing files?

* Or is this something you also get but is *unrelated* to the disappearing files
issue?

* What procedure (if any?) do you use to recover from the AttributeError problem?

[Martijn Faassen (faassen)]

I notice the file that now gives me the AttributeError reports a file-size of 0,
and is not downloadable anymore (instead I get a very strange broken icon when I
try - where this comes from is still a mystery to me).

This is interesting as previously I had files that reported this error and *did*
appear to be downloadable. I'll try to reproduce that.

[Martijn Faassen (faassen)]

Okay, I've just (somehow) reproduced the scenario of a file giving the
AttributeError upon a copy-paste attempt, but it *does* report the correct
filesize and appears downloadable. Both situations seem to appear.

Are the symptoms I'm talking about at all similar to the symptoms of your
problem, or is this a different set of issues I'm chasing?

[Martijn Faassen (faassen)]

In your Config.py you turn on an option that tries to make the paths on the
filesystem the same as the paths in Zope. This works when files are initially
added, but the path is not kept in sync at all later on in the process - if the
containing folder gets renamed for instance, or the file itself, the path to the
filesystem directory is still the same as before.

What is the main reason you map the folder structure to the directory structure
on the filesystem?

[Martijn Faassen (faassen)]

Chatted with Bengt. It might be the problems were introduced when the Config.py
of Extfile was changed to SYNC_ZODB. Before apparently the problems didn't
exist. Bengt is going to experiment with the SLICED_HASH option to see whether
the problems will persist.

I will continue some experiments here to see whether I can reproduce the problem
better, now that I have a better idea of the symptoms after the phone
conversation with Bengt.

[Martijn Faassen (faassen)]

Trying to reproduce the 0 bytes files issue, but haven't managed to yet. Once
earlier today I did seem to get one, but I don't know what I did to create one.
I tried to follow your recipe
in creating a lot of files in a folder, and then cut & paste that folder into
another folder,
but so far all the files look okay.

I will wait for Bengt to report on his experiments in changing the configuration
object to
use SLICED_HASH. Meanwhile I'll look into Silva 1.5 and ExtFile on Zope 2.8.+

[Martijn Faassen (faassen)]

Bengt, any information about your experiments with the different configuration
options? (HASH_SLICE)?

[Martijn Faassen (faassen)]

I've just done some experiments with a Zope 2.8.6 setup and Silva 1.5 with
ExtFile 1.5.0 beta 1 (default config, so HASH_SLICE). So far so good - no error
messages whatsoever, files are uploaded successfully and appear in the
repository. I tested it in a ZEO setup and no problems detected yet. Ran a
'siege' against downloading some files and this was fine as well.

So from initial experiments this looks quite solid, but since I haven't
reproduced the "asset loss" issue yet that doesn't necessarily mean the problem
is gone.

It is however nice to find out Silva 1.5 has a working ExtFile again.

[Sylvain Viollon (thefunny)]

ExtFile is no longer maintenained and no longer supported in Silva. And this is no surprising.