.pyc files caused update-charm to fail with merge conflicts

[Stuart Bishop]

When Python scripts are run, they are compiled and persist as .pyc files. These files can cause charm updates to barf if directories are moved around, and maybe in other situations too.

2013/06/14 13:23:44 ERROR worker/uniter/charm: git command failed: exit status 1
path: /var/lib/juju/agents/unit-postgresql-0/charm
args: []string{"pull", "/var/lib/juju/agents/unit-postgresql-0/state/deployer/current"}
From /var/lib/juju/agents/unit-postgresql-0/state/deployer/current
 * branch HEAD -> FETCH_HEAD
CONFLICT (modify/delete): charmhelpers/core/hookenv.pyc deleted in 4e0da4ad349624d3043c040827068a610a276b6e and modified in HEAD. Version HEAD of charmhelpers/core/hookenv.pyc left in tree.
CONFLICT (modify/delete): charmhelpers/core/__init__.pyc deleted in 4e0da4ad349624d3043c040827068a610a276b6e and modified in HEAD. Version HEAD of charmhelpers/core/__init__.pyc left in tree.
CONFLICT (modify/delete): charmhelpers/__init__.pyc deleted in 4e0da4ad349624d3043c040827068a610a276b6e and modified in HEAD. Version HEAD of charmhelpers/__init__.pyc left in tree.
Auto-merging charm-helpers/charmhelpers/core/host.py
Auto-merging charm-helpers/charmhelpers/core/hookenv.py
Automatic merge failed; fix conflicts and then commit the result.

[Stuart Bishop (stub)]

I think the sequence here was:

deploy
mv charmhelpers charm-helpers
charm-update

The charm-update failed as the charmhelpers directory contained files not under revision control, blocking the directory rename being merged.

The same failure should be possible with any files not under version control, such as any charm state being created and stored in the charm directory.

[William Reade (fwereade)]

Hmm. The general principle I'd like to follow here is that charm authors should be jolly careful about what they choose to write to the charm directory; and in *general* I think that is good advice, because there really will be an important conflict if a charm is upgraded to a new version that wants to write data to a file already in use, and we do need to flag that as we did here.

(aside: the unit should have reported an "error" agent-state in status, right? if so, you can ssh in, fix the conflicts, and then `juju resolved $UNIT` to start the agent up again. it's not ideal, but it's a way out of this situation suitable for a charm author; and it could be fairly argued that the charm author's job is to anticipate such issues and not release to users one that suffers from these problems)

Nonetheless, this behaviour is clearly obnoxious.

[Stuart Bishop (stub)]

This case is particularly interesting, and possibly only applies to Python. Files are being created in the hook directory without the dev being conscious of it. Devs cannot choose an alternative location (so even if an alternative location external to the charm directory was encouraged to store state, it wouldn't apply here).

If we prefer devs to handle this and resolve the conflicts, ensuring end users never see the problem, the Python specific fix is either:

- Invoke Python with the -B argument to stop it writing compiled byte code. 'charm proof' could warn about this.
- Have hooks run with the PYTHONDONTWRITEBYTECODE environment variable set so devs never need to worry.

If this is more than a Python specific problem, is there a way of the charm providing git metadata such as what files to ignore (git's equivalent of .bzrignore or similar)?

[William Reade (fwereade)]

I think I'm inclined to view this as python-specific for now (until I hear differently). A .gitignore would, I think, work out of the box as a workaround, but I wouldn't want anyone to consider that to be encouraged. However, if and when we start getting charms into the store via git, it may become de facto supported regardless of our intentions.

Either way, I think it's reasonable to consider the contents of the charm directory to be the charmer's responsibility. In line with that, I think that clear documentation of that fact, with a strong recommendation that python bytecode be suppressed, is maybe best; I'm not convinced it will be entirely simple to check that reliably with `charm proof`.

[William Reade (fwereade)]

Resolved by fix for lp:1232304 -- now, if a charm dir is removed in a future version of a charm, its contents will also be removed.

If you need to preserve user files across charm upgrades, the new version of the charm needs to itself include any directories used for storing those files in the old version. Charm files in the old one, not referenced in the new one, will still be deleted.