Add a self-test to check that all job files are defined in setup.cfg and POTFILES.in

[Sylvain Pineau]

I'd like to see checkbox failing to build a package if job filenames are not defined in the following files:
- setup.cfg
- POTFILES.in

And also that all suites are properly identified in local.txt.in.

[Daniel Manrique (roadmr)]

Rather than Wishlist, this should be Medium so it doesn't get forgotten. It would be very good to have, since once the package goes out without files that may be needed, things break during testing and that implies a longer degugging cycle. So this will pay for itself in time savings.

[Daniel Manrique (roadmr)]

What would be an appropriate place for setup.cfg and POTFILES.in verification? I don't think it fits within checkbox/tests/. Maybe it should be part of setup.py itself?

Also, suite identification in local.txt.in, how would that work? since job files are slurped in the command (see how it just cats message files), it would possibly require some evil parsing of the command; or actually running the message files through the appropriate plugins, then comparing that with the list of messages that result from just reading all the job files.

The idea of this comment is to start a discussion on how to implement this. We can also take advantage of some face time to solve this in 15 minutes rather than taking hours to do it via comments :)

[Marc Tardif (cr3)]

How about a tests/ directory? Let's discuss...