[2.4] Master process changes can't notify rack controllers to start image import

[Andres Rodriguez]

Recently the master process was refactored to better handle MAAS internal services. This refactoring changed the image import service to run on the master only.

The downside is that its unable to communicate to the rack controllers to start the image import.

Proposed fixes:

1. Have the master process communicate the workers that images have been imported, so they can tell the rack controllers to start the image import.
2. Have the image import run on a worker process.

Decided solution:

Since the master process runs critical operations for MAAS, we've decided to have the workers run the image import and evaluate the impact of performance. However, keeping the master process as light as possible (no image import) would make sure other internal services don't get impacted.

[Chris Gregan (cgregan)]

Escalated to field critical due to it blocking testing completely

[Ashley Lai (alai)]

We still seeing the issue. The log is attached.

2.4.0~beta2-6807-gaa74361-0ubuntu1~18.04.1~20180409~ubuntu18.04.1

Traceback (most recent call last):
  File "foundation/bin/wait-for-image-sync", line 30, in <module>
    main(sys.argv)
  File "foundation/bin/wait-for-image-sync", line 26, in main
    wait_for_rack_image_sync(infra['maas_admin'])
  File "/home/ubuntu/cpe/foundation/bin/maas_cli.py", line 670, in wait_for_rack_image_sync
    sync_status))
Exception: Timed out waiting for racks to sync images: {'synced', 'unknown'}

[Blake Rouse (blake-rouse)]

@alai - please file a new bug. This is specific to internal operation of MAAS that has been fixed.

[John George (jog)]

I've filed bug https://bugs.launchpad.net/maas/+bug/1764830 for the continuing image sync failures we're seeing in the Solutions QA CI.