destroy-environment: Too many files to remove

[Sidnei da Silva]

Trying to move from pyjuju to gojuju, reusing the same environment name and bucket names:

$ juju destroy-environment
error: Too many files to remove: 382

[Sidnei da Silva (sidnei)]

Apparently it's the control-bucket that's contains those files.

[John A Meinel (jameinel)]

This is part of the openstack code:
func (s *storage) deleteAll() error {
        names, err := s.List("")
        if err != nil {
                return err
        }
        // Remove all the objects in parallel so that we incur less round-trips.
        // If we're in danger of having hundreds of objects,
        // we'll want to change this to limit the number
        // of concurrent operations.
        if len(names) > 100 {
                return fmt.Errorf("Too many files to remove: %d", len(names))
        }
        var wg sync.WaitGroup
        wg.Add(len(names))
        errc := make(chan error, len(names))
        for _, name := range names {

[John A Meinel (jameinel)]

Basically, Openstack doesn't have a delete many request, so we have to request them one-by-one. We limited it to 100 because we didn't want to have an unbounded number of goroutines (they are cheap, but how cheap?).

Since we are hitting this in practice, we should switch out the logic for a worker based one. (Spawn up to N workers, all working on a queue of M items to remove.)