[UI] Need to flush the discovered devices list

[Mark Shuttleworth]

Over time the discovered device list will have things in it which are no longer relevant. It would be useful to be able to flush from the list all devices last seen more than <n> hours / days ago.

[Mike Pontillo (mpontillo)]

See also: bug #1680632, which is very similar.

Note that MAAS uses the observed neighbour data to determine which IP address in the available pool was the least-recently used. So I would think about the tradeoffs between deleting the data and simply hiding discoveries from the user that are no longer relevant.

MAAS does have a way to clear out *all* the discoveries via the API/CLI (maas $PROFILE discoveries clear all=true), but (as suggested) we should allow deletion based on a time threshold, and expose it in the UI.

[Mark Shuttleworth (sabdfl)]

Ah, the observed address data as input to address allocation is cool.

But still - an address that has not been seen for a week and does not
respond to a ping or other detection should surely be deletable?

Perhaps there should be a customisable threshold (say, 1 week by
default) for auto-scrubbing. At that point, the address is hard-probed
to see if it is responsive in a way that might cause problems, and if
not, then the record can safely be removed altogether.

Separately, I do think the user should be able to trigger a one-time
iteration of that scrub with a custom threshold ("scrub addresses that
have not been seen in n days/hours").

Mark

[Mike Pontillo (mpontillo)]

Yes, and your comment about hard-probing reminds me of a subtle issue with aging things out: we don't do any hard-probing by default.

That is, discovered neighbour data might be out-of-date if active discovery isn't enabled. So you might bring up a device (let's say something that is fairly transparent on the network, like a firewall or IoT device) that doesn't do much ARP broadcasting. Such a device might keep to itself, and only broadcast ARP initially to find its gateway. Now, if active discovery enabled, every so often we'll confirm that the device is still using the address, and we can be reasonably confident when aging out entries that haven't been seen in awhile.

Unfortunately (and for good reason; this was hotly debated) active discovery is turned off by default on each subnet, and must be specifically enabled. That means you could easily have devices that were seen once, two weeks ago, and is still using its IP address.

So to solve this once-and-for-all, we would want a way to make it easier to have active discovery enabled; the active discovery interval could be coupled to the cleanup mechanism. So if the active discovery interval is set to 3 hours, and it's been some multiple of 3 hours since the device was last seen, we should be able to remove it right then and there.

[Mark Shuttleworth (sabdfl)]

Initially we could keep these matters orthogonal.

We could always (not as an option) keep actively probing things we
observed. In other words, if we saw something on an address, it's on the
observed list, we could ping that once a day. If its not seen for seven
days, we could drop it.

That's independent of HOW we observed something in the first place. If
we are passively watching, and we see something, it's on the probe list.
If we are allowed to actively watch, and we see something, then it's on
the same probe list.

Would you have any objections to pinging something we observed, once a
day? That seems much less contentious to me than an active scan of a VLAN.

Mark

[Mike Pontillo (mpontillo)]

I think a "ping once per day, expire after one week" approach sounds very reasonable. The required scope might be a little much for a bug fix, but we can take a look.

[Martin Storey (cassiocassio)]

At CPT Jan 2019 we discussed UI to:

- clear an individual device from the list
- clear the whole list, and force it to rebuild

Transient interfaces, e.g. phones might never be seen again.

Obviously devices still on the network will reappear at next scan. Clear and "suppress" this interface would require us to track and manage a list of "things to keep hidden" which seems baroque and unnecessary.

We also discussed if the auto-scrubbing "Devices seen in the last 5 days" should be on the controller, or could be a function of the UI, and tended towards the former.

[Mark Shuttleworth (sabdfl)]

It doesn't make sense to maintain a list of 'known but ignored devices',
those should just be added as known devices.

It would make sense to be able to see a list of known devices which have
not been seen in a long time, too. Those might have been decommissioned
and therefor be easy targets for cleanup in the database.

For the UI, it does make sense to be able to zoom in on things which are
'recent' or things which are 'old'.

Mark

[Lilyana Videnova (lilyanavidenova)]

I'm closing this bug as we added:

- the functionality to clear all discoveries
- the functionality to clear an individual discovery
 (MP - https://code.launchpad.net/~steverydz/maas/+git/maas/+merge/363852 )
- in addition, the user can re-order the list of network discoveries by time, thus being able to see recent or old discoveries

We are proposing further investigation into the experience of network discovery as a whole as it is currently obscured and fragmented - parts are in network discovery dashboard, on each subnet page, and the network services tab on the settings page.
With this work we will investigate whether we should surface on the UI the option to actively scan the network, while keeping the user aware of the potential security issues around it (using the 'maas admin discoveries scan' command).
(Github epic https://github.com/ubuntudesign/MAAS-squad/issues/962)