(In reply to Benjamin Kreuter from comment #0)
> It seems that opening the Bluetooth settings will make the device
> discoverable again, but does not make the device undiscoverable after the
> settings are closed (this is not intended behavior; devices should only be
> discoverable when the bluetooth settings UI is open).

It does turn it off. Except that bluetoothd and/or the kernel will fail to make it undiscoverable.

Sender ":1.13387" is the Bluetooth settings. See how it requests for "Discoverable" to be switched off, receives a response to the call saying that Discoverable is now off, but receives a signal that Discoverable changed back to "on" after a short time.

This is a dbus-monitor capture on exit:
method call time=1531995892.499232 sender=:1.13387 -> destination=:1.4 serial=94 path=/org/bluez/hci0; interface=org.freedesktop.DBus.Properties; member=Set
   string "org.bluez.Adapter1"
   string "Discoverable"
   variant       boolean false
method call time=1531995892.499459 sender=:1.13387 -> destination=:1.4 serial=95 path=/org/bluez/hci0; interface=org.freedesktop.DBus.Properties; member=Set
   string "org.bluez.Adapter1"
   string "DiscoverableTimeout"
   variant       uint32 0
method call time=1531995892.499485 sender=:1.13387 -> destination=org.freedesktop.DBus serial=96 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=RemoveMatch
   string "type='signal',sender=':1.4'"
method call time=1531995892.499556 sender=:1.13387 -> destination=org.freedesktop.DBus serial=97 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=RemoveMatch
   string "type='signal',sender='org.freedesktop.DBus',interface='org.freedesktop.DBus',member='NameOwnerChanged',path='/org/freedesktop/DBus',arg0='org.bluez'"
method return time=1531995892.499563 sender=org.freedesktop.DBus -> destination=:1.13387 serial=69 reply_serial=97
method call time=1531995892.499610 sender=:1.13387 -> destination=org.freedesktop.DBus serial=98 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=RemoveMatch
   string "type='signal',sender='org.bluez',interface='org.freedesktop.DBus.ObjectManager',path='/'"
method return time=1531995892.499616 sender=org.freedesktop.DBus -> destination=:1.13387 serial=70 reply_serial=98
method return time=1531995892.499665 sender=:1.4 -> destination=:1.13387 serial=379 reply_serial=95
signal time=1531995892.523461 sender=:1.4 -> destination=(null destination) serial=380 path=/org/bluez/hci0; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.bluez.Adapter1"
   array [
      dict entry(
         string "DiscoverableTimeout"
         variant             uint32 0
      )    
   ]
   array [
   ]
method return time=1531995892.523489 sender=:1.4 -> destination=:1.13387 serial=381 reply_serial=94
signal time=1531995892.533281 sender=:1.4 -> destination=(null destination) serial=382 path=/org/bluez/hci0; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.bluez.Adapter1"
   array [
      dict entry(
         string "Discoverable"
         variant             boolean false
      )    
   ]
   array [
   ]
signal time=1531995892.551915 sender=:1.4 -> destination=(null destination) serial=383 path=/org/bluez/hci0; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.bluez.Adapter1"
   array [
      dict entry(
         string "Discoverable"
         variant             boolean true 
      )    
   ]
   array [
   ]

This corresponds with this warning in my journal:
Jul 19 12:28:20 classic kernel: Bluetooth: hci0: last event is not cmd complete (0x0f)

Let's start reassigning this to bluez to start with, as that's where we'll need to dig.