* Removing from the swatch (but keeping at the bottom palette).
* Moving it to the LAST color, so that it won't break the layout.
* Showing it in its own row, with rest of the colors below it.
Personally, I believe the middle solution (moving to the last) seems to be a good compromise (no loss of functionality, but also easy enough to implement).
About the "X", I can think about:
* Removing from the swatch (but keeping at the bottom palette).
* Moving it to the LAST color, so that it won't break the layout.
* Showing it in its own row, with rest of the colors below it.
Personally, I believe the middle solution (moving to the last) seems to be a good compromise (no loss of functionality, but also easy enough to implement).