It turns out this was a symptom of a bigger problem. Background: custom playlists are stored in the panel tree model (column 2), as well as a playlist-to-iter cache (playlist_nodes). These two need to be always in sync.
The problem is simply that deleted playlists were not deleted from the playlist_nodes cache.
The thing about opening playlist files was only a coincidence; it happens to trigger code that requires playlist_nodes to be up-to-date. (How can opening a playlist file trigger something in the panel UI code? No idea.)
While fixing this, I found another bug: update_playlist_node modifies playlist_nodes, but not the playlist object in the tree model. I've fixed that as well.
Fix committed to 0.3.2 and trunk.