Album art is missing for older albums and some tracks but not others

[Dave Morley]

Description:
In older albums some tracks have album art and others don't, occasionally there is no album art at all and on newer purchases, on the whole, the album art is there.

Service:
U1 Sync for Music
~/.ubuntuone/Purchased from Ubuntu One

U1 Music Player
https://one.ubuntu.com/music/player

Steps:
1. Goto the folder above if you have purchased music previously go through all of the folders note the tracks that don't have album art
2. Goto the link above
3. From the player select the tracks and play them see if there is album art

Expected Results:
I expect to see album art for all tracks.

Actual Results:
Currently the older the purchase the less likely it is to have complete album art coverage.

[James Tait (jamestait)]

Initial impressions are that there are a couple of things going on here, that may or may not be related.

Following the steps outlined above, I see nothing untoward with my own purchased music, regardless of time of purchase - those tracks that do not display a thumbnail in nautilus also show no album art in the ID3 tags (verified with eyeD3). The thumbnails in nautilus are extracted from the ID3 tags by nautilus itself, so this makes perfect sense.

In the web music player things are a little (OK, a lot!) more complicated. This at least partly stems from the fact that we're using two versions of the music API to generate the page - the v2 API for artwork and the v1 api for everything else.

For the artists view, https://one.ubuntu.com/music/player/#view=artists - selected via the Artists link on the left, and also the default when you hit https://one.ubuntu.com/music/player/ - the thumbnails are requested from https://one.ubuntu.com/api/music/v2/artists/<v2_artist_id>/art/ and the image served is the cover art from the ID3 tags of the most recently parsed music file that has cover art and whose artist_id matches the one requested. Though it's an implementation detail, I'll mention that the v2_artist_id is the hex digest of the MD5 hash of the UTF-8 encoding of the lower-cased value of the album_artist ID3 tag, or the artist ID3 tag if album_artist is not present, or 'Unknown artist' if neither is present. The v2_artist_id does not necessarily match the v1_artist_id due to case sensitivity - the v1_artist_id is generated without converting the artist to lower case.

For the albums view, https://one.ubuntu.com/music/player/#view=albums - selected via the Albums link on the left - the thumbnails are requested from https://one.ubuntu.com/api/music/v2/albums/<v2_album_id>/art/ and the image served is the cover art from the ID3 tags of the most recently parsed music file that has cover art and whose album_id matches the one requested. Again, though it's an implementation detail, I'll mention that the v2_album_id is the hex digest of the MD5 hash of the UTF-8 encoding of the lower-cased value of the album ID3 tag, or 'Unknown album' if album is not present, prepended to the UTF-8 encoding of the lower-cased value of the album_artist ID3 tag, or the artist ID3 tag if album_artist is not present, or 'Unknown artist' if neither is present. Again, the v2_album_id does not necessarily match the v1_album_id due to case sensitivity - the v1_album_id is generated without converting the artist to lower case.

Now, I can see some potential problems with the algorithms for selecting artwork, that I'll look into more deeply. Once I'm satisfied that we are indeed requesting the correct artwork from the metadata store, I can dig further.

[James Tait (jamestait)]

I've managed to reproduce the issue in the web UI on production from my own music library. I have two albums purchased from u1ms by Green Day, 25 tracks in all, all of which have album art. In the web UI I see no artist art for Green Day. Clicking on the artist though, I do see the album art for each of the albums.

Looking deeper, I tried issuing v1 API requests from the browser:

  https://one.ubuntu.com/api/music/1.0/getIndexes.view?c=jayteeuk&v=1.2&f=json

In the response I see:

{
    "name": "G",
    "artist": [
        {
            "has_cover_art": true,
            "id": "8c97626c3bda0721059f5eddbd2d255d",
            "name": "Garbage",
            "u1CoverId": "c8e8df895c2cae166bad027fdf15335b"
        },
        {
            "has_cover_art": false,
            "id": "bb28f52f24482ee421aa70bd15a9e7ad",
            "name": "Green Day",
            "u1CoverId": "9f963d602836426c140a637e01f169ac"
        }
    ]
},

The u1CoverId is correct, but for some reason we're getting has_cover_art: false. The equivalent request in the v2 API:

  https://one.ubuntu.com/api/music/v2/artists/

Yields:

{
    "artist": "Green Day",
    "song_count": 171,
    "artist_art_url": "https://one.ubuntu.com/api/music/v2/artists/9f963d602836426c140a637e01f169ac/art/",
    "artist_url": "https://one.ubuntu.com/api/music/v2/artists/9f963d602836426c140a637e01f169ac/",
    "album_ids": [
        "72e473edce568e5c8495da45bb376b35",
        "d1b885315d2e80b866affde980ec4b85",
        "e90bce22fb005f6b8fbe32b512c047be",
        "e35b13184ebdf98bbb7d03774d9fd4cd",
        "d1e9c86f01b3b62041e5feb216696f65",
        "be17407b1e4d68594a4766c72395ba13",
        "42c5213aee8851c9c8578bbe0a840a3e",
        "6e10fbed18ef365ac03092f46e727332",
        "8222b362d22fec0e145b19c89cf99d78",
        "8b812eff93da0634c50af089b26b95a3",
        "a9da66d0bd0c26739571bfb92fea51b4",
        "9530ecb21b4a09a1116146380d769b42"
    ],
    "id": "9f963d602836426c140a637e01f169ac"
},

This includes a bunch of albums that I've ripped myself, none of which has any album art. My gut feel is that this is somehow interfering - that the code that determines whether a given artist has cover art is only looking at the most (or least) recent track for a given artist, and if that has no artwork then any tracks with artwork will be disregarded. The investigation continues.

[James Tait (jamestait)]

And indeed, the culprit appears to be in this part of the code:

        if artist not in processed_artists:
            # Create the artist an on-the-fly generated id, which is just
            # and md5 of its name.
            artist_id = md5(artist).hexdigest()
            artist_dict = {'artist': artist,
                           'id': artist_id,
                           'last_modified': last_modified,
                           'has_cover_art': song['has_cover_art']}
            if request.response_type != 'json':
                for key, value in artist_dict.items():
                    if isinstance(value, basestring):
                        artist_dict[key] = value
            artists[cgi.escape(letter)].append(artist_dict)
            processed_artists.append(artist)

Thus, if the first track for a given artist has no cover art, the artist record returned will have the has_cover_art flag set to False.