\r in metrics API output

[Evan]

Calling the metrics API for nextcloud has a carriage return in "name: candidate\r"

https://dashboard.snapcraft.io/dev/api/snaps/metrics
{'metrics': [{'metric_name': 'weekly_installed_base_by_channel', 'status': 'OK', 'buckets': ['2019-06-05'], 'series': [{'values': [13], 'name': '13/edge'}, {'values': [32], 'name': '13/candidate'}, {'values': [5], 'name': '14/edge'}, {'values': [144], 'name': '15/stable'}, {'values': [4], 'name': 'beta/pr-890'}, {'values': [3], 'name': '12/stable'}, {'values': [9], 'name': 'latest/stable'}, {'values': [25505], 'name': 'stable'}, {'values': [2], 'name': 'beta/pr-688'}, {'values': [1], 'name': 'beta/pr-862'}, {'values': [5], 'name': '11/stable'}, {'values': [2], 'name': 'stable/pr-388'}, {'values': [33], 'name': 'beta'}, {'values': [2], 'name': 'beta/pr-573'}, {'values': [38], 'name': 'candidate'}, {'values': [202], 'name': '16/candidate'}, {'values': [1], 'name': 'beta/pr-628'}, {'values': [22], 'name': '14/stable'}, {'values': [148], 'name': '14/candidate'}, {'values': [10], 'name': 'stable/394-theming-fix'}, {'values': [7], 'name': 'stable/pr-411'}, {'values': [29], 'name': '14/beta'}, {'values': [2], 'name': '16/stable'}, {'values': [22], 'name': '16/edge'}, {'values': [10], 'name': '12/edge'}, {'values': [12], 'name': '15/edge'}, {'values': [231], 'name': '15/candidate'}, {'values': [4], 'name': '15/beta'}, {'values': [2], 'name': '13/beta'}, {'values': [1], 'name': '16/candidate\r'}, {'values': [1], 'name': 'stable/pr-349'}, {'values': [1], 'name': 'latest/edge'}, {'values': [34], 'name': '13/stable'}, {'values': [1], 'name': 'beta/pr-1038'}, {'values': [55], 'name': 'edge'}], 'snap_id': 'njObIbGQEaVx1H4nyWxchk1i8opy4h54'}]}

[Celso Providelo (cprov)]

I am not sure about the origin of `16/candidate\r` channel, but it seems to work just fine for refreshes.

{{{

$ cat bad_channel.json
{
  "context": [
    {"snap-id": "njObIbGQEaVx1H4nyWxchk1i8opy4h54", "revision": 10, "tracking-channel": "16/candidate\r", "instance-key": "a"}
  ],
  "actions": [
    {"action": "refresh", "snap-id": "njObIbGQEaVx1H4nyWxchk1i8opy4h54", "instance-key": "a"}
  ]
}

$ curl -s -H 'Content-Type: application/json' \
        -H 'Snap-Device-Series: 16' \
        -H 'Snap-Device-Architecture: amd64' \
        'https://api.snapcraft.io/v2/snaps/refresh' \
        -d '@bad_channel.json' \
| jq '.'
{
  "error-list": [],
  "results": [
    {
      "effective-channel": "16/candidate",
      "instance-key": "a",
      "name": "nextcloud",
      "released-at": "2019-05-23T19:23:14.127486+00:00",
      "result": "refresh",
      "snap": {
        "created-at": "2019-05-23T16:15:50.409353+00:00",
        "download": {
          "deltas": [],
          "sha3-384": "e07d888235eae56edcae9c085b1d28dbbd0207a9f98ded18bfe6db79e824f9dc19c6dc308fbbe142d9ea1a00be29f815",
          "size": 252055552,
          "url": "https://api.snapcraft.io/api/v1/snaps/download/njObIbGQEaVx1H4nyWxchk1i8opy4h54_13518.snap"
        },
        "license": "AGPL-3.0+",
        "name": "nextcloud",
        "prices": {},
        "publisher": {
          "display-name": "Nextcloud",
          "id": "lBnslZGp88VvX4EJSAvS2AG2Iryb1MNU",
          "username": "nextcloud",
          "validation": "verified"
        },
        "revision": 13518,
        "snap-id": "njObIbGQEaVx1H4nyWxchk1i8opy4h54",
        "summary": "Nextcloud Server - A safe home for all your data",
        "title": "Nextcloud",
        "type": "app",
        "version": "16.0.1snap1-rc1"
      },
      "snap-id": "njObIbGQEaVx1H4nyWxchk1i8opy4h54"
    }
  ]
}

}}}

It seems that we are "trimming" channels before evaluating operations, but the raw content is recorded in the metrics.

IMO, the trimming is masking deeper problems and should be removed, the refresh operation would fail (client would be fixed if necessary) and no metric would be recorded.

[Evan (ev)]

/dev/api/snaps/metrics (for lxd)

{"metrics": [{"status": "OK", "series": [{"values": [1], "name": "3.3/stable"}, {"values": [8], "name": "3.0"}, {"values": [2], "name": "3.0/edge"}, {"values": [14], "name": "latest/stable"}, {"values": [15570], "name": "stable"}, {"values": [58], "name": "candidate"}, {"values": [177], "name": "stable/ubuntu-19.10"}, {"values": [17], "name": "beta"}, {"values": [5], "name": "2.0/stable"}, {"values": [6], "name": "3.0/stable/ubuntu-18.04"}, {"values": [549], "name": "3.0/stable"}, {"values": [6], "name": "stable/ubuntu-18.04"}, {"values": [22135], "name": "stable/ubuntu-19.04"}, {"values": [2], "name": "3.0/candidate"}, {"values": [67625], "name": "stable/ubuntu-18.10"}, {"values": [7], "name": "/stable/ubuntu-19.04"}, {"values": [1], "name": "2.0/candidate"}, {"values": [280], "name": "edge"}, {"values": [152], "name": "3.0/stable/ubuntu-19.04"}, {"values": [1723], "name": "3.0/stable/ubuntu-18.10"}, {"values": [1], "name": "latest/edge"}, {"values": [30], "name": "/stable/ubuntu-18.10"}, {"values": [5], "name": "3.0/stable/ubuntu-19.10"}], "buckets": ["2019-06-06"], "snap_id": "J60k4JY0HppjwOjW8dZdYc8obXKxujRu", "metric_name": "weekly_installed_base_by_channel"}]}

"3.0" and "/stable/ubuntu-19.04" are not valid channel names.

[Evan (ev)]

I've disabled https://app-sjg.marketo.com/#SC9748C3 until this is fixed.

[Simon Davy (bloodearnest)]

I've done some poking around in the db, and the '\r' is in the raw db value for nextcloud, as is the split channels for lxd, so the problem is definitely in the way snapstats miner is generating/storing the aggregates, as opposed to a bad transform when reading them.

I'm looking into fixing the generation, but for bad data already present, it's tricky to know what to do. In this case, on reading the data, we could maybe have a validation of channel names, and just drop bad ones, which should mean the data that is there can be parsed buy the consuming api, although there well be missing data.

Backfilling a fix is tricky, as we don't like to update in cassandra, as it leaves tombstones around.

[William Grant (wgrant)]

It's most likely that the bad character in the channel is coming from a client calling the refresh API with it.

[Simon Davy (bloodearnest)]

On Wed, 12 Jun 2019 at 14:05, William Grant <email address hidden> wrote:

> It's most likely that the bad character in the channel is coming from a
> client calling the refresh API with it.
>

Right, that would make sense. I haven't found anything in the snapstats
code which would explain it, the strings are never parsed, they're opaque.

I would suspect the same for the channel "3.0" and "/stable/ubuntu-18.10",
"/stable/ubuntu-19.04" channel names.

So in that case, the proper fix is channel name validation/sanitization in
snapdevicegw, I think?

> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1831910
>
> Title:
> \r in metrics API output
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snapstore/+bug/1831910/+subscriptions
>

--
Simon

[Celso Providelo (cprov)]

I don't think we should concerned about backfilling metrics, once the ingesting part is fixed, the bad tags will fade in a week.

At this point, I am inclined to stop mangling (trimming, adding missing parts, etc) the given bad-channels and fail the refresh request.

[Simon Davy (bloodearnest)]

FTR, 3.0 is actually a valid channel string, it defaults to 3.0/stable. /stable/ubuntu-{19.04,18.04} is technically not valid, but refresh api actually parses it ok, even it it's actually the wrong value (should be 3.0/stable/ubuntu-19.04)