Compact encoding delivery bodies for webhook

[Shih-Yuan Lee]

Some webhook relay service like https://smee.io/ will make compact encoding [1] delivery bodies automatically when it transfers the delivery bodies it rececived.

If we signed delivery bodies with HMAC-SHA1, the client for https://smee.io/ won't be able to verify the delivery bodies directly.

Is it posiible to make the compact encoding delivery bodies from Launchpad webhook by default so it can save few network usage, improve the compatibilty for other programming languages [2], and avoid the signed delivery bodies problem of some webhook relay service like https://smee.io/?

Current Launchapd delivery body looks like,

{"action": "modified", "new": {"source_git_repository": "/~fourdollars/oem-dev-tools/+git/launchpad.api", "prerequisite_branch": null, "target_branch": null, "description": null, "source_branch": null, "registrant": "/~fourdollars", "queue_status": "Work in progress", "whiteboard": null, "source_git_path": "refs/heads/test", "prerequisite_git_path": null, "target_git_path": "refs/heads/master", "preview_diff": "/~fourdollars/oem-dev-tools/+git/launchpad.api/+merge/385119/+preview-diff/906725", "commit_message": null, "target_git_repository": "/~fourdollars/oem-dev-tools/+git/launchpad.api", "prerequisite_git_repository": null}, "old": {"source_git_repository": "/~fourdollars/oem-dev-tools/+git/launchpad.api", "prerequisite_branch": null, "target_branch": null, "description": null, "source_branch": null, "registrant": "/~fourdollars", "queue_status": "Needs review", "whiteboard": null, "source_git_path": "refs/heads/test", "prerequisite_git_path": null, "target_git_path": "refs/heads/master", "preview_diff": "/~fourdollars/oem-dev-tools/+git/launchpad.api/+merge/385119/+preview-diff/906725", "commit_message": null, "target_git_repository": "/~fourdollars/oem-dev-tools/+git/launchpad.api", "prerequisite_git_repository": null}, "merge_proposal": "/~fourdollars/oem-dev-tools/+git/launchpad.api/+merge/385119"}

Compact encoding delivery body looks like,

{"action":"modified","new":{"source_git_repository":"/~fourdollars/oem-dev-tools/+git/launchpad.api","prerequisite_branch":null,"target_branch":null,"description":null,"source_branch":null,"registrant":"/~fourdollars","queue_status":"Work in progress","whiteboard":null,"source_git_path":"refs/heads/test","prerequisite_git_path":null,"target_git_path":"refs/heads/master","preview_diff":"/~fourdollars/oem-dev-tools/+git/launchpad.api/+merge/385119/+preview-diff/906725","commit_message":null,"target_git_repository":"/~fourdollars/oem-dev-tools/+git/launchpad.api","prerequisite_git_repository":null},"old":{"source_git_repository":"/~fourdollars/oem-dev-tools/+git/launchpad.api","prerequisite_branch":null,"target_branch":null,"description":null,"source_branch":null,"registrant":"/~fourdollars","queue_status":"Needs review","whiteboard":null,"source_git_path":"refs/heads/test","prerequisite_git_path":null,"target_git_path":"refs/heads/master","preview_diff":"/~fourdollars/oem-dev-tools/+git/launchpad.api/+merge/385119/+preview-diff/906725","commit_message":null,"target_git_repository":"/~fourdollars/oem-dev-tools/+git/launchpad.api","prerequisite_git_repository":null},"merge_proposal":"/~fourdollars/oem-dev-tools/+git/launchpad.api/+merge/385119"}

--
[1]:
https://docs.python.org/3/library/json.html

Compact encoding:
>>> import json
>>> json.dumps([1, 2, 3, {'4': 5, '6': 7}], separators=(',', ':'))
'[1,2,3,{"4":5,"6":7}]''

[2]:
Golang: https://golang.org/pkg/encoding/json/#example_Marshal
Rust: https://docs.rs/json/0.12.4/json/
Ruby: https://ruby-doc.org/stdlib-2.6.3/libdoc/json/rdoc/JSON.html
Java: https://code.google.com/archive/p/json-simple/wikis/EncodingExamples.wiki
JavaScript: https://www.w3schools.com/js/js_json_stringify.asp (This is what https://smee.io/ used now.)

[Shih-Yuan Lee (fourdollars)]

FYI, `orig == json.dumps(json.loads(minified))` in Python so I can use json.dumps(json.loads(minified)) to recover the delivery bodies and then check the signature.

[Shih-Yuan Lee (fourdollars)]

>>> minified == json.dumps(json.loads(orig), separators=(',', ':'))
True

[Shih-Yuan Lee (fourdollars)]

https://docs.python.org/3/library/json.html

Compact encoding:
>>> import json
>>> json.dumps([1, 2, 3, {'4': 5, '6': 7}], separators=(',', ':'))
'[1,2,3,{"4":5,"6":7}]'

[Colin Watson (cjwatson)]

There's no particular reason to assume that other relay services might not mutate the body in some other way, nor that smee.io's mutation is going to remain the same forever, so we aren't going to attempt to optimise this specifically for smee.io. The only safe way for a "relay service" to deal with the body of a signed webhook is to leave it untouched; I'd consider this a bug in smee.io.

[Shih-Yuan Lee (fourdollars)]

I agree.
It is a bug in smee.io because it used the result of JSON.parse to save the deliver body and then used the result of JSON.stringify to send the deliver body.

[Shih-Yuan Lee (fourdollars)]

After I compared Python json.dumps() with other program languages, I found most of other major program languages used the compact encoding by default.

Golang: https://golang.org/pkg/encoding/json/#example_Marshal
Rust: https://docs.rs/json/0.12.4/json/
Ruby: https://ruby-doc.org/stdlib-2.6.3/libdoc/json/rdoc/JSON.html
Java: https://code.google.com/archive/p/json-simple/wikis/EncodingExamples.wiki
JavaScript: https://www.w3schools.com/js/js_json_stringify.asp (This is what https://smee.io/ used now.)

[Colin Watson (cjwatson)]

Compact encoding isn't a priority for us at all; micro-optimising the number of bytes we send (over TLS etc., and non-interactively!) isn't something we're concerned about. Readability is generally more important.

[Shih-Yuan Lee (fourdollars)]

To be honest, the default output of json.dumps(orig) doesn't look much different to json.dumps(orig, separators=(',', ':')) unless using json.dumps(orig, indent=4) or json.dumps(orig, indent=2) that is really to make it eaiser to read.

[Shih-Yuan Lee (fourdollars)]

BTW, I found the content returned from https://api.launchpad.net are not compact encoding either.
If making https://api.launchpad.net return the compact encoding content by default, it may save a lot of network usage.

[Colin Watson (cjwatson)]

This is not something we plan to spend any effort on, sorry.

In many cases the network usage will be effectively zero anyway, since differences less than the size of a packet don't matter. And even when that isn't the case, it's negligible.