Add free space and storage plan changes notifications

[Alejandro J. Cura]

We need to add plan changes and free space notifications to the storage protocol as used by syncdaemon, so the server can notify the clients of out-of-quota beforehand.

[John O'Brien (jdobrien)]

So the basic Idea behind the solution presented in this proposal is "The client must keep the user's available space synchronized with the server at all times"

Unfortunately, in the past these solutions have created problems because they didn't consider that it creates a lot of chatter on the wire along with a lot of overhead on the server side. (I can explain this further if need be, but I think most of us are aware of it)

I would recommend we look at this a little differently and solve differently

Here's a different way of looking at the problem:

1) 99.9% of the time, the quota and available space don't matter. Very few users ever go over quota.
2) Knowing the quota (from the client's perspective) only becomes relevant when:
   2a The user is already over quota and the client can't upload anything.
   2b The client is attempting to upload a file large enough where it will go over quota.
3) Because of 1 & 2, the client doesn't really need to constantly synchronize the quota information with the server.

A different solution:

Using a Try, Fail, React pattern for uploads the client always tries to upload a file, 99% of the time this will work. For most clients they will never fail. The client doesn't have a quota that it constantly checks before doing uploads.

React the Failure:

From the user's perspective they should be notified that they need to correct their quota problem by deleting files or buying more space.

From the client and server perspective it needs to handle the two conditions a little differently.

When a client upload fails due to a quota issue, the server will send the client a quota message with available space. Based on the message the client will know if if it was due to either 2a or 2b above and the way it resolves it is slightly differently.

For Condition 2a, the server knows the user is over quota, in this case, the server will send a quota notification when the user does something to correct this (when they buy more space or delete some files) When the client is notified the quota is fixed, it will reattempt the uploads (free up the upload queue)

For Condition 2b, the quota notification sent when the user attempted to go over quota will have a positive number in it. The client could attempt to upload smaller files. When it has reached this condition, the client should retry the upload at a realistic interval.

Note that on the server side there is only one change, we send quota notification only when the available space goes from <=0 to >0 This can happen when the user buys space or they have deleted files.

[Natalia Bidart (nataliabidart)]

Hi John,

thanks for the clear explanation. Despite this solution works from the syncdaemon point of view, it does not work that well from the user point of view in the sense of providing visibility about quota usage. Suppose there is an user with the control panel opened and s/he buys more storage on the web, or s/he uploads a big file (without running out of quota): we will not show the proper quota change in those cases.

If providing notifications for every quota change is too expensive, can we at least notify when:

* the plan changed
* the quantity of the plan changed
* the free space changed by a quantity larger than <some threshold> (maybe 100MB?)

There is already a message in the protocol for account changes, maybe we can use this? we may want to improve it by adding more fields (right now only has one: free_space)?

[John O'Brien (jdobrien)]

On 05/08/2011 07:38 AM, Naty Bidart wrote:
> Hi John,
>
> thanks for the clear explanation. Despite this solution works from the
> syncdaemon point of view, it does not work that well from the user point
> of view in the sense of providing visibility about quota usage. Suppose
> there is an user with the control panel opened and s/he buys more
> storage on the web, or s/he uploads a big file (without running out of
> quota): we will not show the proper quota change in those cases.
>
> If providing notifications for every quota change is too expensive, can
> we at least notify when:
>
> * the plan changed
> * the quantity of the plan changed
> * the free space changed by a quantity larger than<some threshold> (maybe 100MB?)
>
So we need to be clear here, By 'the plan changed' I assume you mean the storage plan.
There are only two ways a storage plan can change, it either changed the quota or the payment period.
This is where we need to be careful. Quota is easy as it changes their max storage bytes. However payment period is not
as easy and will likely change as we move to Ubuntu Pay. I don't think any client should be dependent on whether the
payment period is annually or monthly so I'll assume you didn't mean this.

So the question really is:
1. Did the user's maximum storage space change
2. Did the user's available storage space change.

For 1. we don't have a protocol message for this. Although if this message were added to the protocol, it would not be
expensive as this is not something that changes very often.

For 2. This is the expensive one, which is why I wanted to focus one why a client needs to keep this in sync. I used the
needs of the syncdaemon as an example. Since if it were designed this way, it would create a performance burden.
Additionally this is not really new news, the pre-generation clients rely on the idea of keeping free-space in sync at
all times but due to the overhead we were phasing it out. Now this is being brought up again and I want to be sure we
really need to go down this road.

> There is already a message in the protocol for account changes, maybe we
> can use this? we may want to improve it by adding more fields (right now
> only has one: free_space)?
>

There is no message in the protocol AFAIK about account changes. Can you be more specific?

[Natalia Bidart (nataliabidart)]

Yes, currently there is this message in the protocol:

    416 message AccountInfo {
    417 optional uint64 purchased_bytes = 1;
    418 }

that, as far as I understand, can be sent from the server to the client with the new amount of "purchased_bytes".

We had a session about this yesterday (attendees, among others, were Lucio, Alecu, Martin, Chipaca and myself), and since the notes that Chipaca took seems to have gone away, I will repeat the action items we defined:

* foundations:
  - piggy-back on GetDelta message responses to send the free_space (or quota_used). Since the SV_NEW_GENERATION is already being sent to all the clients, and that triggers a GetDelta, we wouldn't be adding any overhead.
  - start using the AccountInfo message to notify clients when the storage space changes (either increases or decreases).

* desktop+ or foundations:
  - make syncdaemon send two different DBus signals, one for free space changes, and another for storage space changes.

* desktop+:
  - make control panel connect to the aforementioned DBus signals and update the UI accordingly.
  - analyze if some work is needed to update out of quota notifications.
  - add a new message to the message menu (later this will be the services applet) to inform the user that his/her plan downgraded (need confirmation on this item since I don't recall exactly what we agreed on).

* futures:
  - define account rest api v2 to clena up v1 and remove deprecated and unused fields/info.

* web:
  - enrich account rest call to also send the list of plans that the user is subscribed to. Also send information about plan frequency (monthly, etc)

[Natalia Bidart (nataliabidart)]

Related to this bug is LP: #759890.

[Natalia Bidart (nataliabidart)]

Also related: bug #785811