Lots of UserGameProfiles created simultaneously

[Matt Giuca]

For every "real" user game profile created, there are about 10 completely blank ones, without even a user.

[Matt Giuca (mgiuca)]

Actually, I think this is related to bug #788592. Each of those errors is creating a blank UGP, and crashing before it gets to populate it.

[Matt Giuca (mgiuca)]

OK so fixing bug #788592 fixes the lots of blank UGPs problem. But there is still a problem where if a game makes lots of requests as it is starting up, there will be lots of concurrent UGPs created for the same game/user pair. This is an extremely bad situation -- it shouldn't matter much in the long run because once a game is reloaded, it will always choose the same UGP and ignore the others, but it could introduce random bugs into games that are being played for the first time.

[Scott Ritchie (sritchie)]

I believe this is related to an issue we were discussing a while back on
duplication of users was happening in the datastore, when lookup calls were
made too quickly after an object was created. I don't think we ever found a
way around this and just made sure calls weren't being made too quickly on
our end.

The only way I can really see around this at the moment, is if you look up a
UGP by Game/User and the results list has more than 1 element, delete the
rest. This will only happen the second time a lookup is made after
sufficient time has passed since the creation.

On Thu, May 26, 2011 at 11:20 PM, Matt Giuca <email address hidden>wrote:

> OK so fixing bug #788592 fixes the lots of blank UGPs problem. But there
> is still a problem where if a game makes lots of requests as it is
> starting up, there will be lots of concurrent UGPs created for the same
> game/user pair. This is an extremely bad situation -- it shouldn't
> matter much in the long run because once a game is reloaded, it will
> always choose the same UGP and ignore the others, but it could introduce
> random bugs into games that are being played for the first time.
>
> ** Summary changed:
>
> - Lots of blank UserGameProfiles
> + Lots of UserGameProfiles created simultaneously
>
> ** Changed in: mugle
> Importance: Undecided => High
>
> ** Changed in: mugle
> Status: New => Triaged
>
> --
> You received this bug notification because you are subscribed to MUGLE.
> https://bugs.launchpad.net/bugs/788590
>
> Title:
> Lots of UserGameProfiles created simultaneously
>
> Status in Mars Programming Language:
> Invalid
> Status in Melbourne University Game-based Learning Environment:
> Triaged
>
> Bug description:
> For every "real" user game profile created, there are about 10
> completely blank ones, without even a user.
>

[Matt Giuca (mgiuca)]

Yes. The correct way around this is to use transactions. But they are
extremely tricky in App Engine due to the restriction we discussed earlier
(only able to edit one entity group). Having said that, for the purpose of
avoiding duplicates, they shouldn't be too hard to pull off.

[Scott Ritchie (sritchie)]

I'm honestly not sure using transactions works in this case. I had tried
wrapping it all in one transaction the first time to see if i could avoid
this issue, but i wasn't able to get it to work.

On Fri, May 27, 2011 at 12:03 AM, Matt Giuca <email address hidden>wrote:

> Yes. The correct way around this is to use transactions. But they are
> extremely tricky in App Engine due to the restriction we discussed earlier
> (only able to edit one entity group). Having said that, for the purpose of
> avoiding duplicates, they shouldn't be too hard to pull off.
>
> --
> You received this bug notification because you are subscribed to MUGLE.
> https://bugs.launchpad.net/bugs/788590
>
> Title:
> Lots of UserGameProfiles created simultaneously
>
> Status in Mars Programming Language:
> Invalid
> Status in Melbourne University Game-based Learning Environment:
> Triaged
>
> Bug description:
> For every "real" user game profile created, there are about 10
> completely blank ones, without even a user.
>

[Prageeth Silva (prageethsilva)]

Isn't transactions used in cases such as u want multiple operations
happening OR none at all.
This means, that if u have five operations you want happening and and the
third fails, the transaction would roll back the first succeeded operations
and throw and error.

Like Scott mentioned, I'm not sure transactions is the way to work around
this particular issue.

--
*Prageeth Silva*

[Matt Giuca (mgiuca)]

Sorry I didn't reply to this until now.

Transactions *are* *the* (only) way to work around issues where race conditions are causing things to go wrong where they would be fine if executed in sequence. Any time you see a pattern like this:
1. Check if X exists,
2. If not, create X and write it to the datastore.

you know you need transactions, because otherwise, the following situation can always arise:

1. Thread 1 checks if X exists -- it does not,
2. Thread 2 checks if X exists -- it does not,
3. Thread 1 creates a new object X, and writes it to the datastore,
4. Thread 2 creates a new object X, and writes it to the datastore.

This results in duplicate objects called X. No amount of clever code can prevent this from happening, because any two lines of code you write can have another thread come along and do something in the middle. Only transactions, being a database primitive, can prevent this.

> Isn't transactions used in cases such as u want multiple operations
> happening OR none at all.
> This means, that if u have five operations you want happening and and the
> third fails, the transaction would roll back the first succeeded operations
> and throw and error.

Yes, that's what a transaction does. But having multiple operations succeed or fail together is exactly the property that we are interested in here. The way to do the above operation correctly, in a transaction is:

1. Begin a transaction,
2. Check if X exists,
3. If not, create X and write it to the datastore.
4. Commit the transaction.

This transaction has two operations (check and create), and either both of them will succeed, or none of them will (and we will have a ConcurrentModificationException). Now the above sequence looks like this:

1. Thread 1 begins a transaction,
2. Thread 1 checks if X exists -- it does not,
3. Thread 2 begins a transaction,
4. Thread 2 checks if X exists -- it does not,
5. Thread 1 creates a new object X, and writes it to the datastore,
6. Thread 2 creates a new object X, and writes it to the datastore,
7. Thread 1 commits the transaction, and succeeds,
8. Thread 2 commits the transaction and fails with a ConcurrentModificationException.

Now this was bad for thread 2 (the user will see an error and may have to refresh). But importantly, the database integrity is intact. We usually don't want to expose the user to that error, so the last thing we can do in the transaction is catch that exception, and then retry the whole transaction. We repeat it a certain number of times (say, 3), and this is usually enough. In that case, after step 8 there, we will have Thread 2 go and do the whole process again, but this time succeed. If we retry 3 times and still can't do it, we show the user the exception and accept that this will happen very rarely. It is OK for the user to see the occasional Internal Server Error, but it is not OK to put the datastore into an inconsistent state.

So, that's what we *should* be doing basically everywhere in the program. Unfortunately, as Scott found out, App Engine transactions are quite limited and often don't work. I've been (coincidentally, on another project) learning a lot about App Engine transactions this weekend...

[Prageeth Silva (prageethsilva)]

OK Thanks for the long explanation Matt. :)

I think it makes more sense now.

[Scott Ritchie (sritchie)]

I tried using transactions back when we had owned relationships just
on the use class and they didn't work.

On 30/05/2011, at 5:29 PM, Matt Giuca <email address hidden> wrote:

> Sorry I didn't reply to this until now.
>
> Transactions *are* *the* (only) way to work around issues where race
> conditions are causing things to go wrong where they would be fine
> if executed in sequence. Any time you see a pattern like this:
> 1. Check if X exists,
> 2. If not, create X and write it to the datastore.
>
> you know you need transactions, because otherwise, the following
> situation can always arise:
>
> 1. Thread 1 checks if X exists -- it does not,
> 2. Thread 2 checks if X exists -- it does not,
> 3. Thread 1 creates a new object X, and writes it to the datastore,
> 4. Thread 2 creates a new object X, and writes it to the datastore.
>
> This results in duplicate objects called X. No amount of clever code
> can
> prevent this from happening, because any two lines of code you write
> can
> have another thread come along and do something in the middle. Only
> transactions, being a database primitive, can prevent this.
>
>> Isn't transactions used in cases such as u want multiple operations
>> happening OR none at all.
>> This means, that if u have five operations you want happening and
>> and the
>> third fails, the transaction would roll back the first succeeded
>> operations
>> and throw and error.
>
> Yes, that's what a transaction does. But having multiple operations
> succeed or fail together is exactly the property that we are
> interested
> in here. The way to do the above operation correctly, in a transaction
> is:
>
> 1. Begin a transaction,
> 2. Check if X exists,
> 3. If not, create X and write it to the datastore.
> 4. Commit the transaction.
>
> This transaction has two operations (check and create), and either
> both
> of them will succeed, or none of them will (and we will have a
> ConcurrentModificationException). Now the above sequence looks like
> this:
>
> 1. Thread 1 begins a transaction,
> 2. Thread 1 checks if X exists -- it does not,
> 3. Thread 2 begins a transaction,
> 4. Thread 2 checks if X exists -- it does not,
> 5. Thread 1 creates a new object X, and writes it to the datastore,
> 6. Thread 2 creates a new object X, and writes it to the datastore,
> 7. Thread 1 commits the transaction, and succeeds,
> 8. Thread 2 commits the transaction and fails with a
> ConcurrentModificationException.
>
> Now this was bad for thread 2 (the user will see an error and may have
> to refresh). But importantly, the database integrity is intact. We
> usually don't want to expose the user to that error, so the last thing
> we can do in the transaction is catch that exception, and then retry
> the
> whole transaction. We repeat it a certain number of times (say, 3),
> and
> this is usually enough. In that case, after step 8 there, we will have
> Thread 2 go and do the whole process again, but this time succeed.
> If we
> retry 3 times and still can't do it, we show the user the exception
> and
> accept that this will happen very rarely. It is OK for the user to see
> the occasional Internal Server Error, ...

[Prageeth Silva (prageethsilva)]

> I tried using transactions back when we had owned relationships just
> on the use class and they didn't work.

I think that's what Matt was saying about entity groups.

--
*Prageeth Silva*

[Matt Giuca (mgiuca)]

Yep. I don't quite know how an owned relationship works. It may be an entity group, but that doesn't necessarily mean transactions would "just work". There is a long list of conditions you need to satisfy for transactions to work (e.g., queries must be ancestor queries).

[Scott Ritchie (sritchie)]

Ah ok;

Just to clarify entity groups;
In an owned relationship both objects are in the same entity group. Thats
why you can only have very simple entity relations as owned relationships.
I wasn't aware of the other conditions so maybe thats why.
But in that regard, why isnt this transactional nature defualt? Surely you
would *always* want lookups to the datastore to be in transactions to
prevent
the cases we're having to code around here. Thats what i dont understand
about the way GAE works.

On Mon, May 30, 2011 at 8:19 PM, Matt Giuca <email address hidden>wrote:

> Yep. I don't quite know how an owned relationship works. It may be an
> entity group, but that doesn't necessarily mean transactions would "just
> work". There is a long list of conditions you need to satisfy for
> transactions to work (e.g., queries must be ancestor queries).
>
> --
> You received this bug notification because you are subscribed to MUGLE.
> https://bugs.launchpad.net/bugs/788590
>
> Title:
> Lots of UserGameProfiles created simultaneously
>
> Status in Mars Programming Language:
> Invalid
> Status in Melbourne University Game-based Learning Environment:
> Triaged
>
> Bug description:
> For every "real" user game profile created, there are about 10
> completely blank ones, without even a user.
>

[Matt Giuca (mgiuca)]

> Just to clarify entity groups;
> In an owned relationship both objects are in the same entity group.  Thats
> why you can only have very simple entity relations as owned relationships.

OK that makes sense. I'm used to the Python datastore API (which I've
used in more detail). It's lower level than JDO and doesn't have a
concept of "owned relationships", but objects can have "parents",
which places an object, its parents, and its siblings, in the same
entity group, and that lets you do transactions.

Do you remember the exact reason why owned relationships didn't work
out for us? We possibly should have stuck with them. But I suspect it
will be too hard to refactor now.

> But in that regard, why isnt this transactional nature defualt? Surely you
> would *always* want lookups to the datastore to be in transactions to
> prevent
> the cases we're having to code around here. Thats what i dont understand
> about the way GAE works.

Well transactions can't be the "default" in any database system
because you haven't specified what operations need to be done
together. Transactions have a high overhead, and you wouldn't want
them on all the time. For example, consider a request in an email
program that sets the "mark as read" flag to "true" for every row of a
table that matches a certain query, and it includes 1000 rows. There's
no need for that to be done atomically. If it marks 500 emails as
"read" and someone simultaneously queries the state of the inbox, it
won't matter if you show half the emails as read and half as unread,
then a few seconds later you finish showing them all as read. That
doesn't need to be in a transaction. But if you did put it in a
transaction, you would be blocking all mail operations until the 1000
emails are all marked as read.

Besides, it couldn't automatically apply a blanket rule such as
"handle all HTTP requests in a full transaction", because of the
conditions above -- it would restrict your HTTP request to only deal
with a single entity group (which for our program at the moment would
mean a single entity).

[Scott Ritchie (sritchie)]

The reason owned relationships didn't work was it kept throwing an
error along the lines of "entity relations can't have more than one
ancestor". I did initially refactor it to what I thought was a tree
like structure - each entity had only one parent (but multiple
children) but it still complained so I made everything unowned. It
seemed like there could only be one owned parent and one owned child
for each entity from memory of further reading I did at the time

On 31/05/2011, at 10:59 AM, Matt Giuca <email address hidden>
wrote:

>> Just to clarify entity groups;
>> In an owned relationship both objects are in the same entity
>> group. Thats
>> why you can only have very simple entity relations as owned
>> relationships.
>
> OK that makes sense. I'm used to the Python datastore API (which I've
> used in more detail). It's lower level than JDO and doesn't have a
> concept of "owned relationships", but objects can have "parents",
> which places an object, its parents, and its siblings, in the same
> entity group, and that lets you do transactions.
>
> Do you remember the exact reason why owned relationships didn't work
> out for us? We possibly should have stuck with them. But I suspect it
> will be too hard to refactor now.
>
>> But in that regard, why isnt this transactional nature defualt?
>> Surely you
>> would *always* want lookups to the datastore to be in transactions to
>> prevent
>> the cases we're having to code around here. Thats what i dont
>> understand
>> about the way GAE works.
>
> Well transactions can't be the "default" in any database system
> because you haven't specified what operations need to be done
> together. Transactions have a high overhead, and you wouldn't want
> them on all the time. For example, consider a request in an email
> program that sets the "mark as read" flag to "true" for every row of a
> table that matches a certain query, and it includes 1000 rows. There's
> no need for that to be done atomically. If it marks 500 emails as
> "read" and someone simultaneously queries the state of the inbox, it
> won't matter if you show half the emails as read and half as unread,
> then a few seconds later you finish showing them all as read. That
> doesn't need to be in a transaction. But if you did put it in a
> transaction, you would be blocking all mail operations until the 1000
> emails are all marked as read.
>
> Besides, it couldn't automatically apply a blanket rule such as
> "handle all HTTP requests in a full transaction", because of the
> conditions above -- it would restrict your HTTP request to only deal
> with a single entity group (which for our program at the moment would
> mean a single entity).
>
> --
> You received this bug notification because you are subscribed to
> MUGLE.
> https://bugs.launchpad.net/bugs/788590
>
> Title:
> Lots of UserGameProfiles created simultaneously
>
> Status in Mars Programming Language:
> Invalid
> Status in Melbourne University Game-based Learning Environment:
> Triaged
>
> Bug description:
> For every "real" user game profile created, there are about 10
> completely blank ones, without even a user.

[Matt Giuca (mgiuca)]

What is an "owned child"? That sounds contradictory (or rather, the
idea that a parent owns its child is OK, but the idea that the child
owns its parent is not). I think a two-way one-to-many relationship
can be modelled but it needs to be the case that the "one" is the
parent of the "many". This means that the "one" would have a set of
keys linking it back to the "many", but they would just be keys, not
objects in the way that the child would refer to the parent.

I don't know if JDO was a mistake. It seems like an annoying
abstraction -- it's a nice way of thinking about things, but then they
break and you need to know about the details under the abstraction. An
abstraction where you need to know the details underneath is worse
than no abstraction at all. It looks like Java App Engine gives you a
much lower level view of the datastore, but it is too low! The Python
App Engine gives you a middle-level view (somewhere in between the
Java low-level view and the JDO view), which seems about right. So I
very much prefer the Python App Engine.

[Scott Ritchie (sritchie)]

Right, I mean the parent can only "own" one child, not multiple, and
that an entity can only be "owned" by one other.

On 31/05/2011, at 1:05 PM, Matt Giuca <email address hidden> wrote:

> What is an "owned child"? That sounds contradictory (or rather, the
> idea that a parent owns its child is OK, but the idea that the child
> owns its parent is not). I think a two-way one-to-many relationship
> can be modelled but it needs to be the case that the "one" is the
> parent of the "many". This means that the "one" would have a set of
> keys linking it back to the "many", but they would just be keys, not
> objects in the way that the child would refer to the parent.
>
> I don't know if JDO was a mistake. It seems like an annoying
> abstraction -- it's a nice way of thinking about things, but then they
> break and you need to know about the details under the abstraction. An
> abstraction where you need to know the details underneath is worse
> than no abstraction at all. It looks like Java App Engine gives you a
> much lower level view of the datastore, but it is too low! The Python
> App Engine gives you a middle-level view (somewhere in between the
> Java low-level view and the JDO view), which seems about right. So I
> very much prefer the Python App Engine.
>
> --
> You received this bug notification because you are subscribed to
> MUGLE.
> https://bugs.launchpad.net/bugs/788590
>
> Title:
> Lots of UserGameProfiles created simultaneously
>
> Status in Mars Programming Language:
> Invalid
> Status in Melbourne University Game-based Learning Environment:
> Triaged
>
> Bug description:
> For every "real" user game profile created, there are about 10
> completely blank ones, without even a user.

[Matt Giuca (mgiuca)]

OK, well there's something very wrong if a parent can only own one
child (that is; for each entity P, there can only be one entity Q such
that Q.parent = P). You certainly should be able to have multiple Qs
such that Q.parent = P, and I have done so in the Python App Engine.

[Scott Ritchie (sritchie)]

Right, which is why i abandoned the idea together. Unless i made one of the
relationships incorrectly at the time the relationship tree i had was:

User
 -> UserGameProfile
     -> KeyValuePair
     -> UserAchievement

DevTeam
 -> Game
     -> Achievement
     -> GameVersion
         -> GameFile
         -> PromotedGame

And it was throwing errors, As far as im aware each of the relations only
had one parent which owned it. (The other relationships were set up the way
we have them now)

On Tue, May 31, 2011 at 1:58 PM, Matt Giuca <email address hidden>wrote:

> OK, well there's something very wrong if a parent can only own one
> child (that is; for each entity P, there can only be one entity Q such
> that Q.parent = P). You certainly should be able to have multiple Qs
> such that Q.parent = P, and I have done so in the Python App Engine.
>
> --
> You received this bug notification because you are subscribed to MUGLE.
> https://bugs.launchpad.net/bugs/788590
>
> Title:
> Lots of UserGameProfiles created simultaneously
>
> Status in Mars Programming Language:
> Invalid
> Status in Melbourne University Game-based Learning Environment:
> Triaged
>
> Bug description:
> For every "real" user game profile created, there are about 10
> completely blank ones, without even a user.
>

[Matt Giuca (mgiuca)]

That seems like a reasonable relationship tree (except I would
possibly make PromotedGame separate, as I don't consider it to be
"part of" the game). I wonder what went wrong?

[Scott Ritchie (sritchie)]

I'm not sure, it had problems with both UserGameProfile and GameVersion, so
i can only conclude it was having multiple children that caused the errors.
Everything i read from what others had posted indicated that Owned
relationships were very limited and recommended using unowned for
everything.

On Tue, May 31, 2011 at 3:16 PM, Matt Giuca <email address hidden>wrote:

> That seems like a reasonable relationship tree (except I would
> possibly make PromotedGame separate, as I don't consider it to be
> "part of" the game). I wonder what went wrong?
>
> --
> You received this bug notification because you are subscribed to MUGLE.
> https://bugs.launchpad.net/bugs/788590
>
> Title:
> Lots of UserGameProfiles created simultaneously
>
> Status in Mars Programming Language:
> Invalid
> Status in Melbourne University Game-based Learning Environment:
> Triaged
>
> Bug description:
> For every "real" user game profile created, there are about 10
> completely blank ones, without even a user.
>