https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1719N1105
SQL time: 5980 ms
Non-sql time: 59441 ms
Total time: 65421 ms
Statement Count: 1005
Quick analysis shows death by a thousand cuts.
The mail sending is certainly an issue but the big problem is lots of itty bitty tasks for the DB.
989. 13977 2ms SQL-launchpad-main-master SELECT %s FROM Person JOIN TeamParticipation ON TeamParticipation.person = Person.id WHERE Person.id = %s AND TeamParticipation.team = %s AND NOT (TeamParticipation.person = %s)
990. 13981 12ms SQL-launchpad-main-master UPDATE TeamMembership SET date_reviewed=%s, last_change_comment=%s, status=%s, date_joined=%s, last_changed_by=%s, date_last_changed=%s, reviewer_comment=%s, reviewed_by=%s WHERE TeamMembership.id = %s
991. 13998 7ms SQL-launchpad-main-master INSERT INTO TeamParticipation (person, team) SELECT 468328, superteam.team FROM TeamParticipation superteam WHERE superteam.person = 3390563 AND NOT EXISTS ( SELECT 1 FROM TeamParticipation WHERE person = 468328 AND team = superteam.team )
992. 14009 4ms SQL-launchpad-main-master SELECT * FROM ((SELECT Person.account, Person.creation_comment, Person.creation_rationale, Person.datecreated, Person.defaultmembershipperiod, Person.defaultrenewalperiod, Person.displayname, Person.hide_email_addresses, Person.homepage_content, Person.icon, Person.id, Person.logo, Person.mailing_list_auto_subscribe_policy, Person.merged, Person.mugshot, Person.name, Person.personal_standing, Person.personal_standing_reason, Person.registrant, Person.renewal_policy, Person.subscriptionpolicy, Person.teamdescription, Person.teamowner, Person.verbose_bugnotifications, Person.visibility FROM Person, TeamMembership WHERE TeamMembership.team = 3390563 AND TeamMembership.status = 3 AND TeamMembership.person = Person.id AND (1=1)) UNION (SELECT Person.account, Person.creation_comment, Person.creation_rationale, Person.datecreated, Person.defaultmembershipperiod, Person.defaultrenewalperiod, Person.displayname, Person.hide_email_addresses, Person.homepage_content, Person.icon, Person.id, Person.logo, Person.mailing_list_auto_subscribe_policy, Person.merged, Person.mugshot, Person.name, Person.personal_standing, Person.personal_standing_reason, Person.registrant, Person.renewal_policy, Person.subscriptionpolicy, Person.teamdescription, Person.teamowner, Person.verbose_bugnotifications, Person.visibility FROM Person WHERE id = 1015)) AS "_a684" ORDER BY person_sort_key(displayname, name)
993. 14015 2ms SQL-launchpad-main-master SELECT EmailAddress.account, EmailAddress.email, EmailAddress.id, EmailAddress.person, EmailAddress.status FROM EmailAddress WHERE EmailAddress.person = %s AND EmailAddress.status = %s ORDER BY EmailAddress.email
994. 14019 3ms SQL-launchpad-main-master SELECT EmailAddress.account, EmailAddress.email, EmailAddress.id, EmailAddress.person, EmailAddress.status FROM EmailAddress WHERE LOWER(EmailAddress.email) = %s ORDER BY EmailAddress.email
995. 14040 0ms sendmail yltsrc approved by sladen
996. 14041 1ms SQL-launchpad-main-master SELECT EmailAddress.account, EmailAddress.email, EmailAddress.id, EmailAddress.person, EmailAddress.status FROM EmailAddress WHERE LOWER(EmailAddress.email) = %s ORDER BY EmailAddress.email
997. 14063 0ms sendmail yltsrc approved by sladen
998. 14064 2ms SQL-launchpad-main-master SELECT EmailAddress.account, EmailAddress.email, EmailAddress.id, EmailAddress.person, EmailAddress.status FROM EmailAddress WHERE LOWER(EmailAddress.email) = %s ORDER BY EmailAddress.email
999. 14084 0ms sendmail yltsrc approved by sladen
1000. 14085 1ms SQL-launchpad-main-master SELECT EmailAddress.account, EmailAddress.email, EmailAddress.id, EmailAddress.person, EmailAddress.status FROM EmailAddress WHERE LOWER(EmailAddress.email) = %s ORDER BY EmailAddress.email
I agree that it may need batching. However it can probably actually needs to record intent rather than doing the teamparticipation changes directly: even a single team membership change could in principle require hundreds of sql updates to update the cache, and that may be better done in a queue outside the web request.