Sharing code in private branches between teams

[Nigel Pugh]

We need to be able to share and reference code between teams internally within our company. All our code, open sourced or otherwise, should be accessible by default to any member of the company.

The user story goes like this:
As an employee of our company I can access, branch and submit fixes to any code that the company owns by default once I am a member of the company's launchpad private team, but should not be receiving any email as a result of this access by default. This is so that it is easy to share my code with other teams in the company and improve it through their feedback and scrutiny and I can learn from looking at the code generated in other teams within the company.

As this has been under discussion previously there are some existing bugs relating to possible solutions and workarounds for this:

Barry contributed this list: (Thank you!)

https://bugs.edge.launchpad.net/launchpad-registry/+bug/297833
This one almost definitely addresses my recent attempts to join ~canonical to ~private-launchpad-strategy

https://bugs.edge.launchpad.net/launchpad-registry/+bug/297942
Probably not related

https://bugs.edge.launchpad.net/launchpad-registry/+bug/405277
The uber-issue

https://bugs.edge.launchpad.net/launchpad-registry/+bug/411686
Similar "constraint not satisfied" error

[Curtis Hovey (sinzui)]

This issue was discussed at the 2009-09 team leads meeting. This is very hard, it is not a registry problem, it is a launchpad problem. This requires a rethink of launchpad ACL. It was suggested that the Launchpad architect do a spike solution during the 3,1 to evaluate problems. If the problems seem solvable, ACLs could be a focus of the 3.2 series.

If this issue is just about code, this bug should be move to launchpad code where a non-comprehensive solution can be found.

[Steve Alexander (stevea)]

Canonical uses Launchpad to manage both open source and proprietary code. Right now, there's a serious problem that the proprietary code is hard to get access to by people who work at Canonical.

We should learn from Google, who have a single central code repository for all of their code, so any Google employee can browse the code, subscribe to changes, and run (for example) linting or statistical scripts over the whole codebase.

We've said that sharing and reusing code is strategically important for Canonical. The current situation with using Launchpad for proprietary code works against this.

Perhaps a workaround would be to keep outside of Launchpad (maybe on a wiki page) a list of Canonical internal projects. Then, the mainline of each of these projects is automatically checked out, and kept up to date, on a single server somewhere. Anyone who works at Canonical would be able to rsync this entire codebase, or use bzr on the codebase. Something like this would be a good start at a workaround for this problem.

[Christian Reis (kiko)]

Well, actually, I think the immediate problem is restricted to code, Curtis. So just solving that specific use case is probably not something completely out of scope, or requiring ACL redesign.

[Curtis Hovey (sinzui)]

If this is a code problem, then this bug belongs to the code team to solve. I do not think this is a code team issue because a private team can be subscribed to every canonical project and it will not get unwanted emails. So this bug exists for other reasons, most likely because setting this on every project is cumbersome,

[Neil Levine (levine)]

On Fri, Nov 20, 2009 at 11:11:22PM -0000, Curtis Hovey said:
> If this is a code problem, then this bug belongs to the code team to
> solve. I do not think this is a code team issue because a private team
> can be subscribed to every canonical project and it will not get
> unwanted emails. So this bug exists for other reasons, most likely
> because setting this on every project is cumbersome,

By "unwanted emails", do you mean I can be a member of a private team,
that team can be then be subscribed to all internal Canonical projects
and I will not receive notification of every bug report etc, but will
be able to view code?

Neil

[Curtis Hovey (sinzui)]

On Mon, 2009-11-23 at 12:29 +0000, Neil Levine wrote:
> On Fri, Nov 20, 2009 at 11:11:22PM -0000, Curtis Hovey said:
> > If this is a code problem, then this bug belongs to the code team to
> > solve. I do not think this is a code team issue because a private team
> > can be subscribed to every canonical project and it will not get
> > unwanted emails. So this bug exists for other reasons, most likely
> > because setting this on every project is cumbersome,
>
> By "unwanted emails", do you mean I can be a member of a private team,
> that team can be then be subscribed to all internal Canonical projects
> and I will not receive notification of every bug report etc, but will
> be able to view code?

No, I mean that as a member of a private team, that has "public"
permissions set for EVERY individual Canonical project I will not
receive email. Policy can be set on a project group so that a team has
access to all the branches that belong to the child projects.

Branch visibility has no connection to bugs. It is a wildly different
implementation and no code or data is shared between bugs and branches.

--
__Curtis C. Hovey_________
http://launchpad.net/

[Curtis Hovey (sinzui)]

It is tempting to move this bug to launchpad-code since the discussion is just about that application's features. I added it instead because I still think there is something unsaid that is about the teams or projects.

[Neil Levine (levine)]

On Mon, Nov 23, 2009 at 01:46:50PM -0000, Curtis Hovey said:
>
> No, I mean that as a member of a private team, that has "public"
> permissions set for EVERY individual Canonical project I will not
> receive email. Policy can be set on a project group so that a team has
> access to all the branches that belong to the child projects.
>
> Branch visibility has no connection to bugs. It is a wildly different
> implementation and no code or data is shared between bugs and branches.

I am not sure I understand your email. Please assume I know nothing
about Launchpad internals (which is almost true!). I am just
interested in the following two user stories. Your email sounds like
at last the first one is possible now. Please can you confirm.

User story one: A new developer joins the company. He wants to be able
to view all proprietary code written by Canonical without receiving
any email related to bugs about that code. He just wants to be able to
check it out.

User story two: A developer who has access to Canonical proprietary
code (due to user story one) decides he wants to see what bugs have
been filed against a particular project because he thinks he has found
one. He does not want to receive all bug reports associated with the
project the code is from but does want to ensure that he doesn't file
a bug which has already been reported.

N

[Curtis Hovey (sinzui)]

On Mon, 2009-11-23 at 16:32 +0000, Neil Levine wrote:
> On Mon, Nov 23, 2009 at 01:46:50PM -0000, Curtis Hovey said:
> >
> > No, I mean that as a member of a private team, that has "public"
> > permissions set for EVERY individual Canonical project I will not
> > receive email. Policy can be set on a project group so that a team has
> > access to all the branches that belong to the child projects.
> >
> > Branch visibility has no connection to bugs. It is a wildly different
> > implementation and no code or data is shared between bugs and branches.
>
> I am not sure I understand your email. Please assume I know nothing
> about Launchpad internals (which is almost true!). I am just
> interested in the following two user stories. Your email sounds like
> at last the first one is possible now. Please can you confirm.
>
> User story one: A new developer joins the company. He wants to be able
> to view all proprietary code written by Canonical without receiving
> any email related to bugs about that code. He just wants to be able to
> check it out.

Add the user to the canonical team...assuming the canonical team has
been give public branch access to all canonical owned projects. I am
certain this qualification has not been met because branch visibility
has supported this feature for all of 2009.

Part of the problem I think is that I am the only launchpad team members
commenting on this issue, but the issues hear involve features created
by other teams. The registry is not directly involved in this issue

> User story two: A developer who has access to Canonical proprietary
> code (due to user story one) decides he wants to see what bugs have
> been filed against a particular project because he thinks he has found
> one. He does not want to receive all bug reports associated with the
> project the code is from but does want to ensure that he doesn't file
> a bug which has already been reported.

This is nigh impossible. There are two problems. The one is that bug
privacy is married to subscriptions--you must subscribe to email to get
access. There are hacks to get around this such as subscribing a team
with an email address that will never ever be checked. The second
problem is that private teams cannot be members of other teams, bug
private bugs require the team to be a member of the bug supervisors.

The bug privacy implementation must be reinvented to support this use
case. While I do want to allow private teams to join other teams, that
is not the real issue here.

And to repeat, I think the problem here is that this issue involves the
code team, not the registry team. I will add the launchpad-bugs to this
bug.

--
__Curtis C. Hovey_________
http://launchpad.net/

[Tim Penhey (thumper)]

Branch visibility is controlled solely through branch subscriptions.

You can subscribe to a branch and not get any emails.

Who gets subscribed to private branches is controlled by the branch visibility policy of the project.

The simple solution:
 * change all the branch visibility policies to use the canonical team for the privacy restrictions.
(possibly go through all the branches on all the private projects and subscribe the canonical team - SQL would be fastest).

[Michael Hudson-Doyle (mwhudson)]

Well, there *are* projects on launchpad that are proprietary and not owned by canonical. But if the policy is that all canonical-owned code is visible by all canonical employees (something that wasn't the case in the past, I think?) then setting the canonical team as the privacy team in the branch visibility policy will fix this for all new branches going forward. For existing branches, the canonical team will have to be subscribed, which can probably also be done with some sql once if the list of "canonical" projects can be determined somehow.

[Neil Levine (levine)]

On Tue, Nov 24, 2009 at 12:57:59AM -0000, Michael Hudson said:
> Well, there *are* projects on launchpad that are proprietary and not
> owned by canonical. But if the policy is that all canonical-owned code
> is visible by all canonical employees (something that wasn't the case in
> the past, I think?) then setting the canonical team as the privacy team
> in the branch visibility policy will fix this for all new branches going
> forward. For existing branches, the canonical team will have to be
> subscribed, which can probably also be done with some sql once if the
> list of "canonical" projects can be determined somehow.

So, let me see if I can summarise:

- A team can be created (canonical-global) which is automatically subscribed
  to all projects which are identified as being Canonical-owned.
- Members of this team will not receive email related to any activity
  on the project.
- All members of this team can view the branches for all
  Canonical-owned projects to which it is subscribed.

If this is the case, then it sounds like we need to:

- Identify all Canonical-owned projects (presumably anything with the
  word Canonical in it for a start).
- Create the canonical-global team if it doesn't exist and subscribe
  it as a "privacy team" as part of the "branch visibility project".

Please confirm. If so, then it sounds like, unless I am mistaken, an
action for the OSAs to perform.

Neil

[Michael Hudson-Doyle (mwhudson)]

Neil Levine wrote:
> On Tue, Nov 24, 2009 at 12:57:59AM -0000, Michael Hudson said:
>> Well, there *are* projects on launchpad that are proprietary and not
>> owned by canonical. But if the policy is that all canonical-owned code
>> is visible by all canonical employees (something that wasn't the case in
>> the past, I think?) then setting the canonical team as the privacy team
>> in the branch visibility policy will fix this for all new branches going
>> forward. For existing branches, the canonical team will have to be
>> subscribed, which can probably also be done with some sql once if the
>> list of "canonical" projects can be determined somehow.
>
> So, let me see if I can summarise:
>
> - A team can be created (canonical-global) which is automatically subscribed
> to all projects which are identified as being Canonical-owned.

Well, two things: there is already a team ~canonical, why create another
for this?

And to be pedantic, it is not "subscribed to all projects which are
identified". Apologies if what follows is patronizing, but it seems
wise to be correct here.

Central to the private branches feature on Launchpad is the "branch
visibility policy". This is attached to a project, and specifies a list
of policies, a number of which can apply to teams and one of which is an
"everyone else' policy and applies to anyone who's not a member of a
team covered by another policy.

Each policy can say:

 - public only -- people can create branches, but they're always public.
   Most projects in Launchpad have the simple policy "everyone: public
   only".

 - forbidden -- people cannot create branches.

 - private by default -- people can create branches, they're private by
   default, they can disclose them one-by-one.

 - private only -- people can create branches, they're private by
   default, they cannot disclose them.

(you'll notice the lack of the potentially useful "public by default"
but that's a separate issue)

What I'm suggesting we do is that for all canonical owned proprietary
projects, we set the policy to be:

 - ~canonical: private by default or private only as appropriate
 - everyone else: probably forbidden

When a private branch is created, the team whose policy allowed the
creation of the branch is subscribed, so all members of that team can
see the branch.

> - Members of this team will not receive email related to any activity
> on the project.

Correct, the "privacy subscriber" does not get any mail about the branch.

> - All members of this team can view the branches for all
> Canonical-owned projects to which it is subscribed.

If the above made sense, it should be clear that they will be able to
view all the branches for all the Canonical proprietary projects that
are created *after the policy is changed*. Existing branches also need
to be fixed up somehow.

> If this is the case, then it sounds like we need to:
>
> - Identify all Canonical-owned projects (presumably anything with the
> word Canonical in it for a start).

That's a start, but probably not a very good one... projects with a
non-trivial branch visibility policy whose team's owner is a member of
~canonical sounds better.

> - Create th...

[Francis J. Lacoste (flacoste)]

The only problem with ~canonical is that it's a private membership team and it seems that the constraint allowing these teams to subscribe to branches and be part of the visibility policy has not been lifted.

[Curtis Hovey (sinzui)]

On Thu, 2009-11-26 at 18:25 +0000, Francis J. Lacoste wrote:
> The only problem with ~canonical is that it's a private membership team
> and it seems that the constraint allowing these teams to subscribe to
> branches and be part of the visibility policy has not been lifted.

Brad and I tested changing the team to a private team. We certainly can
do that. There are no super-team/sub-team conflicts. If we change it via
the UI the team will be renamed ~private-canonical. A LOSA is needed to
rename it back to ~canonical. The other way to address this is to change
the team's visibility by changing the database.

--
__Curtis C. Hovey_________
http://launchpad.net/

[Francis J. Lacoste (flacoste)]

Curtis, ok, and a private team can be linked to branch subscriptions and branch policy?

[Curtis Hovey (sinzui)]

Yes, a private team is *designed* to work with branch subscriptions and branch policy. Brad and I did test this on staging earlier this week to verify there were no regressions.

[Francis J. Lacoste (flacoste)]

Cool, then Michael and Tim can you coordinate with the LOSAs, to make this happen. They'll need help with the various SQL queries needed to turn the team in a private one, find the canonical projects, change the policy and migrate the existing branches.

We can probably consider this bug Fix released once that's done.

[Tom Haddon (mthaddon)]

Curtis, I wasn't able to do this (branch subscriptions and branch policy for a private team) in production last week. Is there something that's waiting to be rolled out that will fix this?

[Curtis Hovey (sinzui)]

I was able to do this with ~canonical after making it a private team last week on staging. In production, ~canonical is a private *membership* team so it cannot subscribe to branches or have a policy.

You can change the team to Private using 'Change details' link, but that will also rename the team to ~private-canonical. We can change the team's visibility to private using SQL to preserve the team name.
    UPDATE Person SET visibility = 30 WHERE name = 'canonical';

[Tom Haddon (mthaddon)]

Are there any other implications to setting ~canonical as a private team. Since this team is used for so many things I'm hesitant to make any changes to it without being very sure about them. If you can confirm this won't affect anything else, I'll be happy to run this SQL.

[Curtis Hovey (sinzui)]

There were no consequences when I used the UI to change it on staging. If there was a membership conflict the change would have been denied. Once the change is made, ~canonical will not be able to join teams. Since it has not, I do not think this is an issue.

[Michael Hudson-Doyle (mwhudson)]

I cooked up this query to try to find the "canonical" projects: https://pastebin.canonical.com/25151/

It's all projects and project groups that have a branchvisibilitypolicy permitting private branches where the owner of the team that owns the project group is a member of ~canonical. What you really want to do is, starting with the owner of the project, keep following teamowner until you get to a non team, then check that person, but that's either impossible or at least really hard in SQL. This query finds most projects I expected to be found, but has at least one -- zope3 -- I don't think should be included in this work. Can someone else read over it?

[Robert Collins (lifeless)]

This will be doable once we enable the new policy rules for branches - just for each company project grant the company team access to proprietary branches.

[Robert Collins (lifeless)]

(Oh, and we can assist setting that up in bulk, if needed)

[Curtis Hovey (sinzui)]

William, you can close this bug when we enter the Sharing beta.