Ensure People CSV upload works for OSF

[Paul Everitt]

Nat and Jim are having trouble uploading users via CSV. The error message is obscure. In a way, this is a ticket to reclaim CSV Upload as something the core team knows about and can support.

[Paul Everitt (paul-agendaless)]

Ok, this discussion is moved into LP. Carlos is going to make a CSV sample file and work on it.

Nat and Jim, please send Carlos (in private, not in LP) the CSV files you were using.

[Carlos de la Guardia (cguardia)]

I created a plain text file and it imported fine. Then I tried with open office and it seems that it's very easy to save the file in a way that causes problems with the upload. I think we might need to make the parsing code stronger. It would be very helpful to see other 'real life' files to get to the bottom of this.

[Paul Everitt (paul-agendaless)]

Can you attach to this ticket a file that is known to work?

--Paul

On Jul 14, 2012, at 10:28 AM, Carlos de la Guardia wrote:

> I created a plain text file and it imported fine. Then I tried with open
> office and it seems that it's very easy to save the file in a way that
> causes problems with the upload. I think we might need to make the
> parsing code stronger. It would be very helpful to see other 'real life'
> files to get to the bottom of this.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1024643
>
> Title:
> Ensure People CSV upload works for OSF
>
> Status in KARL3:
> New
>
> Bug description:
> Nat and Jim are having trouble uploading users via CSV. The error
> message is obscure. In a way, this is a ticket to reclaim CSV Upload
> as something the core team knows about and can support.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/1024643/+subscriptions

[Carlos de la Guardia (cguardia)]

I included two example files that work well on my sandbox.

[Nat Katin-Borland (nborland)]

Carlos, have you gotten the examples I sent around to work for you? I was able to upload your example files, but I still can't upload the actual CSV for the users I need to upload. I'm still getting crashes. Is the problem the number of users in the list?

[Carlos de la Guardia (cguardia)]

I got the files now, thanks. It's not the number of users. I need to analyze the files and code a bit, but if you are in a real hurry I could clean up the csv and send it to you for import, then fix the problem in the code with a bit less urgency.

[Carlos de la Guardia (cguardia)]

I went ahead and sent you the file.

The problem is that apparently excel breaks up text columns containing more than one word into separate columns for each word. There should be an option on export to force quoting of all text fields and avoid this problem.

Not sure anything should be done at the code level now.

[Nat Katin-Borland (nborland)]

Hi Carlos,

Thanks for sending around a clean copy, but I still get it to work on Staging. I’m getting a General Error across all browsers (see traceback below). I guess the problem is that it doesn’t work with Excel yet?

-Nat

<!--
 Traceback (most recent call last):
 File "/srv/karlstaging/staging/1/eggs/pyramid-1.2.1-py2.6.egg/pyramid/tweens.py", line 17, in excview_tween
 response = handler(request)
 File "/srv/karlstaging/staging/1/eggs/pyramid-1.2.1-py2.6.egg/pyramid/router.py", line 153, in handle_request
 response = view_callable(context, request)
 File "/srv/karlstaging/staging/1/eggs/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 186, in _secured_view
 return view(context, request)
 File "/srv/karlstaging/staging/1/eggs/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 292, in rendered_view
 result = view(context, request)
 File "/srv/karlstaging/staging/1/eggs/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 390, in _class_view
 response = inst()
 File "/srv/karlstaging/staging/1/src/karl/karl/views/admin.py", line 588, in __call__
 reactivate, i)
 File "/srv/karlstaging/staging/1/src/karl/karl/views/admin.py", line 663, in _add_user_csv_row
 if merge_profile.security_state != 'inactive':
 AttributeError: 'Invitation' object has no attribute 'security_state'

 -->

[Carlos de la Guardia (cguardia)]

Hmm, that seems to be a different issue. I'll try it on karldev.

[Carlos de la Guardia (cguardia)]

Worked fine here:

https://karldev.gocept.com/master/vanilla/people/all/all/

Seems to be that there's something else that's wrong on staging?

[Nat Katin-Borland (nborland)]

Very strange. If there's something wrong on Staging, does that means we have the same issue on production? I don't want to break anything on production if I try to upload directly on Production without confirming on Staging first.

[Paul Everitt (paul-agendaless)]

I agree that, if Excel is splitting multi-word last names into two values, then that's not our fault. We shouldn't code around that.

--Paul

On Jul 16, 2012, at 8:57 PM, Carlos de la Guardia wrote:

> I went ahead and sent you the file.
>
> The problem is that apparently excel breaks up text columns containing
> more than one word into separate columns for each word. There should be
> an option on export to force quoting of all text fields and avoid this
> problem.
>
> Not sure anything should be done at the code level now.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1024643
>
> Title:
> Ensure People CSV upload works for OSF
>
> Status in KARL3:
> In Progress
>
> Bug description:
> Nat and Jim are having trouble uploading users via CSV. The error
> message is obscure. In a way, this is a ticket to reclaim CSV Upload
> as something the core team knows about and can support.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/1024643/+subscriptions

[Carlos de la Guardia (cguardia)]

And what do you think about his other issue. That one certainly doesn't have anything to do with the imported content. It's probably related to this other issue:

https://bugs.launchpad.net/karl3/+bug/965169

In the case of the failed upload on staging, it appears that one of the imported users is using an existing id and the system crashes when trying to get that 'security_state'.

I checked and security_state appears to have been added in an evolve state from more than a year ago. Is it possible  that somehow that step wasn't run on all old content?

Carlos de la Guardia

>________________________________
> From: Paul Everitt <email address hidden>
>To: <email address hidden>
>Sent: Tuesday, July 17, 2012 1:27 AM
>Subject: Re: [Bug 1024643] Ensure People CSV upload works for OSF
>
>I agree that, if Excel is splitting multi-word last names into two values, then that's not our fault. We shouldn't code around that.
>
>--Paul
>
>On Jul 16, 2012, at 8:57 PM, Carlos de la Guardia wrote:
>
>> I went ahead and sent you the file.
>>
>> The problem is that apparently excel breaks up text columns containing
>> more than one word into separate columns for each word. There should be
>> an option on export to force quoting of all text fields and avoid this
>> problem.
>>
>> Not sure anything should be done at the code level now.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1024643
>>
>> Title:
>>  Ensure People CSV upload works for OSF
>>
>> Status in KARL3:
>>  In Progress
>>
>> Bug description:
>>  Nat and Jim are having trouble uploading users via CSV. The error
>>  message is obscure. In a way, this is a ticket to reclaim CSV Upload
>>  as something the core team knows about and can support.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/karl3/+bug/1024643/+subscriptions
>
>--
>You received this bug notification because you are a bug assignee.
>https://bugs.launchpad.net/bugs/1024643
>
>Title:
>  Ensure People CSV upload works for OSF
>
>Status in KARL3:
>  In Progress
>
>Bug description:
>  Nat and Jim are having trouble uploading users via CSV. The error
>  message is obscure. In a way, this is a ticket to reclaim CSV Upload
>  as something the core team knows about and can support.
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/karl3/+bug/1024643/+subscriptions
>
>
>

[Nat Katin-Borland (nborland)]

I'm attempting to download Libre Office to see if I'm able to upload the spreadsheet in that format.

[Carlos de la Guardia (cguardia)]

If the file I sent you didn't work, I don't think this will help. There seems to be another problem on staging. Question: if one of the new users has the same login than a previous user, would you be able to tell which? Could there be more than one? If there's a quick way for you to remove any such users from the file, the import should work.

Carlos de la Guardia

>________________________________
> From: Nat Katin-Borland <email address hidden>
>To: <email address hidden>
>Sent: Tuesday, July 17, 2012 12:24 PM
>Subject: [Bug 1024643] Re: Ensure People CSV upload works for OSF
>
>I'm attempting to download Libre Office to see if I'm able to upload the
>spreadsheet in that format.
>
>--
>You received this bug notification because you are a bug assignee.
>https://bugs.launchpad.net/bugs/1024643
>
>Title:
>  Ensure People CSV upload works for OSF
>
>Status in KARL3:
>  In Progress
>
>Bug description:
>  Nat and Jim are having trouble uploading users via CSV. The error
>  message is obscure. In a way, this is a ticket to reclaim CSV Upload
>  as something the core team knows about and can support.
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/karl3/+bug/1024643/+subscriptions
>
>
>

[Nat Katin-Borland (nborland)]

OK, I won't try with Libre Office then. No one answered my question about Production and Staging. If something is broken on Staging, does that mean the same thing is broken on Production? I don't want to crash production by attempting to upload the users. If you think it will be fine, then let me know and I'll just try to upload on production.

I don't know of a quick way to determine existing users other than searching for each one manually. Paul, can you think of another way?

[Carlos de la Guardia (cguardia)]

I don't think production would crash on a a failed import. You would just get the error screen. I think getting the same error is very likely though, since staging has the same db.

>________________________________
> From: Nat Katin-Borland <email address hidden>
>To: <email address hidden>
>Sent: Tuesday, July 17, 2012 1:07 PM
>Subject: [Bug 1024643] Re: Ensure People CSV upload works for OSF
>
>OK, I won't try with Libre Office then.  No one answered my question
>about Production and Staging.  If something is broken on Staging, does
>that mean the same thing is broken on Production?  I don't want to crash
>production by attempting to upload the users.  If you think it will be
>fine, then let me know and I'll just try to upload on production.
>
>I don't know of a quick way to determine existing users other than
>searching for each one manually.  Paul, can you think of another way?
>
>--
>You received this bug notification because you are a bug assignee.
>https://bugs.launchpad.net/bugs/1024643
>
>Title:
>  Ensure People CSV upload works for OSF
>
>Status in KARL3:
>  In Progress
>
>Bug description:
>  Nat and Jim are having trouble uploading users via CSV. The error
>  message is obscure. In a way, this is a ticket to reclaim CSV Upload
>  as something the core team knows about and can support.
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/karl3/+bug/1024643/+subscriptions
>
>
>

[Carlos de la Guardia (cguardia)]

This bug as reported was fixed by exporting with string quoting. We identified the user id that was causing the other problem that came up during import tests. I will create another ticket to follow that up.