Ubuntu
charm package

Charm push crashes when pushing 'big' charm.

Bug #1592822 reported by Tengu CI on 2016-06-15

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	charm (Ubuntu)	Confirmed	High	Uros Jovanovic

Bug Description

running `charm push` for a very large charm causes a crash. Removing the largest files from the charm fixes the issue. If large charms are not supported, it might be better to throw a helpful error message.

```
ERROR cannot post archive: cannot unmarshal error response "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">\n<html><head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n<title>ERROR: The requested URL could not be retrieved</title>\n<style type=\"text/css\"></style>\n</head><body id=ERR_ZERO_SIZE_OBJECT>\n<div id=\"titles\">\n<h1>ERROR</h1>\n<h2>The requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div id=\"content\">\n<p>The following error was encountered while trying to retrieve the URL: <a href=\"http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?\">http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?</a></p>\n\n<blockquote id=\"error\">\n<p><b>Zero Sized Reply</b></p>\n</blockquote>\n\n<p>Squid did not receive any data for this request.</p>\n\n<p>Your cache administrator is <a href=\"mailto:webmaster?subjec ... [3528 bytes omitted]": invalid character '<' looking for beginning of value
```

Putting the charm in attachment causes a launchpad crash. Please let me know if I can upload the charm somewhere else.

Uros Jovanovic (uros-jovanovic) on 2016-06-16

Changed in juju:
importance:	Undecided → High
assignee:	nobody → Uros Jovanovic (uros-jovanovic)

Revision history for this message

Uros Jovanovic (uros-jovanovic) wrote on 2016-06-16:

The new charm publishing is in the charm package.

Changed in juju:
assignee:	Uros Jovanovic (uros-jovanovic) → nobody
affects:	juju → charm (Ubuntu)
Changed in charm (Ubuntu):
assignee:	nobody → Uros Jovanovic (uros-jovanovic)

Revision history for this message

Jay R. Wren (evarlast) wrote on 2016-06-20:

I replied on email list:

Hello Merlijn,

I can replicate the problem and I can work around it by using a faster internet connection.

At some point, tcp connections have to time out. I can only replicate the issue when that timeout is reached. If you have the means to relocate to a faster internet connection temporarily for pushing to charmstore, please do. You might also try recompressing any items in the charm using a higher compression level. xz -9 instead of gzip -3 or whatever things may be using now.

We are aware this is a poor longterm solution. We are investigating better solutions for uploads. As you've mentioned, resources will also help the situation.

I am sorry that I do not have a better solution.

-------

The timeout is apache setting `Timeout` which defaults to 300. It is likely squid and/or haproxy also have a similar timeout. We cannot support arbitrary large file uploads at arbitrary low speeds without exhausting TCP connections. It seems we may wish to state 200MB as a practical charm size limit.

Changed in charm (Ubuntu):
status:	New → Confirmed

Revision history for this message

John A Meinel (jameinel) wrote on 2016-07-10:

Is this a case that we are successfully sending data to squid at a reasonable but slow rate, such that it takes more than 5 minutes to upload the whole content and squid doesn't send anything to Apache until all of it has been transferred.
It sounds like we'll have the same problem with resources. While it might allow us to break up a large charm into a few smaller resources, AIUI some of the charms have a single tarball that is more than 1GB in size. Arbitrarily limiting the size is going to just cause people to work around the system (self hosting, etc) Which breaks many of the benefits of having resources (demure transfers, offline mirroring, etx)

Revision history for this message

Uros Jovanovic (uros-jovanovic) wrote on 2016-07-10: Re: [Bug 1592822] Re: Charm push crashes when pushing 'big' charm.

Download full text (3.2 KiB)

That's all true. We've increased the timeout for now, however, the proper
fix requires quite some effort. We need to move the GridFS blobs to Swift
and then storage management will become simpler (production wise). This
also covers multi-charm store syncing at the same time. I'm also going to
propose that we introduce quotas for charm store on user namespace. You can
ask for more space but that's how we can prevent from some CI to upload
gigabytes of data per day by mistake. Log analysis will help as well, but
that's also on ToDo ATM :-/

On Sun, Jul 10, 2016 at 9:28 AM, John A Meinel <email address hidden>
wrote:

> Is this a case that we are successfully sending data to squid at a
> reasonable but slow rate, such that it takes more than 5 minutes to upload
> the whole content and squid doesn't send anything to Apache until all of it
> has been transferred.
> It sounds like we'll have the same problem with resources. While it might
> allow us to break up a large charm into a few smaller resources, AIUI some
> of the charms have a single tarball that is more than 1GB in size.
> Arbitrarily limiting the size is going to just cause people to work around
> the system (self hosting, etc) Which breaks many of the benefits of having
> resources (demure transfers, offline mirroring, etx)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1592822
>
> Title:
> Charm push crashes when pushing 'big' charm.
>
> Status in charm package in Ubuntu:
> Confirmed
>
> Bug description:
> running `charm push` for a very large charm causes a crash. Removing
> the largest files from the charm fixes the issue. If large charms are
> not supported, it might be better to throw a helpful error message.
>
>
> ```
> ERROR cannot post archive: cannot unmarshal error response "<!DOCTYPE
> html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"
> http://www.w3.org/TR/html4/strict.dtd\">\n<html><head>\n<meta
> http-equiv=\"Content-Type\" content=\"text/html;
> charset=utf-8\">\n<title>ERROR: The requested URL could not be
> retrieved</title>\n<style type=\"text/css\"></style>\n</head><body
> id=ERR_ZERO_SIZE_OBJECT>\n<div id=\"titles\">\n<h1>ERROR</h1>\n<h2>The
> requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div
> id=\"content\">\n<p>The following error was encountered while trying to
> retrieve the URL: <a href=\"
> http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?\">
> http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?</a></p>\n\n<blockquote
> id=\"error\">\n<p><b>Zero Sized Reply</b></p>\n</blockquote>\n\n<p>Squid
> did not receive any data for this request.</p>\n\n<p>Your cache
> administrator is <a href=\"mailto:webmaster?subjec ... [3528 bytes
> omitted]": invalid character '<' looking for beginning of value
> ```
>
> Putting the charm in attachment causes a launchpad crash. Please let
> me know if I can upload the charm somewhere else.
>
> To manage notifi...

On Sun, Jul 10, 2016 at 9:28 AM, John A Meinel <john@arbash-meinel.com>
wrote:

> Is this a case that we are successfully sending data to squid at a
> reasonable but slow rate, such that it takes more than 5 minutes to upload
> the whole content and squid doesn't send anything to Apache until all of it
> has been transferred.
> It sounds like we'll have the same problem with resources. While it might
> allow us to break up a large charm into a few smaller resources, AIUI some
> of the charms have a single tarball that is more than 1GB in size.
> Arbitrarily limiting the size is going to just cause people to work around
> the system (self hosting, etc) Which breaks many of the benefits of having
> resources (demure transfers, offline mirroring, etx)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1592822
>
> Title:
>   Charm push crashes when pushing 'big' charm.
>
> Status in charm package in Ubuntu:
>   Confirmed
>
> Bug description:
>   running `charm push` for a very large charm causes a crash. Removing
>   the largest files from the charm fixes the issue. If large charms are
>   not supported, it might be better to throw a helpful error message.
>
>
>   ```
>   ERROR cannot post archive: cannot unmarshal error response "<!DOCTYPE
> html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"
> http://www.w3.org/TR/html4/strict.dtd\">\n<html><head>\n<meta
> http-equiv=\"Content-Type\" content=\"text/html;
> charset=utf-8\">\n<title>ERROR: The requested URL could not be
> retrieved</title>\n<style type=\"text/css\"></style>\n</head><body
> id=ERR_ZERO_SIZE_OBJECT>\n<div id=\"titles\">\n<h1>ERROR</h1>\n<h2>The
> requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div
> id=\"content\">\n<p>The following error was encountered while trying to
> retrieve the URL: <a href=\"
> http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?\">
> http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?</a></p>\n\n<blockquote
> id=\"error\">\n<p><b>Zero Sized Reply</b></p>\n</blockquote>\n\n<p>Squid
> did not receive any data for this request.</p>\n\n<p>Your cache
> administrator is <a href=\"mailto:webmaster?subjec ... [3528 bytes
> omitted]": invalid character '<' looking for beginning of value
>   ```
>
>   Putting the charm in attachment causes a launchpad crash. Please let
>   me know if I can upload the charm somewhere else.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/charm/+bug/1592822/+subscriptions
>

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntucharm package

Charm push crashes when pushing 'big' charm.

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
charm package