cloud-worker stuck after an invalid request

Bug #2059728 reported by Brian Murray
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Auto Package Testing
Fix Released
High
Skia

Bug Description

ubuntu@juju-7f2275-prod-proposed-migration-environment-2:~/autopkgtest-cloud/tools$ systemctl status --no-pager -l <email address hidden>
● <email address hidden> - autopkgtest worker lcy02-8
     Loaded: loaded (/<email address hidden>; enabled; vendor preset: enabled)
    Drop-In: /<email address hidden>
             └─ensure-adt-image.conf
     Active: active (running) since Thu 2024-03-28 10:11:17 UTC; 9h ago
   Main PID: 865222 (worker)
      Tasks: 1 (limit: 19154)
     Memory: 8.3M
     CGroup: /<email address hidden>
             └─865222 /usr/bin/python3 /home/ubuntu/autopkgtest-cloud/worker/worker --config /home/ubuntu/worker-lcy02.conf -v LXD_ARCH= -v LXD_REMOTE=lcy02 -v SECGROUP=autopkgtest-juju-7f227
<email address hidden>

Mar 28 10:22:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Acknowledging request cassiopee
                                                                                                                        {"all-proposed": 1, "requester": "vorlon", "submit-time": "2024-03-28 0
5:40:45", "triggers": ["cassiopee/1.0.9-4build2"], "uuid": "1888eea1-bdcb-4408-8040-7472f702ede1"}
Mar 28 10:22:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Waiting for and processing AMQP requests
Mar 28 10:22:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Received request for package cl-postmodern on noble/i386; params:
 {'triggers': ['ecl/21.2.1+ds-4.1ubuntu1'], 'all-proposed': '1', 'submit-time': '2024-03-28 09:50:20'}
Mar 28 10:22:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Running /home/ubuntu/autopkgtest/runner/autopkgtest --output-dir
/tmp/autopkgtest-work.o_0ihtgb/out --timeout-copy=6000 -a i386 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-production/setup-canonical.sh --apt-pocket=proposed --apt-upgrade
cl-postmodern --timeout-short=300 --timeout-copy=20000 --timeout-build=20000 --env=ADT_TEST_TRIGGERS=ecl/21.2.1+ds-4.1ubuntu1 -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor aut
opkgtest --security-groups <email address hidden> --name adt-noble-i386-cl-postmodern-20240328-102239-juju-7f2275-prod-proposed-migrati
on-environment-2-66bee8db-aaa8-437a-913b-a9d1f6f558dd --image adt/ubuntu-noble-amd64-server --keyname testbed-juju-7f2275-prod-proposed-migration-environment-2 --net-id=net_prod-proposed-migr
ation -e TERM=linux -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,login.ubuntu.com,localhost,localdomain,novalocal,in
ternal,archive.ubuntu.com,ports.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpadlibrarian.net,launchpadcontent.net,launchpad.net,10.24.0.0/24,keystone.ps5.canon
ical.com,objectstorage.prodstack5.canonical.com' --mirror=http://ftpmaster.internal/ubuntu/
Mar 28 10:52:37 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: autopkgtest exited with code 8
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Putting results into swift autopkgtest-noble noble/i386/c/cl-post
modern/20240328_105239_f7be3@
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Acknowledging request cl-postmodern
                                                                                                                        {"triggers": ["ecl/21.2.1+ds-4.1ubuntu1"], "all-proposed": "1", "submit
-time": "2024-03-28 09:50:20"}
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Waiting for and processing AMQP requests
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: ERROR: Received invalid request format "b'libbytes-random-secure-perl\n
{"all-proposed": "1", "triggers": ["perl/5.38.2-3.2"], "uuid": "cead739f-4eab-47a0-92bc-f73721f82337", "submit-time": "2024-03-27 11:07:06"}'
                                                                                                                        "
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Waiting for and processing AMQP requests

stracing the process showed:

ubuntu@juju-7f2275-prod-proposed-migration-environment-2:~/autopkgtest-cloud/tools$ sudo strace -p 865222
strace: Process 865222 attached
recvfrom(4,

and the journal output:

buntu@juju-7f2275-prod-proposed-migration-environment-2:~/autopkgtest-cloud/tools$ journalctl -u <email address hidden> --since "9.5h ago"
-- Logs begin at Mon 2024-03-25 15:03:10 UTC, end at Thu 2024-03-28 20:07:20 UTC. --
Mar 28 10:52:37 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: autopkgtest exited with code 8
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Putting results into swift autopkgtest-noble noble/i386/c/cl-pos>
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Acknowledging request cl-postmodern
                                                                                                                        {"triggers": ["ecl/21.2.1+ds-4.1ubuntu1"], "all-proposed": "1", "submi>
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Waiting for and processing AMQP requests
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: ERROR: Received invalid request format "b'libbytes-random-secure-perl\>
                                                                                                                        "
Mar 28 10:52:39 juju-7f2275-prod-proposed-migration-environment-2 /home/ubuntu/autopkgtest-cloud/worker/worker[865222]: INFO: Waiting for and processing AMQP requests

Related branches

summary: - cloud-worker hung after an invalid request
+ cloud-worker stuck after an invalid request
Changed in auto-package-testing:
importance: Undecided → High
Changed in auto-package-testing:
status: New → In Progress
assignee: nobody → Skia (hyask)
Paride Legovini (paride)
Changed in auto-package-testing:
status: In Progress → Fix Committed
Revision history for this message
Tim Andersson (andersson123) wrote :

This has been deployed.

Changed in auto-package-testing:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.