MAAS

[2.4] Serialization errors in single region/rack, due to each worker update every 1 minute

Bug #1756341 reported by Andres Rodriguez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Blake Rouse
MAAS 2.4.0beta1

Bug Description

A lot of serialization errors on a single region/rack (It is node with ID=1) (see below).

Also, while MAAS is deploying/commissioning machines, I see https://pastebin.ubuntu.com/p/Y37HvkQx7F/. There's 4 errors, which seems to imply 4 workers.

2018-03-16 13:04:51.661 UTC [9139] LOG: listening on IPv4 address "127.0.0.1", port 5432
2018-03-16 13:04:51.662 UTC [9139] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2018-03-16 13:04:51.674 UTC [9140] LOG: database system was shut down at 2018-03-16 13:04:50 UTC
2018-03-16 13:04:51.677 UTC [9139] LOG: database system is ready to accept connections
2018-03-16 13:04:52.252 UTC [9147] [unknown]@[unknown] LOG: incomplete startup packet
2018-03-16 13:06:01.782 UTC [9139] LOG: received fast shutdown request
2018-03-16 13:06:01.783 UTC [9139] LOG: aborting any active transactions
2018-03-16 13:06:01.786 UTC [9139] LOG: worker process: logical replication launcher (PID 9146) exited with exit code 1
2018-03-16 13:06:01.789 UTC [9141] LOG: shutting down
2018-03-16 13:06:01.796 UTC [9139] LOG: database system is shut down
2018-03-16 13:06:01.947 UTC [10575] LOG: listening on IPv4 address "127.0.0.1", port 5432
2018-03-16 13:06:01.948 UTC [10575] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2018-03-16 13:06:01.957 UTC [10576] LOG: database system was shut down at 2018-03-16 13:06:01 UTC
2018-03-16 13:06:01.960 UTC [10575] LOG: database system is ready to accept connections
2018-03-16 13:06:02.541 UTC [10583] [unknown]@[unknown] LOG: incomplete startup packet
2018-03-16 13:06:04.333 UTC [10760] maas@template1 FATAL: password authentication failed for user "maas"
2018-03-16 13:06:04.333 UTC [10760] maas@template1 DETAIL: Role "maas" does not exist.
 Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5"
2018-03-16 13:06:04.335 UTC [10761] maas@template1 FATAL: password authentication failed for user "maas"
2018-03-16 13:06:04.335 UTC [10761] maas@template1 DETAIL: Role "maas" does not exist.
 Connection matched pg_hba.conf line 92: "host all all 127.0.0.1/32 md5"
2018-03-16 13:06:04.552 UTC [10838] postgres@template1 ERROR: role "maas" does not exist
2018-03-16 13:06:04.552 UTC [10838] postgres@template1 STATEMENT: ALTER USER "maas"
2018-03-16 13:06:04.983 UTC [10995] postgres@maasdb FATAL: database "maasdb" does not exist
2018-03-16 13:06:25.868 UTC [11089] maas@maasdb ERROR: relation "maasserver_zone_serial_seq" already exists
2018-03-16 13:06:25.868 UTC [11089] maas@maasdb STATEMENT: CREATE SEQUENCE maasserver_zone_serial_seq INCREMENT BY 1
 MINVALUE 1 MAXVALUE 4294967295 CYCLE OWNED BY maasserver_dnspublication.serial;

2018-03-16 13:06:45.048 UTC [11436] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:45.048 UTC [11436] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:06:44.790233'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 1
2018-03-16 13:06:45.062 UTC [11444] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:45.062 UTC [11444] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:06:45.061018'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:06:45.204 UTC [11444] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:45.204 UTC [11444] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:06:45.203814'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 2
2018-03-16 13:06:45.229 UTC [11468] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:45.229 UTC [11468] maas@maasdb STATEMENT: UPDATE "maasserver_service" SET "updated" = '2018-03-16T13:06:45.228894'::timestamp, "status" = 'running' WHERE "maasserver_service"."id" = 3
2018-03-16 13:06:45.844 UTC [11494] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:45.844 UTC [11494] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:06:45.803510'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:06:45.995 UTC [11494] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:45.995 UTC [11494] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:06:45.981264'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 3
2018-03-16 13:06:46.402 UTC [11488] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:46.402 UTC [11488] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:06:46.294984'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:06:49.124 UTC [11504] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:49.124 UTC [11504] maas@maasdb CONTEXT: SQL statement "UPDATE maasserver_node SET managing_process_id = region_process.id
   WHERE maasserver_node.id = rack.id"
 PL/pgSQL function sys_core_set_new_region(maasserver_node) line 9 at SQL statement
 SQL statement "SELECT sys_core_set_new_region(rack_controller)"
 PL/pgSQL function sys_core_rpc_insert() line 14 at PERFORM
2018-03-16 13:06:49.124 UTC [11504] maas@maasdb STATEMENT: INSERT INTO "maasserver_regionrackrpcconnection" ("created", "updated", "endpoint_id", "rack_controller_id") VALUES ('2018-03-16T13:06:49.011263'::timestamp, '2018-03-16T13:06:49.011263'::timestamp, 4, 1) RETURNING "maasserver_regionrackrpcconnection"."id"
2018-03-16 13:06:49.264 UTC [11436] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:49.264 UTC [11436] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:06:49.244633'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:06:49.265 UTC [11482] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:49.265 UTC [11482] maas@maasdb CONTEXT: SQL statement "UPDATE maasserver_node SET managing_process_id = region_process.id
   WHERE maasserver_node.id = rack.id"
 PL/pgSQL function sys_core_set_new_region(maasserver_node) line 9 at SQL statement
 SQL statement "SELECT sys_core_set_new_region(rack_controller)"
 PL/pgSQL function sys_core_rpc_insert() line 14 at PERFORM
2018-03-16 13:06:49.265 UTC [11482] maas@maasdb STATEMENT: INSERT INTO "maasserver_regionrackrpcconnection" ("created", "updated", "endpoint_id", "rack_controller_id") VALUES ('2018-03-16T13:06:49.250851'::timestamp, '2018-03-16T13:06:49.250851'::timestamp, 3, 1) RETURNING "maasserver_regionrackrpcconnection"."id"
2018-03-16 13:06:52.416 UTC [11441] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:06:52.416 UTC [11441] maas@maasdb STATEMENT: DELETE FROM "maasserver_notification" WHERE "maasserver_notification"."id" IN (3)
2018-03-16 13:07:06.079 UTC [11534] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.085 UTC [11314] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.090 UTC [11468] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.092 UTC [11494] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.096 UTC [11310] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.097 UTC [11308] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.098 UTC [11306] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.100 UTC [11304] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.101 UTC [11302] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.101 UTC [11302] maas@maasdb LOG: unexpected EOF on client connection with an open transaction
2018-03-16 13:07:06.102 UTC [11300] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.103 UTC [11298] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.105 UTC [11296] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.106 UTC [11278] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.108 UTC [11513] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.109 UTC [11504] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.111 UTC [11488] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.113 UTC [11482] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.115 UTC [11479] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.116 UTC [11473] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.117 UTC [11453] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.119 UTC [11441] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.120 UTC [11439] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.121 UTC [11436] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.122 UTC [11524] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.124 UTC [11458] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:06.125 UTC [11444] maas@maasdb LOG: could not receive data from client: Connection reset by peer
2018-03-16 13:07:21.978 UTC [11873] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:21.978 UTC [11873] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:07:21.922712'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 5
2018-03-16 13:07:22.353 UTC [11873] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:22.353 UTC [11873] maas@maasdb STATEMENT: UPDATE "maasserver_service" SET "updated" = '2018-03-16T13:07:22.353078'::timestamp, "status_info" = '25% connected to region controllers.' WHERE "maasserver_service"."id" = 8
2018-03-16 13:07:22.379 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:22.379 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:07:22.378972'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 6
2018-03-16 13:07:22.432 UTC [11888] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:22.432 UTC [11888] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:07:22.397079'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 7
2018-03-16 13:07:22.716 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:22.716 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_service" SET "updated" = '2018-03-16T13:07:22.715763'::timestamp, "status_info" = '0% connected to region controllers.' WHERE "maasserver_service"."id" = 8
2018-03-16 13:07:23.171 UTC [11873] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:23.171 UTC [11873] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:07:22.470775'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:07:23.172 UTC [11896] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:23.172 UTC [11896] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:07:22.581732'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:07:23.172 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:23.172 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:07:22.817116'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:07:23.242 UTC [11873] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:23.242 UTC [11873] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:07:23.222562'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:07:23.352 UTC [11873] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:23.352 UTC [11873] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:07:23.335998'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:07:24.819 UTC [11852] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:24.819 UTC [11852] maas@maasdb CONTEXT: SQL statement "UPDATE maasserver_node SET managing_process_id = region_process.id
   WHERE maasserver_node.id = rack.id"
 PL/pgSQL function sys_core_set_new_region(maasserver_node) line 9 at SQL statement
 SQL statement "SELECT sys_core_set_new_region(rack_controller)"
 PL/pgSQL function sys_core_rpc_insert() line 14 at PERFORM
2018-03-16 13:07:24.819 UTC [11852] maas@maasdb STATEMENT: INSERT INTO "maasserver_regionrackrpcconnection" ("created", "updated", "endpoint_id", "rack_controller_id") VALUES ('2018-03-16T13:07:24.742274'::timestamp, '2018-03-16T13:07:24.742274'::timestamp, 6, 1) RETURNING "maasserver_regionrackrpcconnection"."id"
2018-03-16 13:07:24.875 UTC [11877] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:24.875 UTC [11877] maas@maasdb CONTEXT: SQL statement "UPDATE maasserver_node SET managing_process_id = region_process.id
   WHERE maasserver_node.id = rack.id"
 PL/pgSQL function sys_core_set_new_region(maasserver_node) line 9 at SQL statement
 SQL statement "SELECT sys_core_set_new_region(rack_controller)"
 PL/pgSQL function sys_core_rpc_insert() line 14 at PERFORM
2018-03-16 13:07:24.875 UTC [11877] maas@maasdb STATEMENT: INSERT INTO "maasserver_regionrackrpcconnection" ("created", "updated", "endpoint_id", "rack_controller_id") VALUES ('2018-03-16T13:07:24.863502'::timestamp, '2018-03-16T13:07:24.863502'::timestamp, 5, 1) RETURNING "maasserver_regionrackrpcconnection"."id"
2018-03-16 13:07:24.875 UTC [11863] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:07:24.875 UTC [11863] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:07:24.867795'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:08:53.193 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:08:53.193 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_service" SET "updated" = '2018-03-16T13:08:53.169790'::timestamp, "status" = 'off' WHERE "maasserver_service"."id" = 4
2018-03-16 13:09:22.556 UTC [12129] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:22.556 UTC [12129] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:09:22.379879'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 5
2018-03-16 13:09:22.606 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:22.606 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:09:22.605989'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 6
2018-03-16 13:09:22.622 UTC [11877] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:22.622 UTC [11877] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:09:22.622172'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 7
2018-03-16 13:09:22.895 UTC [11877] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:22.895 UTC [11877] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:09:22.715815'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:09:22.895 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:22.895 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:09:22.719805'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:09:22.895 UTC [11887] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:22.895 UTC [11887] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:09:22.743859'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:09:23.143 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:23.143 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:09:22.981560'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:09:23.143 UTC [11877] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:23.143 UTC [11877] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:09:22.987816'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:09:23.417 UTC [11875] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:09:23.417 UTC [11875] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:09:23.251810'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:10:22.082 UTC [11873] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:10:22.082 UTC [11873] maas@maasdb STATEMENT: UPDATE "maasserver_regioncontrollerprocess" SET "updated" = '2018-03-16T13:10:21.938138'::timestamp WHERE "maasserver_regioncontrollerprocess"."id" = 5
2018-03-16 13:10:22.377 UTC [11863] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:10:22.377 UTC [11863] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:10:22.243852'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:10:22.377 UTC [11850] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:10:22.377 UTC [11850] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:10:22.348937'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:10:22.574 UTC [11863] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:10:22.574 UTC [11863] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:10:22.422307'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:10:22.574 UTC [11887] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:10:22.574 UTC [11887] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:10:22.558908'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:10:22.660 UTC [11863] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:10:22.660 UTC [11863] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:10:22.610920'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:19:22.015 UTC [14002] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:19:22.015 UTC [14002] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:19:22.013792'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:22:22.085 UTC [14002] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:22:22.085 UTC [14002] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:22:22.084141'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:22:22.387 UTC [14599] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:22:22.387 UTC [14599] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:22:22.321535'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:23:22.462 UTC [14243] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:23:22.462 UTC [14243] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:23:22.212767'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:23:22.501 UTC [14599] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:23:22.501 UTC [14599] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:23:22.499462'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:23:22.957 UTC [14599] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:23:22.957 UTC [14599] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:23:22.608715'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:23:22.957 UTC [15010] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:23:22.957 UTC [15010] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:23:22.710849'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:23:23.451 UTC [14599] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:23:23.451 UTC [14599] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:23:23.058358'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:23:27.889 UTC [14507] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:23:27.889 UTC [14507] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 8 FOR UPDATE NOWAIT
2018-03-16 13:24:22.212 UTC [15012] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:24:22.212 UTC [15012] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:24:22.102957'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:24:22.502 UTC [15122] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:24:22.502 UTC [15122] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:24:22.399510'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:24:22.798 UTC [15033] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:24:22.798 UTC [15033] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:24:22.687781'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:25:22.654 UTC [15012] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:25:22.654 UTC [15012] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:25:22.421342'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:25:22.655 UTC [15525] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:25:22.655 UTC [15525] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:25:22.590309'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:25:23.256 UTC [15525] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:25:23.256 UTC [15525] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:25:22.741103'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:25:23.256 UTC [15010] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:25:23.256 UTC [15010] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:25:22.828018'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:25:23.687 UTC [15525] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:25:23.687 UTC [15525] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:25:23.371286'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:26:01.325 UTC [15028] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:01.325 UTC [15028] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:26:01.318252'::timestamp WHERE "metadataserver_scriptset"."id" = 13
2018-03-16 13:26:01.326 UTC [15572] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:01.326 UTC [15572] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:26:01.324332'::timestamp WHERE "metadataserver_scriptset"."id" = 13
2018-03-16 13:26:02.450 UTC [15061] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:02.450 UTC [15061] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:26:02.435403'::timestamp WHERE "metadataserver_scriptset"."id" = 13
2018-03-16 13:26:21.719 UTC [15012] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:26:21.719 UTC [15012] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 18 FOR UPDATE NOWAIT
2018-03-16 13:26:22.139 UTC [15007] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:22.139 UTC [15007] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:26:22.108980'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:26:22.523 UTC [14993] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:22.523 UTC [14993] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:26:22.444226'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:26:22.876 UTC [15061] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:22.876 UTC [15061] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:26:22.653581'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:26:32.026 UTC [14982] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:32.026 UTC [14982] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:26:32.016320'::timestamp WHERE "metadataserver_scriptset"."id" = 11
2018-03-16 13:26:33.151 UTC [15562] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:33.151 UTC [15562] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:26:33.151254'::timestamp WHERE "metadataserver_scriptset"."id" = 11
2018-03-16 13:26:34.251 UTC [14993] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:26:34.251 UTC [14993] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:26:34.250606'::timestamp WHERE "metadataserver_scriptset"."id" = 11
2018-03-16 13:27:22.015 UTC [15530] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:27:22.015 UTC [15530] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 18 FOR UPDATE NOWAIT
2018-03-16 13:27:22.159 UTC [15011] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:27:22.159 UTC [15011] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:27:22.123295'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:27:22.499 UTC [14993] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:27:22.499 UTC [14993] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:27:22.380607'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:27:22.761 UTC [15029] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:27:22.761 UTC [15029] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:27:22.611157'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:27:28.345 UTC [15272] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:27:28.345 UTC [15272] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 26 FOR UPDATE NOWAIT
2018-03-16 13:27:28.364 UTC [15272] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:27:28.364 UTC [15272] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 26 FOR UPDATE NOWAIT
2018-03-16 13:27:28.416 UTC [15272] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:27:28.416 UTC [15272] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 26 FOR UPDATE NOWAIT
2018-03-16 13:27:28.874 UTC [15060] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:27:28.874 UTC [15060] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 26 FOR UPDATE NOWAIT
2018-03-16 13:27:59.667 UTC [15825] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:27:59.667 UTC [15825] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:27:59.663597'::timestamp WHERE "metadataserver_scriptset"."id" = 19
2018-03-16 13:28:12.196 UTC [15797] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:12.196 UTC [15797] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:12.177401'::timestamp WHERE "metadataserver_scriptset"."id" = 23
2018-03-16 13:28:13.433 UTC [15833] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:13.433 UTC [15833] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:13.433058'::timestamp WHERE "metadataserver_scriptset"."id" = 23
2018-03-16 13:28:14.597 UTC [15853] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:14.597 UTC [15853] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:14.596420'::timestamp WHERE "metadataserver_scriptset"."id" = 23
2018-03-16 13:28:14.725 UTC [15562] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:14.725 UTC [15562] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:14.721930'::timestamp WHERE "metadataserver_scriptset"."id" = 15
2018-03-16 13:28:15.913 UTC [15623] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:15.913 UTC [15623] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:15.913187'::timestamp WHERE "metadataserver_scriptset"."id" = 15
2018-03-16 13:28:16.992 UTC [15826] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:16.992 UTC [15826] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:16.992103'::timestamp WHERE "metadataserver_scriptset"."id" = 15
2018-03-16 13:28:20.302 UTC [15825] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:20.302 UTC [15825] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:20.290376'::timestamp WHERE "metadataserver_scriptset"."id" = 25
2018-03-16 13:28:20.303 UTC [15263] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:20.303 UTC [15263] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:20.280810'::timestamp WHERE "metadataserver_scriptset"."id" = 25
2018-03-16 13:28:21.471 UTC [15825] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:21.471 UTC [15825] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:21.450224'::timestamp WHERE "metadataserver_scriptset"."id" = 25
2018-03-16 13:28:21.491 UTC [15122] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:21.491 UTC [15122] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:21.490826'::timestamp WHERE "metadataserver_scriptset"."id" = 25
2018-03-16 13:28:22.724 UTC [15776] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:22.724 UTC [15776] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:22.721283'::timestamp WHERE "metadataserver_scriptset"."id" = 25
2018-03-16 13:28:22.821 UTC [15544] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:22.821 UTC [15544] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:22.820477'::timestamp WHERE "metadataserver_scriptset"."id" = 25
2018-03-16 13:28:30.061 UTC [15798] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:30.061 UTC [15798] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:30.057021'::timestamp WHERE "metadataserver_scriptset"."id" = 21
2018-03-16 13:28:41.207 UTC [15544] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:41.207 UTC [15544] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:41.201155'::timestamp WHERE "metadataserver_scriptset"."id" = 27
2018-03-16 13:28:50.259 UTC [15825] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:50.259 UTC [15825] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:50.254388'::timestamp WHERE "metadataserver_scriptset"."id" = 31
2018-03-16 13:28:50.703 UTC [15718] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:28:50.703 UTC [15718] maas@maasdb STATEMENT: UPDATE "metadataserver_scriptset" SET "last_ping" = '2018-03-16T13:28:50.699204'::timestamp WHERE "metadataserver_scriptset"."id" = 29
2018-03-16 13:33:22.154 UTC [16086] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:33:22.154 UTC [16086] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:33:22.152830'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:33:22.524 UTC [18522] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:33:22.524 UTC [18522] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:33:22.411532'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:33:22.856 UTC [19747] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:33:22.856 UTC [19747] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:33:22.678316'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:34:22.128 UTC [16086] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:34:22.128 UTC [16086] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:34:22.094923'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:34:22.933 UTC [18522] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:34:22.933 UTC [18522] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:34:22.698028'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:34:24.066 UTC [19747] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:34:24.066 UTC [19747] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:34:23.278590'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:35:22.115 UTC [19651] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:35:22.115 UTC [19651] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:35:22.114230'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:35:22.422 UTC [18522] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:35:22.422 UTC [18522] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:35:22.371841'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:35:22.745 UTC [19749] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:35:22.745 UTC [19749] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:35:22.649842'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:36:22.860 UTC [19427] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:36:22.860 UTC [19427] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:36:22.393813'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:36:22.860 UTC [21867] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:36:22.860 UTC [21867] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:36:22.680815'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:36:22.961 UTC [19673] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:36:22.961 UTC [19673] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:36:22.960379'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:36:23.314 UTC [19427] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:36:23.314 UTC [19427] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:36:22.907791'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:36:23.314 UTC [19673] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:36:23.314 UTC [19673] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:36:23.102176'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:36:23.834 UTC [19673] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:36:23.834 UTC [19673] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:36:23.415148'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:36:24.917 UTC [21235] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:36:24.917 UTC [21235] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 35 FOR UPDATE NOWAIT
2018-03-16 13:37:22.229 UTC [19428] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:37:22.229 UTC [19428] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:37:22.084430'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:37:22.529 UTC [21867] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:37:22.529 UTC [21867] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:37:22.425571'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:37:22.788 UTC [19673] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:37:22.788 UTC [19673] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:37:22.687868'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:38:22.772 UTC [21866] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:38:22.772 UTC [21866] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:38:22.303838'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:38:22.772 UTC [21867] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:38:22.772 UTC [21867] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:38:22.613157'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:38:22.812 UTC [19749] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:38:22.812 UTC [19749] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:38:22.810761'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:38:23.229 UTC [21867] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:38:23.229 UTC [21867] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:38:22.835811'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:38:23.229 UTC [19749] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:38:23.229 UTC [19749] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:38:22.913995'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:38:23.637 UTC [21867] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:38:23.637 UTC [21867] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:38:23.322633'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:39:22.439 UTC [21235] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:39:22.439 UTC [21235] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:39:22.438064'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:39:22.636 UTC [19762] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:39:22.636 UTC [19762] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:39:22.581370'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:40:21.564 UTC [19671] maas@maasdb ERROR: could not obtain lock on row in relation "metadataserver_scriptset"
2018-03-16 13:40:21.564 UTC [19671] maas@maasdb STATEMENT: SELECT "metadataserver_scriptset"."id", "metadataserver_scriptset"."last_ping", "metadataserver_scriptset"."node_id", "metadataserver_scriptset"."result_type", "metadataserver_scriptset"."power_state_before_transition", "metadataserver_scriptset"."requested_scripts" FROM "metadataserver_scriptset" WHERE "metadataserver_scriptset"."id" = 37 FOR UPDATE NOWAIT
2018-03-16 13:40:22.772 UTC [25313] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:40:22.772 UTC [25313] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:40:22.419845'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:40:22.772 UTC [24477] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:40:22.772 UTC [24477] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:40:22.692861'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:40:23.323 UTC [25313] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:40:23.323 UTC [25313] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:40:22.879818'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:40:23.323 UTC [27318] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:40:23.323 UTC [27318] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:40:22.946798'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:40:23.711 UTC [25313] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:40:23.711 UTC [25313] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:40:23.432054'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:41:22.245 UTC [22482] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:41:22.245 UTC [22482] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:41:22.111828'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:41:22.464 UTC [26117] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:41:22.464 UTC [26117] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:41:22.378789'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:41:22.740 UTC [27396] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:41:22.740 UTC [27396] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:41:22.634994'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:42:22.529 UTC [25313] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:42:22.529 UTC [25313] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:42:22.352957'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1
2018-03-16 13:42:22.808 UTC [25238] maas@maasdb ERROR: could not serialize access due to concurrent update
2018-03-16 13:42:22.808 UTC [25238] maas@maasdb STATEMENT: UPDATE "maasserver_node" SET "updated" = '2018-03-16T13:42:22.683816'::timestamp, "status_expires" = NULL WHERE "maasserver_node"."id" = 1

See original description
Tags: performance

Related branches

~blake-rouse/maas:ipc-endpoints
MAAS Lander: Approve
Alberto Donato (community): Approve
Diff: 4220 lines (+1137/-1712)
16 files modified
src/maasserver/eventloop.py (+12/-23)
src/maasserver/ipc.py (+521/-21)
src/maasserver/models/node.py (+4/-3)
src/maasserver/rack_controller.py (+5/-7)
src/maasserver/regiondservices/service_monitor_service.py (+6/-7)
src/maasserver/regiondservices/tests/test_service_monitor_service.py (+12/-24)
src/maasserver/rpc/regionservice.py (+37/-602)
src/maasserver/rpc/tests/test_regionservice.py (+74/-830)
src/maasserver/server.py (+4/-1)
src/maasserver/tests/test_eventloop.py (+7/-21)
src/maasserver/tests/test_ipc.py (+352/-21)
src/maasserver/tests/test_plugin.py (+1/-3)
src/maasserver/tests/test_rack_controller.py (+39/-44)
src/maasserver/triggers/tests/test_websocket_listener.py (+2/-1)
src/maasserver/views/rpc.py (+41/-40)
src/maasserver/views/tests/test_rpc.py (+20/-64)
Andres Rodriguez (andreserl)
Changed in maas:
milestone: none → 2.4.0beta1
tags: added: performance
Changed in maas:
importance: Undecided → High
status: New → Triaged
assignee: nobody → Blake Rouse (blake-rouse)
Andres Rodriguez (andreserl)
description: updated
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Other errors happen every 1 minute: https://pastebin.ubuntu.com/p/6zB53kWXJk/

Andres Rodriguez (andreserl)
summary: - [2.4] Serialization errors in single region/rack
+ [2.4] Serialization errors in single region/rack, every 1 minute
summary: - [2.4] Serialization errors in single region/rack, every 1 minute
+ [2.4] Serialization errors in single region/rack, due to each worker
+ update every 1 minute
Blake Rouse (blake-rouse)
Changed in maas:
status: Triaged → In Progress
Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.4.0beta1 → 2.4.0beta2
Andres Rodriguez (andreserl)
Changed in maas:
status: In Progress → Fix Committed
milestone: 2.4.0beta2 → 2.4.0beta1
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.