Empty exchanges after sequence number jumping.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Landscape Client |
Fix Released
|
High
|
Kevin Nasto | ||
landscape-client (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
If landscape-server next-expected-
Reproduction:
1. register client. Let it exchange.
2. on the server reset the sequence number to something high:
psql landscape-test-main -c 'update computer_status set next_expected_
Result
If debuging is set on the client, you'll see only empty outgoing messsages:
{.. 'messages': [], 'sequence': 9000, 'server-api': '3.3', 'total-messages': 0}
Expected
If the expected sequence is above the current message number, client should reset its sequence number
to the next-expected-
How we got there is still somewhat of a mystery, but my best guess is there
were stale message-server processes still using the old database server when
we did the DB server switch. This led to an exchange which messed clients
expectation of sequence numbers:
Client: 10 messages, sequence 100
New server: ack. next-expected-
Client: 2 messages, sequence 110
Stale server: (rejects message) next-expected-
Client: (tries to replay from 100. can't. resets sequence to 100)
Client: 2 messages, sequence: 100
New server: (ignores messages). next-expected-
* At this point the client is stuck sending empty messages (as it has no message 110)
with the same sequence number (110).
Sequence numbers don't increase (as empty messages are sent).
Server considers those empty message as successful exchanges.
Those empty messages will carry on and client broker just drops future outgoing messages.
Also, I suspect it's possible to get back to that issue with database backups restoration.
information type: | Private → Public |
Changed in landscape-client: | |
assignee: | nobody → Kevin Nasto (silverdrake11) |
status: | New → In Progress |
Changed in landscape-client: | |
status: | In Progress → Fix Committed |
no longer affects: | landscape |
Changed in landscape-client: | |
status: | Fix Committed → Fix Released |
The pull request is here https:/ /github. com/CanonicalLt d/landscape- client/ pull/102