sahara job binary copy times out when files is too big
Bug #1705762 reported by
Telles Mota Vidal Nóbrega
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Sahara |
In Progress
|
Medium
|
Telles Mota Vidal Nóbrega |
Bug Description
Sahara times out during the copy of job binary into the cluster if the binary is too big. The size of file tested was around 115MB and it took around 15 minutes to copy.
The copy is done using paramiko sftp on the sahara/
def _write_fl(sftp, remote_file, data):
fl = sftp.file(
fl.write(data)
fl.close()
A little research says that increasing transfer window size could be helpful but we need deeper investigation on it.
Another possible solution is change sftp write for sftp put.
Changed in sahara: | |
assignee: | nobody → Telles Mota Vidal Nóbrega (tellesmvn) |
importance: | Undecided → Medium |
Changed in sahara: | |
status: | New → Triaged |
To post a comment you must log in.
Another possible solution is moving job binary retrieving to be inside the cluster itself instead of inside Sahara.