Search is slow on high volume logs.

Bug #1367328 reported by Ben Kietzman on 2014-09-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Logger
Medium
Ben Kietzman

Bug Description

Fetching search results for high volume logs is rather slow. Need to look into quicking the search capabilities.

Ben Kietzman (ben-kietzman) wrote :

Application indexes and their associated data files are organized by day (YYYYMMDD). The current logic has one thread allocated for searching through all the indexes and their associated data files. A possible solution would be to allocate one thread for each daily index file. That way all the files are searched in parallel instead of one at a time.

Changed in logger:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Ben Kietzman (ben-kietzman)
Ben Kietzman (ben-kietzman) wrote :

Completed the new logic. Decided to not thread out the entire search process since searching the data files in a threaded fashion ended up causing the I/O wait time to jump to the ceiling. Ended up doing all file I/O in the main thread and then threaded out the content of each dated index file in order to speed up the searche time.

Changed in logger:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers