Parallel processing of large data files on distributed file systems with dynamic workload balancing

Parallel processing files on a distributed file system, responds to a request to process the file, by partitioning the file into a set of predetermined sized blocks by a single coordinator and assigns blocks to a reader in a set of readers. Block assignments are recorded to a control table accessibl...

Full description

Saved in:
Bibliographic Details
Main Authors Kunic, Toni, Raspudic, Steven, Abrashkevich, Alexander
Format Patent
LanguageEnglish
Published 27.11.2018
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC COMMUNICATION TECHNIQUE
ELECTRIC DIGITAL DATA PROCESSING
ELECTRICITY
PHYSICS
TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHICCOMMUNICATION
Online AccessGet full text

Cover

More Information
Summary:Parallel processing files on a distributed file system, responds to a request to process the file, by partitioning the file into a set of predetermined sized blocks by a single coordinator and assigns blocks to a reader in a set of readers. Block assignments are recorded to a control table accessible to all readers. The set of readers search the control table for a row assigned to a respective reader. Responsive to locating the row, the respective reader changes a state of the row to in progress updating an associated timestamp. The respective reader scans the control table for an available set of the blocks, when all blocks currently assigned are exhausted and dynamically re-assigns an available block from another owning reader to itself and changes ownership to itself. When no rows are available, the respective reader completes processing and returns an end of file indication to the single coordinator.
Bibliography:Application Number: US201514866966