MSDF: Memory Statistics Data Format used in system monitoring

Abstract

High performance computing (HPC) system or computing system is very different from ordinary service system. In general, service system only run some specific applications, e.g. web server or mail server to serve as many requests from users as possible while in computing system, users have the permission to run their own applications and isolated with each other. Monitoring technique is the key to ensure system efficiency and users satisfaction, and by combining monitoring together with data analysis, system administrators can solve several operating problems specific to computing system such as resource allocation, application scheduling, abnormal detection, etc. Different from service system while administrators usually prefer system overall information rather than information of each individual user applications in computing system. Since computing system usually contains many applications executed simultaneously, monitoring computing system with traditional approaches would potentially consume a huge amount of storage space and would cost more charge fee if system is deployed in cloud environment. This article focuses on analyzing monitored memory usage data retrieved from computing program in order to benefit its next resource allocation. Different from traditional approaches with batch processing technique in which collected data is all stored in database before analyzing, we utilized online analysis approaches in which every new coming data is captured, processed, cached in order to transform into useful information, and only allow necessary data be stored in database. We propose Memory Statistics Data Format (MSDF), an on-the-fly processing technique used in monitoring memory usage of computing application for saving storage space while still preserve enough information to solve resource allocation problem. MSDF can help to save more than 95% of storage space while allocation efficiency is always guaranteed depend on the εparameter and MSDF can be extended to solve other operating problem or adapted to montior and analyze other remaining application metrics.