接收与处理分离的实时大数据处理模型
TP311; 在大数据处理过程中，系统必须有非常高的数据处理效率。为了满足对大数据实时、高效、稳定处理的需求，提出了一种接收与处理分离的数据处理模型。该数据处理模型由数据接收单元、内存数据库、原始数据分发单元、数据处理单元、处理数据分发单元、数据归并单元组成。接收单元负责接收、整合结构化数据与非结构化数据，把每条完整的数据放入内存数据库中；分发单元从内存数据库中检测获取数据，按照海量数据负载均衡算法把数据分发到数据处理单元；数据处理单元处理数据，处理结果放入内存数据库；处理数据分发单元继续从内存数据库中获取处理后的数据，并按照海量数据负载均衡算法把数据分发给数据归并单元。实验证明，使用该模型方...
Saved in:
| Published in | 计算机科学与探索 no. 8; pp. 906 - 913 |
|---|---|
| Main Authors | , , , , |
| Format | Journal Article |
| Language | Chinese |
| Published |
河海大学 计算机与信息学院,南京,210098%常州工学院,江苏 常州,213002
2015
|
| Subjects | |
| Online Access | Get full text |
| ISSN | 1673-9418 |
| DOI | 10.3778/j.issn.1673-9418.1411041 |
Cover
| Abstract | TP311; 在大数据处理过程中，系统必须有非常高的数据处理效率。为了满足对大数据实时、高效、稳定处理的需求，提出了一种接收与处理分离的数据处理模型。该数据处理模型由数据接收单元、内存数据库、原始数据分发单元、数据处理单元、处理数据分发单元、数据归并单元组成。接收单元负责接收、整合结构化数据与非结构化数据，把每条完整的数据放入内存数据库中；分发单元从内存数据库中检测获取数据，按照海量数据负载均衡算法把数据分发到数据处理单元；数据处理单元处理数据，处理结果放入内存数据库；处理数据分发单元继续从内存数据库中获取处理后的数据，并按照海量数据负载均衡算法把数据分发给数据归并单元。实验证明，使用该模型方法，系统保持了非常高效的处理效率。 |
|---|---|
| AbstractList | TP311; 在大数据处理过程中，系统必须有非常高的数据处理效率。为了满足对大数据实时、高效、稳定处理的需求，提出了一种接收与处理分离的数据处理模型。该数据处理模型由数据接收单元、内存数据库、原始数据分发单元、数据处理单元、处理数据分发单元、数据归并单元组成。接收单元负责接收、整合结构化数据与非结构化数据，把每条完整的数据放入内存数据库中；分发单元从内存数据库中检测获取数据，按照海量数据负载均衡算法把数据分发到数据处理单元；数据处理单元处理数据，处理结果放入内存数据库；处理数据分发单元继续从内存数据库中获取处理后的数据，并按照海量数据负载均衡算法把数据分发给数据归并单元。实验证明，使用该模型方法，系统保持了非常高效的处理效率。 |
| Abstract_FL | The system must have high data processing efficiency during processing big data. This paper designs a data processing model based on receiving and processing separation. Using the method the system can real-time, effi-ciently, stably process data. The model includes the data receiving unit, the memory database, the original data distri-bution unit, the data processing unit, the processed data distribution unit and the data merging unit. The receiving unit receives and integrates structure and unstructured data to a whole structure datum and puts it into the memory database. The original data distribution unit checks data from the memory database and dispatches the data to the data processing unit using the big data load balancing method. The data processing unit processes data and puts the result data into the memory database again. The processed data distribution unit gets data from the memory database and dispatches the data to the data merging unit using the big data load balancing method. The experimental results show that the system keeps with the high efficiency using the model. |
| Author | 李臣明 彭建华 邱军林 李晓芳 徐立中 |
| AuthorAffiliation | 河海大学 计算机与信息学院,南京,210098%常州工学院,江苏 常州,213002 |
| AuthorAffiliation_xml | – name: 河海大学 计算机与信息学院,南京,210098%常州工学院,江苏 常州,213002 |
| Author_FL | XU Lizhong LI Xiaofang LI Chenming PENG Jianhua QIU Junlin |
| Author_FL_xml | – sequence: 1 fullname: PENG Jianhua – sequence: 2 fullname: LI Chenming – sequence: 3 fullname: QIU Junlin – sequence: 4 fullname: LI Xiaofang – sequence: 5 fullname: XU Lizhong |
| Author_xml | – sequence: 1 fullname: 彭建华 – sequence: 2 fullname: 李臣明 – sequence: 3 fullname: 邱军林 – sequence: 4 fullname: 李晓芳 – sequence: 5 fullname: 徐立中 |
| BookMark | eNo9jbtKA0EYRqeIYIx5B1uLXeff-XdmtpTgDQJp0ofZ3RnJKhNwFLUPQRCDhUGwCATipQpWQfB13NnXMBCxOvAd-M4WqdmB1YTsAA2ZEHKvCPvO2RC4YEGCIENAAIpQI_X_bZM0neunNEaMQHBZJ8yPX_3T8udrXM6H1eOovBtVb9_Vy7BcTP3zspy_-8mnf1isrf-YldP7bbJh1LnTzT82SPfwoNs6Dtqdo5PWfjvIOKx6NDUooySGXOSMS60x5gyVkEJgKigKpZjODZhEMpqB0lHKtNECM0TDU9Ygu-vba2WNsqe9YnB1YVfBXuGKs5vbSxdRiKmklLFfeYZcmA |
| ClassificationCodes | TP311 |
| ContentType | Journal Article |
| Copyright | Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| Copyright_xml | – notice: Copyright © Wanfang Data Co. Ltd. All Rights Reserved. |
| DBID | 2B. 4A8 92I 93N PSX TCJ |
| DOI | 10.3778/j.issn.1673-9418.1411041 |
| DatabaseName | Wanfang Data Journals - Hong Kong WANFANG Data Centre Wanfang Data Journals 万方数据期刊 - 香港版 China Online Journals (COJ) China Online Journals (COJ) |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| DocumentTitle_FL | Real-Time Big Data Processing Model Based on Receiving and Processing Separation |
| EndPage | 913 |
| ExternalDocumentID | jsjkxyts201508003 |
| GrantInformation_xml | – fundername: The National Natural Science Foundation of China under Grant No.51179047 |
| GroupedDBID | 2B. 4A8 92I 93N ALMA_UNASSIGNED_HOLDINGS M~E PSX TCJ |
| ID | FETCH-LOGICAL-c613-90bf482951d7d368ee45634a78774b7047aa3edf1f9830c1ae2b3efe74c44f6b3 |
| ISSN | 1673-9418 |
| IngestDate | Thu May 29 04:00:16 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 8 |
| Keywords | 实时分析处理 load balancing 负载均衡 big data 大数据 real-time analysis and processing |
| Language | Chinese |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c613-90bf482951d7d368ee45634a78774b7047aa3edf1f9830c1ae2b3efe74c44f6b3 |
| PageCount | 8 |
| ParticipantIDs | wanfang_journals_jsjkxyts201508003 |
| PublicationCentury | 2000 |
| PublicationDate | 2015 |
| PublicationDateYYYYMMDD | 2015-01-01 |
| PublicationDate_xml | – year: 2015 text: 2015 |
| PublicationDecade | 2010 |
| PublicationTitle | 计算机科学与探索 |
| PublicationTitle_FL | Journal of Frontiers of Computer Science & Technology |
| PublicationYear | 2015 |
| Publisher | 河海大学 计算机与信息学院,南京,210098%常州工学院,江苏 常州,213002 |
| Publisher_xml | – name: 河海大学 计算机与信息学院,南京,210098%常州工学院,江苏 常州,213002 |
| SSID | ssib054421768 ssib002040941 ssib002423894 ssib051375751 ssib023646573 ssib036438069 ssib002040926 |
| Score | 1.9800471 |
| Snippet | TP311;... |
| SourceID | wanfang |
| SourceType | Aggregation Database |
| StartPage | 906 |
| Title | 接收与处理分离的实时大数据处理模型 |
| URI | https://d.wanfangdata.com.cn/periodical/jsjkxyts201508003 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV3Na9RAFA9tvXgRRcVvijinkppkJvNxnNnNUoR6WqG3kmwmSoUV7Ba0B0-lCGLxoAgeCkL9OBVPRfBv8ebu_hu-N0mzad1KFZbw8ubNb96bt9n3Jjsv8bw7ghc0j23sKxZbn3GW-7IXFD5PZUGzTNrAYr3z8n2-9IDdW4lXZmZ_NnYtbQyyxd7m1LqS__Eq8MCvWCX7D56tQYEBNPgXjuBhOJ7KxyThRCZEx0goRgwnCRwlMpOYaMgTGUkEUQGRHDlSOkIQzYkxrkk7GRBOiEocjnA4rrsWjhMTE7ix2ig2BZkTLYkOsQlApGmmvCSR2AtbhRulxGwRox0HQFxH3UatJvqXpkUoY0CTehc0ypo2iiNhHEyMqslkIgIDtB0KYMEY1HHkERFFZESMGxpMUMaJoHpTUIBQRFEHp4mhzXslZZ2o-16joAFQN5wB7URjHisLF_46H7X1QHScDExDSHSnMUMKlYGJiVqV4YjAsLtuARNW1wGYGsVufiR-EqeKSipCxyegcZwP1XFmGiI7C1MgcAAaBI27xSEXFC7CSYDbONwgXUYrFfBG4qPKouDjMZUKIV1MRcjFGhJCLGSO5UPLjj2xfG197fGz54N19AAuSOisdyYSIsQdt8svkkl2CAFENVe3eM6OlFlDOl2HC3zVAY8n2TacUhnwOhuPQyrwX8T6nDFYb5fFsIdal3v50KS7Jxnkqvf6Rdp_2Eg0u-e9c9UKcV6Xl_sFb2bz0UWPjnY-jd4e_Pq-M9zbGr_ZHr7cHn_-Mf6wNdzfHb0_GO59Gb37Nnq9X7aOvn4c7r665HU7Sbe15FfvO_F7kFT7KsgKJiNY8uQip1xaC4sbylIIqYJlImAiTanNi7BQkga9MLVRRm1hBesxVvCMXvbm-k_69oo3H4s8i7I0D8KeZVkeZRwSdSmtTMOwsEV-1btd2bha_Zytr_7htWunEbrunUW6vCl5w5sbPN2wNyFNH2S3nLN_A66ppKY |
| linkProvider | ISSN International Centre |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%E6%8E%A5%E6%94%B6%E4%B8%8E%E5%A4%84%E7%90%86%E5%88%86%E7%A6%BB%E7%9A%84%E5%AE%9E%E6%97%B6%E5%A4%A7%E6%95%B0%E6%8D%AE%E5%A4%84%E7%90%86%E6%A8%A1%E5%9E%8B&rft.jtitle=%E8%AE%A1%E7%AE%97%E6%9C%BA%E7%A7%91%E5%AD%A6%E4%B8%8E%E6%8E%A2%E7%B4%A2&rft.au=%E5%BD%AD%E5%BB%BA%E5%8D%8E&rft.au=%E6%9D%8E%E8%87%A3%E6%98%8E&rft.au=%E9%82%B1%E5%86%9B%E6%9E%97&rft.au=%E6%9D%8E%E6%99%93%E8%8A%B3&rft.date=2015&rft.pub=%E6%B2%B3%E6%B5%B7%E5%A4%A7%E5%AD%A6+%E8%AE%A1%E7%AE%97%E6%9C%BA%E4%B8%8E%E4%BF%A1%E6%81%AF%E5%AD%A6%E9%99%A2%2C%E5%8D%97%E4%BA%AC%2C210098%25%E5%B8%B8%E5%B7%9E%E5%B7%A5%E5%AD%A6%E9%99%A2%2C%E6%B1%9F%E8%8B%8F+%E5%B8%B8%E5%B7%9E%2C213002&rft.issn=1673-9418&rft.issue=8&rft.spage=906&rft.epage=913&rft_id=info:doi/10.3778%2Fj.issn.1673-9418.1411041&rft.externalDocID=jsjkxyts201508003 |
| thumbnail_s | http://utb.summon.serialssolutions.com/2.0.0/image/custom?url=http%3A%2F%2Fwww.wanfangdata.com.cn%2Fimages%2FPeriodicalImages%2Fjsjkxyts%2Fjsjkxyts.jpg |