BigBird: Big Data Storage and Analytics at Scale in Hybrid Cloud

Implementing big data storage at scale is a complex and arduous task that requires an advanced infrastructure. With the rise of public cloud computing, various big data management services can be readily leveraged. As a critical part of Twitter's "Project Partly Cloudy", the cold stor...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Deochake, Saurabh, Channapattan, Vrushali, Steelman, Gary
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 22.03.2022
Subjects
Big Data
Cloud computing
Cold storage
Data analysis
Data management
Data storage
Management services
Mathematical analysis
Online AccessGet full text

Cover

Abstract Implementing big data storage at scale is a complex and arduous task that requires an advanced infrastructure. With the rise of public cloud computing, various big data management services can be readily leveraged. As a critical part of Twitter's "Project Partly Cloudy", the cold storage data and analytics systems are being moved to the public cloud. This paper showcases our approach in designing a scalable big data storage and analytics management framework using BigQuery in Google Cloud Platform while ensuring security, privacy, and data protection. The paper also discusses the limitations on the public cloud resources and how they can be effectively overcome when designing a big data storage and analytics solution at scale. Although the paper discusses the framework implementation in Google Cloud Platform, it can easily be applied to all major cloud providers.
AbstractList Implementing big data storage at scale is a complex and arduous task that requires an advanced infrastructure. With the rise of public cloud computing, various big data management services can be readily leveraged. As a critical part of Twitter's "Project Partly Cloudy", the cold storage data and analytics systems are being moved to the public cloud. This paper showcases our approach in designing a scalable big data storage and analytics management framework using BigQuery in Google Cloud Platform while ensuring security, privacy, and data protection. The paper also discusses the limitations on the public cloud resources and how they can be effectively overcome when designing a big data storage and analytics solution at scale. Although the paper discusses the framework implementation in Google Cloud Platform, it can easily be applied to all major cloud providers.
Author Channapattan, Vrushali
Steelman, Gary
Deochake, Saurabh
Author_xml – sequence: 1
  givenname: Saurabh
  surname: Deochake
  fullname: Deochake, Saurabh
– sequence: 2
  givenname: Vrushali
  surname: Channapattan
  fullname: Channapattan, Vrushali
– sequence: 3
  givenname: Gary
  surname: Steelman
  fullname: Steelman, Gary
BookMark eNqNjL0KwjAYAIMoWLXv8IFzISb9w0lbFfe6l88mlpSQaJIOfXs7-ABOd8NxG7I01sgFiRjnh6RMGVuT2PuBUsrygmUZj8ipUn2lnDjCLHDBgNAE67CXgEbA2aCeguo8YICmQy1BGbhPT6cE1NqOYkdWL9Rexj9uyf52fdT35O3sZ5Q-tIMd3bzxLctTRkualwX_r_oCn345Dw
ContentType Paper
Copyright 2022. This work is published under http://creativecommons.org/licenses/by-nc-sa/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml – notice: 2022. This work is published under http://creativecommons.org/licenses/by-nc-sa/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID 8FE
8FG
ABJCF
ABUWG
AFKRA
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
HCIFZ
L6V
M7S
PIMPY
PQEST
PQQKQ
PQUKI
PRINS
PTHSS
DatabaseName ProQuest SciTech Collection
ProQuest Technology Collection
Materials Science & Engineering Collection
ProQuest Central (Alumni)
ProQuest Central
ProQuest Central Essentials
ProQuest Central
Technology Collection
ProQuest One Community College
ProQuest Central Korea
SciTech Premium Collection
ProQuest Engineering Collection
Engineering Database
Publicly Available Content Database
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Academic
ProQuest One Academic UKI Edition
ProQuest Central China
Engineering Collection
DatabaseTitle Publicly Available Content Database
Engineering Database
Technology Collection
ProQuest Central Essentials
ProQuest One Academic Eastern Edition
ProQuest Central (Alumni Edition)
SciTech Premium Collection
ProQuest One Community College
ProQuest Technology Collection
ProQuest SciTech Collection
ProQuest Central China
ProQuest Central
ProQuest Engineering Collection
ProQuest One Academic UKI Edition
ProQuest Central Korea
Materials Science & Engineering Collection
ProQuest One Academic
Engineering Collection
DatabaseTitleList Publicly Available Content Database
Database_xml – sequence: 1
  dbid: 8FG
  name: ProQuest Technology Collection
  url: https://search.proquest.com/technologycollection1
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Physics
EISSN 2331-8422
Genre Working Paper/Pre-Print
GroupedDBID 8FE
8FG
ABJCF
ABUWG
AFKRA
ALMA_UNASSIGNED_HOLDINGS
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
FRJ
HCIFZ
L6V
M7S
M~E
PIMPY
PQEST
PQQKQ
PQUKI
PRINS
PTHSS
ID FETCH-proquest_journals_26420806873
IEDL.DBID BENPR
IngestDate Thu Oct 10 15:59:16 EDT 2024
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-proquest_journals_26420806873
OpenAccessLink https://www.proquest.com/docview/2642080687?pq-origsite=%requestingapplication%
PQID 2642080687
PQPubID 2050157
ParticipantIDs proquest_journals_2642080687
PublicationCentury 2000
PublicationDate 20220322
PublicationDateYYYYMMDD 2022-03-22
PublicationDate_xml – month: 03
  year: 2022
  text: 20220322
  day: 22
PublicationDecade 2020
PublicationPlace Ithaca
PublicationPlace_xml – name: Ithaca
PublicationTitle arXiv.org
PublicationYear 2022
Publisher Cornell University Library, arXiv.org
Publisher_xml – name: Cornell University Library, arXiv.org
SSID ssj0002672553
Score 3.383723
SecondaryResourceType preprint
Snippet Implementing big data storage at scale is a complex and arduous task that requires an advanced infrastructure. With the rise of public cloud computing, various...
SourceID proquest
SourceType Aggregation Database
SubjectTerms Big Data
Cloud computing
Cold storage
Data analysis
Data management
Data storage
Management services
Mathematical analysis
Title BigBird: Big Data Storage and Analytics at Scale in Hybrid Cloud
URI https://www.proquest.com/docview/2642080687
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3dS8MwED9ci-Cbn_gxR0Bfg236scYXpbO1CBvDKextpE0ig7HNtnvwxb_dS-n0QdhbQiAhx-Xud5f7ALhFFKo94ee00JGkfqQ05TLn1NHIUajuNecmOXk4CrN3_2UaTFuHW9WGVW5lYiOo5aowPvI7VNwM0U0Y9R_Wn9R0jTK_q20LjQ7YDC0FxwI7Tkbj118vCwv7iJm9f4K20R7pIdhjsVblEeyp5THsN0GXRXUCj_H8I56X8p7ggDyJWpAJmsD4wgna96SpF2KqKBNRkwnSUpH5kmRfJseKDBarjTyFmzR5G2R0e-ysZY1q9ncR7wwstPHVORDXVUFQIBpRCHA8pXngSFUEOaIDX7iCX0B3106Xu5ev4ICZqH3Ho4x1warLjbpGXVrnPehE6XOvJRvOht_JD9T-fR0
link.rule.ids 783,787,12777,21400,33385,33756,43612,43817
linkProvider ProQuest
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3NS8MwFH_ohujNT_yYGtBrsG3SrvGibFqrbkPYhN1K2iRSkG223cH_3pfS6UHYLRBIyCN5v997eR8A18hCDZM8pZkJFeWhNlSoVFDH4I1CuDdC2OTk4SiI3_nL1J82DreyCatc6cRaUat5Zn3kNwjcHrKbIOzeLb6o7Rplf1ebFhqb0OYMsdpmikdPvz4WL-giY2b_1GyNHdEutN_kQhd7sKFn-7BVh1xm5QHc9_KPXl6oW4ID8iArScZoAOP7Jmjdk7paiK2hTGRFxihJTfIZib9thhXpf86X6hCuosdJP6arbZPmYpTJ3zHYEbTQwtfHQFxX-36GXEQjvWHaCN9ROvNT5AZculKcQGfdSqfrpy9hO54MB8ngefR6Bjuejd93GPW8DrSqYqnPEVWr9KIW3Q_TPnyR
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BigBird%3A+Big+Data+Storage+and+Analytics+at+Scale+in+Hybrid+Cloud&rft.jtitle=arXiv.org&rft.au=Deochake%2C+Saurabh&rft.au=Channapattan%2C+Vrushali&rft.au=Steelman%2C+Gary&rft.date=2022-03-22&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422