Data Storage and Processing in the 21st Century: A Guide - Research Paper

Introduction

The 21st century is an information era; therefore, data is a crucial resource to stakeholders, especially in decision-making. Traditionally, data storage and processing involved simple data mining, data structures, and databases consisting of information with simple letters and numbers represented by small units in a single byte (Razbonyali & Guvenoglu, 2016). As technology advanced, there was also a rise in computer usage hence an increase in data. This expansion saw the data storage media exceed its analysis capacity. Therefore, new analytical tools, apart from databases and structures, were developed. Software and hardware development saw the birth of data warehouses, which meant all data, was utilized through the creation of decision-support infrastructure. The transition from traditional to big data, due to the ever-increasing data volumes pose a new set of data management challenges.

Is your time best spent reading someone else’s essay? Get a 100% original essay FROM A CERTIFIED WRITER!

Save 25%

The data warehouse prepares data for queries in or out of organizations and institutions (Razbonyali & Guvenoglu, 2016). How the data warehouse works, is such a system where data from relational databases, including the transactional system, flows smoothly for loading, processing, and consumption by business analysts, data scientists, and decision-makers using tools like spreadsheets. Srinath (2008) explains that a warehouse works with different kinds of data sources; hence, a standard representation is required. Challenges of data warehouses include time consumption in implementation, use of different technology, hence incompatibility in integration, high costs of maintenance, and limited usage due to the presence of private data (Almeida, 2017). The information processed is a lot of, hence, these challenges.

Traditional warehouses' limitations that led to the development of modern warehouses include data nature where the warehouses cannot handle unstructured and semi-structured data. Decision-making is affected since data cannot be available when needed. Performance is a derailed penalty of utilizing the same controllers. A rise in new business requirements and new infrastructure need new architectures and data management (Sebaa et al., 2017). More data is being collected in this information era, most especially with the use of smartphones and information technology management. As the number of data increases, so does the speed of accessing it - this where the concept of big data comes in. Big data is simply clusters of information, complex and intense that exceeds the processing, collection, analysis, and storage limits of software elements and data management systems with data size increasing for a terabyte to a petabyte, which is the same as 1,015 bytes. Social media sharing has intensified the collection of data to the extent that traditional structures cannot handle (Razbonyali & Guvenoglu, 2016). Both big data and traditional analytics have the same algorithms and models; however, big data analytics are characterized by a complexity that requires top-notch programming, while traditional analytics are more user-friendly.

Big data analytics have been applied in several fields like security, education, and many other sectors that have a large amount of information production. For instance, in astronomy, sky picture taking has been made easier since a database exists, and an astronomer only has to pick interesting subjects through the Sloan Digital Sky Survey. Scientific data is now accessible in a public repository for scientists use, a compliment of the big data analytics (Adiba et al., 2016). With such advancements, the world has completely transformed into a technologically advanced world. Kolajo, Ogbuju, & Adewumi (2017) explain some of the big data analytics applications. The applications include homeland security, where big data analytics can be used to identify possible threats through surveillance of people considered suspicious. Big data helps to analyze their itineraries, their location, their social media, mobile, and email communications, and their financial dealings. Another sector that has applied big data analytics is traffic control, which uses satellite information in analyzing road conditions, hence better time estimation and better route plans. The manufacturing sector can now monitor the delivery and quality of products in a bid to scale up customer service. Education has not been left behind as tracking progress, and analysis is now easier. Better education is now achievable through the collaborative library.

Big data management has no doubt brought beautiful and tremendous transformations o the world. However, challenges still stand on its way. Some of the problems associated with big data management include attaining cost-effectiveness in executing large clusters of data processing and frequent system failures due to large clusters of information, as pointed out by (Adiba et al., 2016). Data streaming is in large quantities due to numerous users accessing at the same time, and users expect faster processing as well, hence system malfunction is inevitable. Other challenges of big data as explained by Almeida & Calistru (2013) include lack of sufficient privacy in terms of retention, presentation, storage, use and acquisition, reluctance in information access and sharing due to legal considerations and difficulty in access and transfer of data due to technical and institutional issues, for instance, how data is stored.

Hadoop was developed as a solution to some of the big data management solutions. Big data analytics do not guarantee partition tolerance, consistency, and availability, hence systems like Hadoop came in to place. Awadallah (2011) explains that data grows, it consumes a lot of disk space, and the traditional databases could not support archiving. Hadoop provides an archiving solution that kept data alive, no matter how old. Data set for analysis is stored in HDFS, which stands for Hadoop File System. Hadoop is developed in a manner that makes sure that jobs and data are distributed as it processes on data bunches from other computers. Hadoop is widely accepted for the calculation of big data analytics in a medium that is easily accessible. Low-cost, unstructured data storage, analysis, and panel programming distribution also characterize Hadoop. These features make it preferable by Yahoo, Google, and Facebook, who are the forerunners in Big Data. Initially, Hadoop could not analyze data in real-time; thus, it introduced Apache Spark, which is based on flexible data distribution, providing results in less than half a second (Razbonyali & Guvenoglu, 2016).

Quantum computing, as Singh (2015) puts it has an association with big data analytics. Though not commercialized yet, developments are already underway. They are expected to be a game-changer in all aspects of life, for instance, control systems, instrumentation, mobile, and hand-held devices in terms of speed error management. He also points out to a movement of making analytics more-user friendly. Another emerging trend is the In-memory processing, which allows faster processing for specific data, for instance, SAP's HANA (High-Performance Analytic Appliance), which is column-oriented. With such innovations, the future seems brighter.

Conclusion

Data management has evolved from simple to complex processes. As time goes by, people are more inclined to technology for everything. The more technology is used, the more data flows in systems. Traditional analytics of data management was cast aside for big data management because data volumes keep increasing at an alarming rate. Big data management has made some major transformations in sectors such as education, national security, and astronomy. Although it has made major progress, the increase in the flow of data remains to be a source of difficulties for data management. Challenges like attaining cost-efficiency and frequent system failures are as a result of the increment in the data flow. For effective data management, it is crucial to come up with solutions that can handle large entries of data without straining the systems.

References

Kolajo, T., Ogbuju, E., & Adewumi, S. E. (2017). Trends and technologies in big data analytics: a review. Confluence Journal of Pure and Applied Sciences (CJPAS), 1(1), 1-278. https://www.researchgate.net/publication/328531358_TRENDS_AND_TECHNOLOGIES_IN_BIG_DATA_ANALYTICS_A_REVIEW

Sebaa, A., Chikh, F., Nouicer, A., & Tari, A. (2017). Research in big data warehousing using Hadoop. Journal of Information Systems Engineering & Management, 2(2), 1-5. https://www.researchgate.net/publication/316340915_Research_in_Big_Data_Warehousing_using_Hadoop

Adiba, M., Castrejon-Castillo, J.-C., Oviedo, J. A., Solar, G. V., & Zechinell, J.-L. (2016). Big data management challenges, approaches, tools, and their limitations.

Almeida, F. (2017). Concepts and fundaments of data warehousing and OLAP. ISSUU Publishing.

Almeida, F., & Calistru, C. (2013). The main challenges and issues of big data management. International Journal of Research Studies in Computing, 2(1), 11-20. https://www.researchgate.net/publication/272696610_The_main_challenges_and_issues_of_big_data_management

Awadallah, A. [Stanford]. (2011, November 16). Introducing Apache Hadoop: the modern data operating system. Retrieved from https://www.youtube.com/watch?v=d2xeNpfzsYI

Razbonyali, C., & Guvenoglu, E. (2016). Traditional data storage methods and big data concepts. International Research Journal of Engineering and Technology (IRJET), 3(6), 2556-2561. https://www.researchgate.net/publication/331062772_Traditional_Data_Storage_Methods_and_the_Big_Data_Concepts

Singh, N. (2015). Emerging Trends in technologies for big data. The International Technology Management Review, 5(4), 202-210. https://www.researchgate.net/publication/299497214_Emerging_Trends_in_Technologies_for_Big_Data

Srinath, S. (2008). [nptelhrd]. (n.d). Lecture 30: Introduction to data warehousing and OLAP. Indian Institute of Technology Madras. Retrieved from https://www.youtube.com/watch?v=m-aKj5ovDfg