
Measures for Managing Unstructured Data
CIO Review APAC | Monday, December 12, 2022

Data is predicted to more than double again over the next few years, reaching 175 zettabytes in 2025.
FREMONT, CA: More data has been generated globally in the last few years than in the previous two decades. According to IDC, data is expected to more than quadruple once more during the following few years, hitting 175 zettabytes in 2025.
Documents, videos, photos, data from instruments and sensors, text and chats, and other types of data make up most of this unstructured data. As unstructured data is scattered across numerous applications and storage sites inside and outside the company rather than existing in rows and columns in a database, it is more difficult to locate, transport, and manage.
Stay ahead of the industry with exclusive feature stories on the top companies, expert insights and the latest news delivered straight to your inbox. Subscribe today.
Enterprise IT departments and data storage specialists face various new issues due to the 1data explosion and data type diversity. These include rising storage and backup costs, complex management, security issues, and a visibility gap that limits opportunities.
Businesses require new, clever measures and analytics to address these problems.
These need to move beyond traditional storage measures and emphasise comprehending data and incorporating application owners, department stakeholders, and other business stakeholders in data management decisions. These metrics should also track and improve energy use to achieve more general sustainability goals, which are crucial in the modern era of cyclical energy shortages and climate change.
The traditional IT infrastructure measurements mentioned above are prerequisites for any enterprise IT organisation. There are many new data-centric indicators to comprehend and report on in the environment when data drives all choices. Leaders are now frequently responsible for monitoring and paying for their data usage. When business leaders are uncomfortable archiving or deleting their data, discussions with IT companies can get tense. This is especially true when IT attempts to save money and free up capacity. These measurements aid in closing the gap:
Top users and owners of data: This can reveal patterns in usage and point out any instances of policy violations, such as when a user stores an excessive amount of video files or PII files are kept in an improper place.
Common File Types: A research team gathering data from certain apps or equipment might not be aware of the quantity or location of all the data it has. Having access to data by file extension can help guide upcoming research projects. Finding all the log files, trace files, or extracts from a specific application or instrument and acting on them could suffice.
Storage Costs for Chargeback or Showback: Stakeholders should understand expenses and be able to dig into metrics even if a department doesn't participate in a chargeback model. As a result, they can pinpoint places where low-cost storage or data tiering to archival storage can be used to cut costs.
Data Growth Rates: Overall growing data keeps IT and business leaders in sync so they can work together on new approaches to manage skyrocketing data volumes. Stakeholders can see which projects and organisations are generating data faster and ensure that data storage and development are acceptable for the overall business goal.
Data Age and Access Trends: Most businesses have a sizable amount of cold data that hasn't been accessed in at least a year. To make sure that data is living in the appropriate location at the appropriate time following its business value, metrics showing the percentage of cold, warm, and hot data are essential.
IT and departments can collaborate to make better decisions when they have visibility into data-centric versus storage-centric metrics. Due to the ubiquity of data silos in businesses and the fact that data is dispersed across numerous applications and storage settings, including on-premises, at the edge, and in the cloud, these metrics have previously been challenging to collect.
Finding and indexing data across vendor boundaries, including cloud providers, utilising a single pane of glass is necessary to obtain this data. It is feasible to compile data from all of your storage providers to obtain these metrics, but it is laborious and prone to mistakes when done manually. These larger and deeper analytics objectives can be attained with independent data management systems.
Corporate sustainability initiatives and investments in new green technology are being fueled by the global energy crisis, which has been made worse by the conflict in Ukraine and the rise in demand from the post-pandemic economic rebound. Responsible data management is a significant component of this whole project. The majority of businesses have hundreds of gigabytes of data that can be destroyed but is either hidden or isn't understood well enough to be managed properly.
To slow down climate change, data centres must lower their carbon footprints. The data management metrics for sustainability listed below can assist in measuring and lowering energy use related to data storage.
Data access and age metrics can help with decisions about shifting data to a lower-carbon storage place, like cloud object storage. Last access time and creation time:
Reduced Duplicate Data: The storage footprint and energy consumption naturally decrease when unnecessary data is deleted. Datasets are frequently duplicated for various studies and testing but never removed, especially in research institutions.
Data Kept by the Vendor: As a whole, legacy storage technologies like RAID, SAN, and tape are more inefficient, which is why SSD and all-flash storage have seen rapid growth. Modern storage methods consume less electricity since they are faster and more effective than spinning discs. Understanding the amount of data kept on older systems is a good place to start when deciding how and when to switch to more advanced technologies, including cloud storage.
Usability is a gauge of how much work is involved in a task. Any technology that is more manageable and effective is greener. It has greater features for automation and needs less labour and data centre resources. For instance, compared to 8PB or less utilising earlier technologies, one storage architect may handle 50PB of data and higher today.
More in News