Best Practices for Data Retention

Incorporating data retention best practices as part of an organization’s overall data management processes is a key strategy to wrapping sensitive data with layers of protection and intent throughout its entire lifecycle. Applying best practices around retention and purging data can increase an organization’s overall security posture and strengthen defense against cybercrime or inadvertent mishandling that could lead to a data breach.

Implementing MFT is a Baseline Data Protection Tactic

Installing a managed file transfer (MFT) solution as the standard when exchanging data puts organizations miles ahead of businesses still using standard FTP (File Transfer Protocol) or email to transfer files. An MFT platform offers a way to centralize data transfer and more importantly, to apply policies to data to protect it from threats such as inbound malware or from employees sending files to platforms beyond organizational control such as Google Drive or Dropbox.

MFT has the built-in advantage of “forcing” the application of policies designed to better protect data such as encryption, monitoring, and auditing. However, the mountains of data now accessible, while being protected in transit, can still be ultimately exposed should the MFT platform itself be targeted.

Therefore, proactively limiting the amount of data that could be potentially exposed and at risk, should that platform be targeted, needs to be a high priority.

Concerns Around Volume of Data Assets

It’s been said that data is “the new oil” in a nod to the value it holds in the right hands (and the value it can fetch in the wrong ones). Therefore, data needs to be protected as the sensitive, valuable commodity that it is. Limiting outsider threats such as malware, ransomware, accidental data loss, third-party exposure, and zero-day attacks, as well as identifying and closing security gaps, and meeting compliance requirements are just the first, although vital, steps. Limiting the vast amount of data retained and potentially at risk should be a close second.

According to Edge Delta, users generated 64.2 Zettabytes (ZB) of data in 2020, which exceeded the number of detectable stars in the cosmos. Experts expect this figure to rise, with data creation reaching 147 ZB by the end of 2024.

One analogy I like to use when talking about data risk compares bank asset storage to data storage. We no longer see all money stored in a single bank vault. Instead, that asset is now decentralized and surrounded by multiple security measures. Likewise, data should be decentralized and protected with layers of security, even around its retention.

Banks improved their asset protection by keeping only the minimum necessary cash on hand in the vault; requiring dual control access such as cameras and security guards and applying multiple layers of security prior to accessing the money they manage.

As repositories of data, organizations need to pose questions about the data being held on to for months, years, or even decades, such as:

• Is the data still relevant and necessary? 
• Does the data contain sensitive information?
• How sensitive is that data?
• Are there policies in place to classify the data and help make informed decisions around how long that data should live? 

How Long Does Data Need to Be in an Archive Directory?

It’s common for organizations using MFT to automatically archive that data indefinitely after it has been sent and staged for use downstream. However, after data has served its purpose, “forever” is probably not the answer to the question of how long to hang onto it in a directory, subfolder, or trading partner’s folder.

Without good retention and deletion practices, more data than necessary is at risk, should the MFT platform be breached. Not all data needs to be in the hands of the MFT platform for all time, especially if it has already served its lifecycle.

Purge Data with Regularity and Purpose

To quote decluttering guru, Marie Kondo, “Does this spark joy? If it does, keep it. If not, dispose of it.” When it comes to retaining sensitive data, the question should be, “Has this data outlived its lifecycle and by sitting here be at risk? If the answer is yes, it’s time to purge it to reduce exposure. 

Having a “vault” full of old data associated with the server handling day-to-day business processes can be risky business should that server get breached. Therefore, purging data with a regular cadence is important if you don't need it for business processing.  And, just like purging a house full of clutter starts with sorting everything into distinct categories, grouping data to determine what classes of data are retained and for what term is a critical first step. 

Data Classification Simplifies Secure Data Retention and Purging

A content engine can classify data based on its sensitivity, criticality, and compliance requirements. By segregating data into different tiers, such as sensitive, confidential, highly classified, or public, the appropriate retention rules can be automatically applied to each category to better prioritize protection and retention based on the data’s value and risk.

Layered Security Can Help Mitigate Retention Risks 

While secure file transfer platforms encrypt data at rest and in transit, establish audit logs, and separate and apply multiple security controls, organizations can take protection a step further with a layered security approach.

MFT solutions with web application firewalls can prevent things like SQL injection, cross-site forgery requests, and distributed denial of service (DDOS), for an additional layer of protection in the MFT space, where the exposure plane has primarily been via HTTPS. 

In addition, an antivirus engine can help ensure content entering an organization’s environment does not contain threats. And a Data Loss Prevention (DLP) solution leveraging a Secure ICAP Gateway can help ensure data going out of the organization is protected by surrounding data that falls under questions such as:

• Is it sensitive?
• Do we need to mask it? 
• Do we need to redact it?
• Do we need to apply digital rights management to hypersensitive information to better control what happens to data sent to a third party?

Best Practices Around Data Retention

Yes. Some data does need to hang around for some time, such as needing to retain audit logs and files for two years because of industry requirements. However, applying multiple layers of control to it as you bide your time until its purged reduces that risk exposure. Letting data sit in an archive folder underneath the normal landing zone for inbound file transfers is akin to having a bank vault next to a defenseless teller should either locale be targeted.

Managing all that data is easier with best practices in place, such as:

Establishing a Data Retention Policy

- Meet with key stakeholders

- Establish a well-defined policy that aligns with legal and/or compliance requirements, industry standards, and organizational needs.

Optimizing Data Retention

- Clearly outline the types of data to be retained

- Define retention periods

- Establish disposal procedures 

Adopting a Data Classification Scheme. 

- Install a data classification tool to assign priorities around the types and sensitivity levels of the data you retain.

Creating Role-based Access Controls to Data Storage Locations

- Grant least privileged access

- Utilize disk-level encryption

- Require multi-factor authentication

Encrypting Business-Critical Data

- Add additional object encryption mechanisms such as PGP, GPG

- Utilize a third-party encryption tool where encryption and decryption processes lie outside of the MFT platform. A DRM or secure collaboration tool can deliver even greater control, allowing organizations to apply end-to-end security to data, revoke access to documents at any time, no matter where files land, determine how long data can be viewed and accessed, and more.

Automating the Retention Management Process.

- Implement retention schedules to trigger automatic deletion or archival based on predefined criteria.

- Retain detailed audit records to meet data retention and compliance policies.

How MFT Solutions Help Streamline Data Retention

By using advanced workflow functionality, such as that in Fortra’s GoAnywhere MFT, the workflow or automation engine can be pre-built to shoulder some of the decision-making based on specific requirements or triggers from a trading partner. For example, a policy can be set to keep selected data for a year, for compliance audit log-keeping purposes.

MFT automation can also create schedules and file and folder monitors to respond to triggers to help with periodic checks. With a classification tool applied, employees could check to see if data in a folder is older than a year and then notify the data’s owner about the retention policy before purging.

Secure Archiving and Removal Considerations 

Moving data to a longer-term storage location could be done with a separate device with PGP encryption applied or with digital rights management applied to the data so that if that data is compromised, there’s an additional layer of encryption protecting it.

We recommend secure deletion, an operating system-level manner of fully deleting a file, versus the “regular” delete process, which can be technically recovered.  And, depending on the sensitivity of your data, you can take that secure removal even further.

After your audit logs have run their course, or your data retention classification policy calls for removal after a set period, you “could” automatically remove or delete it. However, if there is data that may still be leveraged, you might need to determine who needs to approve the removal of this data using a workflow or ticket to notify an individual or a group of users, that a data file is ready to be retired to make an informed decision on whether or not the data is still truly needed.  

For the highest level of data removal, physical destruction of the disks data is written on is also an option.

Secure Archiving

Secure archiving is the “vault” or long-term data storage mechanism. Before anything goes into long-term storage, it is important to scan data for threats, breaking that data into various tiers of security:

• Base-level archiving: Simply move data to a write-only location. If someone gets into the MFT platform unauthorized, they're not able to read anything in there. It is one-way archiving.
• Next-level object encryption archiving: Putting PGP encryption on the data before sending it to the archive directory adds more protection. Even better - moving decryption keys outside of the MFT tool to make it harder to use any data if someone somehow gets to it.
• Best in class-level secure archiving: Using the key management feature in GoAnywhere MFT adds even more protection, as the solution has a built-in key vault and decryption capabilities when using advanced workflows. 

Layered Security is the Best Security for Data Retention

Combining MFT with a Secure ICAP Gateway is one layered combination to ensure threats do not enter the MFT environment. Data is scanned as it comes in and is automatically rejected if malware is detected in a file being uploaded.

Another layering technique is taking the extra step to classify data. Take one more additional step and PGP-encrypt that data. The retention policies organizations choose ultimately depend on the sensitivity of the data to be protected. MFT, in combination with other data protection tools, can make moving and retaining data more secure and streamlined.