Skip to main content

Cloud Confidentiality

Today, no CSP can guarantee that your data will be secured “For Your Eyes Only.” Encryption algorithms and compliance policies can only achieve so much.

From the CSP perspective, we have to take reasonable measures to ensure customer data is not used in any way by the provider that is unintended by the customer.As a way to mitigate exposure of customer data, some CSPs encrypt data at rest using encrypted hard drives or encrypted file systems. The other part of the equation for risk mitigation is proper device destruction either logically, using an appropriate method of sanitization likeDoD 5220.22-M or physically DSS Clearing and Sanitization Matrix and Special Publication 800-88: Guidelines for Media Sanitization.

And then, there are the backups. CSPs manage several copies of data to prevent total failure in both onsite and offsite facilities, and more than likely, the data stored on tape or other hdds is encrypted.

Once you have encrypted drives and encrypted backups, you have to deal with a little problem called key management. This turns out to be a problem because some CSPs use only a handful (if that many) of encryption keys that are known to a select few of the system administrators in charge of ops and compliance. That begs the question: If a system administrator leaves (on their own free will or not), are the keys changed? More than likely, yes. But what about the backups? How long is the time window during which the now departed administrator can access customer data? I usually hear crickets when I ask this question of CSPs - even those with plenty of acronyms and certifications behind their name.

From the customer perspective, I don't want any of my data readable by anyone except me. If my CSP encrypts my data when it's at rest, I want to make sure no one can access it in other ways, such as from a backup my CSP performs regularly. What this means to me is, before data leaves my control and into my CSP, all data must be encrypted using standard encryption algorithms and key(s) that I manage.

For eg:- S3 is a HDFS-like entity made up of NameNodes and DataNodes.

Currently, Amazon offers Service Side Encryption (SSE) for S3. When data is put into S3, a process blocks your data (into X MB chunks) where each block of data is encrypted by a key chosen from this process. Every file has a separate key and after T amount of time, the file is re-encrypted with a new key.

Except that we don’t know where the keys are stored. Are there backups of the keys for disaster recovery and high availability purposes (think of HDFS NameNodes)? We are back in the same conundrum where we started with encrypted backups.


What Can Be Done Today


Two options based on their tolerance for risk:
  • Leave data unencrypted at rest and trust the CSP
  • Encrypt all data before it is sent to the CSP
In this post, I wanted to bring out the issues with confidential customer data in the cloud, as this is not a solved problem. Encryption is only meant to provide some assurance that no one other than key holders can view the data.

Comments

Popular posts from this blog

Python and Parquet Performance

In Pandas, PyArrow, fastparquet, AWS Data Wrangler, PySpark and Dask. This post outlines how to use all common Python libraries to read and write Parquet format while taking advantage of  columnar storage ,  columnar compression  and  data partitioning . Used together, these three optimizations can dramatically accelerate I/O for your Python applications compared to CSV, JSON, HDF or other row-based formats. Parquet makes applications possible that are simply impossible using a text format like JSON or CSV. Introduction I have recently gotten more familiar with how to work with  Parquet  datasets across the six major tools used to read and write from Parquet in the Python ecosystem:  Pandas ,  PyArrow ,  fastparquet ,  AWS Data Wrangler ,  PySpark  and  Dask . My work of late in algorithmic trading involves switching between these tools a lot and as I said I often mix up the APIs. I use Pandas and PyArrow for in-RAM comput...

Ingesting IoT Sensor Data Into S3 With an RPI3

StreamSets Data Collector Edge is a lightweight agent used to create end-to-end data flow pipelines. We'll use it help stream data collected from a sensor. Due to the increasing amount of data produced from outside source systems, enterprises are facing difficulties in reading, collecting, and ingesting data into a desired, central database system. An edge pipeline runs on an edge device with limited resources, receives data from another pipeline or reads the data from the device, and controls the device based on the data. StreamSets Data Collector (SDC) Edge, an ultra-lightweight agent, is used to create end-to-end data flow pipelines in StreamSets Data Collector and to run the pipelines to read and export data in and out of systems. In this blog, StreamSets Data Collector Edge is used to read data from an air pressure sensor (BMP180) from an IoT device (Raspberry Pi3). Meanwhile, StreamSets Data Collector is used to load the data into Amazon's Simple Storage Service ...

Real-Time Talk: Windows 10 IoT Core Background Tasks and ASP.NET Core Web Apps

Display useful information from your Windows 10 IoT Core application in an ASP.NET Core web app, essential for integrating IoT data into a solution. Windows 10 IoT background task talk with a web application using WebSockets. Problems As my path to this solution has been troublesome, I am listing here the main problems I faced so my dear readers have a better idea of dead-end streets along the way: I was not able to make the ASP.NET Core web application run under a Windows 10 IoT background service. I found no information about when or if it will be supported in the near future. ASP.NET MVC and ASP.NET Core have different SignalR implementations. I was not able to make a SignalR client for .NET Core work with SignalR hosted on a web application. I was able to make things work by directly using a WebSocket. It’s not as nice a solution as I had in my mind, but it works until things get better. Making the Background Task and Web Application Talk I worked out sim...