Big Data and Cloud Computing

Amazon Web Services (AWS) has launched a web identity service with support for Google, Facebook and its own AWS Identity and Access Management (IAM).  The service allows developers to grant temporary authorization to people using these three services and simplifies development as all the identity management is done by AWS. All the server-side code is managed without long-term credentials for the app. The service introduces a new AWS Security Token Service (STS) API that allows for temporary security credentials for customers who have been authenticated by Amazon.com, Facebook, or Google. The “app can then use the temporary security credentials to access AWS resources such as Amazon Simple Storage Service (S3) objects, DynamoDB tables, or Amazon Simple Queue Service queues.” This means that an app developer can more easily integrate identity features into an app. AWS uses the example of allowing end users to upload an image file as their personal avatar. In this case, a ...

Amazon Elastic Transcoder with an initial set of features and a promise to iterate quickly based on customer feedback. You've supplied us with plenty of feedback (primarily via the Elastic Transcoder Forum) and have a set of powerful enhancements ready as a result. Here's what's new: Apple HTTP Live Streaming (HLS) Support . Amazon Elastic Transcoder can create HLS-compliant pre-segmented files and playlists for delivery to compatible players on iOS and Android devices, set-top boxes and web browsers. You can use our new system-defined HLS presets to transcode an input file into adaptive-bitrate filesets for targeting multiple devices, resolutions and bitrates.  You can also create your own presets. WebM Output Support . Amazon Elastic Transcoder can now transcode content into VP8 video and Vorbis audio, for playback in browsers, like Firefox, that do not natively support H.264 and AAC. MPEG2-TS Output Container Support . Amazon Elastic Transcoder can...

Amazon RDS monitors the replication status of your Read Replicas and updates the  Replication State  field to  Error  if replication stops for any reason (e.g., running DML queries on your replica that conflict with the updates made on the master database instance could result in a replication error). You can review the details of the associated error thrown by the MySQL engine by viewing the  Replication Error  field and take an appropriate action to recover from it.  If a replication error is fixed, the  Replication State  changes to  Replicating . Amazon RDS Event Notifications  automatically get notified when you encounter a replication error. Separately, you can also  monitor  the  Replication Lag  metric and set up a CloudWatch alarm to receive a notification when the lag crosses a particular threshold tolerable by your application.

A bug that steals cash by racking up charges from sending premium rate text messages has been found in Google Play.  Security researchers have identified 32 apps on Google Play that harbour the bug called BadNews. A security firm Lookout, which uncovered BadNews, said that the malicious program lays dormant on handsets for weeks to escape detection.  The malware targeted Android owners in Russia, Ukraine, Belarus and other countries in eastern Europe. 32 apps were available through four separate developer accounts on Google Play. Google has now suspended those accounts and it has pulled all the affected apps from Google Play, it added. Half of the 32 apps seeded with BadNews are Russian and the version of AlphaSMS it installed is tuned to use premium rate numbers in Russia, Ukraine, Belarus, Armenia and Kazakhstan.

Testing sessions in full swing It was a full day long session and consisted of two groups of 25 youngster 1. Too much guidence When you are moderating a testing session, try to talk about the game or app as little as possible. It is perfectly ok to be mute and not give the player any background information about the game at all.  Let them figure it out themselves. Players need to understand the game mechanics from the moment they install and run it for the first time ever. If they don´t then you have some work to do. 2. Assuming too much Don´t assume that the player always understands your in-game menu. Before testing the game itself, try to get the test subjects to speak about the menus and items in the game. Do the players understand what each setting and button does? How do they think they can move around the different menu items? During the sessions I witnessed teams skipping past the start screen and also the menu options, which is a bad move. You might hav...

Decompose the Application by Workload Applications are typically composed of multiple workloads. Different workloads can, and often do, have different requirements, different levels of criticality to the business, and different levels of financial consideration associated with them. By decomposing an application into workloads, an organization provides itself with valuable flexibility. A workload-centric approach provides better controls over costs, more flexibility in choosing technologies best suited to the workload, workload specific approaches to availability and security, flexibility and agility in adding and deploying new capabilities, etc. Scenarios When thinking about resiliency, it’s sometimes helpful to do so in the context of scenarios. The following are examples of typical scenarios: Scenario 1 – Sports Data Service  A customer provides a data service that provides sports information. The service has two primary workloads. The first provides statistics for th...

Cloud computing is distributed computing; distributing computing requires thoughtful planning and delivery – regardless of the platform choice. The purpose of this document is to provide thoughtful guidance based on real-world customer scenarios for building scalable applications Fail-safe   noun . Something designed to work or function automatically to prevent breakdown of a mechanism, system, or the like. Individuals - whether in the context of employee, citizen, or consumer – demand instant access to application, compute and data services. The number of people connected and the devices they use to connect to these services are ever growing. In this world of always-on services, the systems that support them must be designed to be both available and resilient. The Fail-Safe initiative  is intended to deliver general guidance for building resilient cloud architectures, guidance for implementing those architectures  and recipes for implementing these architectures...

Federations in SQL Database are a way to achieve greater scalability and performance from the database tier of your application through horizontal partitioning. One or more tables within a database are split by row and portioned across multiple databases (Federation members). This type of horizontal partitioning is often referred to as ‘sharding’. The primary scenarios in which this is useful are where you need to achieve scale, performance, or to manage capacity. SQL Database can deliver scale, performance, and additional capacity through federation, and can do so dynamically with no downtime; client applications can continue accessing data during repartitioning operations with no interruption in service. Federation Architecture A  federation  is a collection of database partitions that are defined by a federation distribution scheme, known as the  federation scheme . The federation scheme defines a  federation distribution key , which determines the dis...

I get strange looks when I talk to developers about the difference between developing an application to a product versus developing an application to a service.  The application you write on premise is written to a piece of software purchased, installed and configured on a piece of computer hardware that you privately own.  The application you write in the cloud is written to a set of services that are available to you as well as the public to exploit.  So let’s explore how they are different. When you write your application for your on-premises server there are expectations that you have from this set of hardware and software.  You can expect that you will connect successfully every time you logon.  You control what applications run on this server so you expect the same level of performance every time.  You know and control the security context of your data.  You configured the software and hardware so you expect that if something goes wrong all y...

A bug that steals cash by racking up charges from sending premium rate text messages has been found in Google Play.  Security researchers have identified 32 apps on Google Play that harbour the bug called BadNews. A security firm Lookout, which uncovered BadNews, said that the malicious program lays dormant on handsets for weeks to escape detection.  The malware targeted Android owners in Russia, Ukraine, Belarus and other countries in eastern Europe. 32 apps were available through four separate developer accounts on Google Play. Google has now suspended those accounts and it has pulled all the affected apps from Google Play, it added. Half of the 32 apps seeded with BadNews are Russian and the version of AlphaSMS it installed is tuned to use premium rate numbers in Russia, Ukraine, Belarus, Armenia and Kazakhstan.

 While standardization on the construct of one or more "VM" resource pools for the server resource requirements is necessary and an important step, it is not sufficient. We need to look at the entire solution across a number of dimensions in order to safely and successfully deploy complex solutions onto a virtualized-dominated datacenter world. However, it would be too complex to introduce solution specific resource definitions for each and every solution that a given customer might deploy. We need to find a workable compromise that allows complex services to benefit from the virtualized and highly automated environment while at the same time ensuring optimal deployment for the solution requirements. After reviewing a number of complex solutions including SharePoint and Exchange, it appears that a number of dimensions have to be expressed and designed into any resource architecture that will host complex services: Hypervisor feature support  – a better definition might...

The newly launched   Mingle SaaS offering   runs entirely on the AWS cloud. As discussed in our earlier blog on  Layering the Cloud , because there is no existing system that we have to modify or integrate with, we've got the freedom to design the architecture from scratch. This has led us to rethink the role of environments in our development and deployment process. What’s wrong with the traditional approach to environments? In traditional data-center-based applications there are usually a small, fixed number of environments into which the application is deployed. For example there might be the production environment, a staging environment where candidate builds are deployed before they go into production, a test environment where new work is verified and a development environment where developers can run new code as part of a full system. The availability and nature of these environments is strongly constrained by the availability of hardware and infrastructu...

To help, you can split your cloud architecture into three layers: Visible Volatile Persistant Visible This is the layer between the cloud and the rest of the world. It is mainly public DNS entries, load balancers, VPN connections etc. etc. These things are fairly static and consistent. If you have a website you will want the same A Name records pointing at your static IP. Things in the visible layer rarely change or if they do they are for very deliberate reasons. Volatile This is where the majority of your servers are. It’s called volatile because it tends to change a lot. The known state (how many servers, what versions of which software they run etc.) are changing frequently, perhaps even several times per hour. Even in a traditional data centre your servers are being upgraded, having security patches applied, new versions of software deployed, extra servers added etc. On the cloud this is even more volatile when you use patterns such as the Phoenix server and m...

I would like to take some time to discuss a wonderful coding tools that is provided by a number of modern chips makers. I am (of course) talking about the Non Maskable Interrupt (NMI). In general, there are two System Registers that are used to manage system interrupts – the Interrupt Mask Register and the Interrupt Cause Register. The Interrupt Mask Register allows the root user to disable/enable specific interrupts. This register contains a bit for each interrupt type. The Interrupt Cause Register indicates when the interrupts are ready for service. This register also contains a bit for each interrupt type. So when an interrupt comes into the system, the appropriate bit is set in the Interrupt Cause Register. If the appropriate bit is set in the Interrupt Mask Register, then the interrupt is generated and serviced by the appropriate Exception Handler. Otherwise, the system is not interrupted and the interrupt is essentially ignored. Non Maskable Interrupt Defined By defi...

Shared Memory File System Goals 1. MOUNTED IN SHARED MEMORY The result is a very fast, real time file system. We use Shared Memory so that the file system is public and not private. 2. PERSISTS TO DISK When the file system is unmounted, what happens to it? We need to be able to save the file system so that a system reboot does not destroy it. A great way to achieve this is to save the file system to disk. 3. EXTENSIBLE IN PLACE We want to be able to grow the file system in place. 4. SUPPORTS CONCURRENCY We want multiple users to be able to access the file system at the same time. In fact, we want multiple users to be able to access the same file at the same time. With the goals now in mind we can now talk about the major design issues: FAT File System & Design Issues The  FAT File System  has been around for quite some time. Basically it provides a pretty good file structure. But I have two problems with it: 1. FAT IS NOT EXTENSIBLE IN PLAC...

Any decent memory file system starts with a Boot Block. This is the first disk block on the file system partition. The Boot Block contains the file system data structure. And here is what it looks like: *) Magic number. We use magic numbers to help detect file system corruption. *) Number of disk blocks. The total number of disk blocks that is used by the file system. *) Disk Block Table. The Disk Block Table is an array that contains an entry for every disk block in the file system. Disk Block Table The Disk Block Table is an array that contains an entry for every disk block in the file system. If an array entry is -1, then the disk block is free and thus not used. Otherwise, the entry contains the index for the next disk block in a linked list of disk blocks for a file or directory. An entry of -2 indicates the end of a linked list of disk blocks. So the first entry in the Disk Block Table describes disk block 0, which is the Boot Block. Normally this entry conta...

Today I will be talking about the service delivery model for cloud computing. My presentation today would consists of, why cloud. We are all familiar with the terms about IaaS, PaaS and SaaS, what do we actually mean in terms of service deliverables. In hence I would be talking about what are the constituents of service delivery model in a cloud environment. We shall have a look at what is happening behind the scene, who benefits and how. Now in every service delivery model we have to create a win-win situation where the customer benefits and I should be able to do so demonstrate the same to you through a small case study. We follow this up with the question and answer session. The basic issues of why the cloud starts with let us understand, what are the IT challenges that the environment or the industry or the enterprise is facing today? I started my career 25years ago and in that point of time I head to explain a profiterole to a user. Today the environment is much different and t...