Thursday, June 27, 2013

Links for 06-27-2013

Posted by Gordon Haff at 4:35 PM 0 comments

IT risk isn't just about security

Because it mirrors a point I often feel compelled to make when discussing security and cloud computing, I wanted to highlight a couple of paragraphs from Cloud Computing: Assessing the Risks by Jared Carstensen, Bernard Golden and J.P. Morgenthal as excerpted in Tech Target.

It's important to understand that this risk limitation [whereby service providers shift the primary responsibility for risk to consumers] is not unique to Cloud Computing. Outsource providers (e.g. firms that take over operating a company's IT data centre) also limit their financial responsibility in the event of an outage. Therefore, it is important not to regard this risk limitation as a complete restriction on using a Cloud provider, unless, that is, a company regards any risk limitation by a service provider as unacceptable. In that case, the company should continue to operate its own computing environment and forego use of an external Cloud provider.

The important point from this discussion is that when Cloud Computing security is raised as an issue, other issues are often being addressed. It's important to distinguish what type of issue is of concern, as that will change the method of evaluating the issue, the demarcation of the trust boundary and the appropriate actions to be taken by the Cloud user.

One of the reasons why I think this point is important is that discussing overall IT governance discussions solely in terms of security (whether we're talking public clouds, private clouds, or--increasingly--some manner of hybrid IT) is far too narrow a framing. This narrow framing, in turn, often leads to thinking about the issue in narrow technical terms such as multi-tenant security features, encryption and key management, and physical facility protection.

These are important matters certainly. But they're also matters that public cloud providers (like other types of outsources) can reasonably argue they have well in-hand  using well-established procedures and processes. The more difficult answers about where workloads should run come down to broader questions--and those answers may well change over time.

(I covered some of these broader issues in a presentation at the Red Hat Summit in June. I'm hoping to get a version of that presentation Beyond Safety: Controlling Clouds posted over the next month or so.)

Posted by Gordon Haff at 10:46 AM 0 comments

Wednesday, June 26, 2013

Links for 06-26-2013

Posted by Gordon Haff at 4:26 PM 0 comments

Monday, June 24, 2013

Links for 06-24-2013

Posted by Gordon Haff at 4:04 PM 0 comments

Friday, June 21, 2013

The GigaOm Structure 2013 zeitgeist

Go structure logo 2013

I like GigaOm Structure. I find it gives a good sense of the current zeitgeist in cloud computing and related areas. What's being talked about and what isn't? What new or reimagined techs are emerging as memes? 

GigaOm's own writers (among others) covered the event in considerable depth and I won't attempt to recreate their reporting here. Rather, I wanted to hit on some general themes I noticed and a few points that particularly struck me. So with no further and in no particular order, here we go.

OpenStack was omnipresent. Other on-premise IaaS? Not so much.

Mind you. There was still a bit of commentary about how OpenStack was still in relatively early days. Maybe. Ryan Granard of PayPal told the audience that his company runs 20 percent of its production infrastructure on OpenStack. As readers probably know, there was much ado about PayPal's adoption of OpenStack--they were and are heavy VMware users--a while back. One of Grandard's points though was that PayPal has a strategy of deliberately making several bets as a way of getting velocity while still having a robust infrastructure. 

Notably absent from this Structure was the once ubiquitous Marten Mickos of Eucalyptus. Nor did I hear much mention of CloudStack--though Citrix did have a sponsor workshop, which I didn't attend.

Speaking of PayPal. A nice endorsement for PaaS and OpenShift.

Granard also articulated, as well as I've heard it from anyone, why PaaS is such a big deal for organizations. As reported by GigaOm's Jordan Novet:

Companies big and little have been jumping aboard the concept of on-premise PaaS, to some degree because security, regulatory compliance and cloud vendor lock-in fears remain part of the conversation about running on public infrastructure.

How is PayPal going about this? It’s been running Red Hat’s OpenShift on-premise PaaS to build out products such as PayPal Here — the company’s answer to Square — as well as a developer sandbox.

With that tool, Granard said, a developer chooses a product to work on “and in minutes, we have you up and running in a fully connected container” with infrastructure resources immediately allocated.

The real money quote for me though was that PaaS lets PayPal "enable developers and get out of the way."

x86 vs. ARM. Come back next year.

By which I mean that there were a few threads on this topic. Especially in the vein of whether ARM will make a meaningful dent in the server world. But no clear resolution.

To the degree that there was something of a consensus among folks I spoke with, it largely parallels my opinion and goes something like the following: x86 is the clear incumbent on the server. ARM is the clear incumbent in new-style mobile (tablets, cell phones, etc.). There's considerable inertia to that default condition for reasons of ecosystem and other things. For either architecture to make a major dent (narrow use cases aside) outside of its home base will require it to develop a 10x advantage--which most people don't think is going to happen.

What's that SDN stuff anyway?

There was some skepticism. For example, Arne Josefsberg CTO of ServiceNow said that  “The conversations today sound almost exactly like the conversations we had three to four years ago." Indeed, the "hot or not" panel he sat on declared SDN a loser technology.

But that seemed to be a minority opinion. Session after session returned to the idea that the networking component of infrastructures need the same sort of rewiring in software that you can do with compute and storage if the whole dynamic IT process is going to be realized. While I think it's fair to observe that there were still a lot of open questions about how we're going to get there (and what exactly it will look like), the consensus was squarely behind SDN--at least as a concept.

Private, private, hybrid

This topic really deserves a separate post, especially given that I was on a panel about private clouds at the ODCA Forecast event preceding structure. Suffice it to say that it's a complicated topic for a variety of reasons:

  • Depending upon the specific requirements, there are strong economic reasons to choose private over public or vice versa.
  • Different organizations have strong pre-dispositions for in-sourcing vs. out-sourcing
  • Existing applications can't be ignored
  • Regulation is a factor that may or may not be "fixed" (from the perspective of public clouds.

The bottom line is that there are plenty of arguments and cherry-picked examples available to bolster "your" side. That said, there was widespread agreement that, for at least the next n years (where n is much less agreed upon), the cloud world will be hybrid.

 

Posted by Gordon Haff at 1:28 PM 0 comments

Podcast: Geo, mobile, and more on OpenShift

Red Hat's OpenShift lets you skip the unproductive work associated with setting up development environments, work that Red Hat developer evangelist Steven Citron-Pousty (@TheSteve0 on twitter) call "yak shaving." Listen to this podcast to learn how easy it is to get started with OpenShift.

Here are links to some of the posts and other topics covered in the podcast:
OpenShift
Spatial MongoDB in OpenShift, be the next FourSquare - Part 1 | OpenShift by Red Hat
REST web services with Python, MongoDB, and Spatial data in the Cloud - Part 2 | OpenShift 
Kinvey
Appcelerator
Adopt-a-Hydrant

Shameless plug: I'll be on a panel with GigaOm analysts talking about PaaS Tuesday, June 25: Red Hat-Flexibility with PaaS: How to Keep Your Options Open — GigaOM Pro.


Listen to MP3 (0:17:57)
Listen to OGG (0:17:57)
Posted by Gordon Haff at 11:59 AM 0 comments

Monday, June 17, 2013

Links for 06-17-2013

Posted by Gordon Haff at 1:23 PM 0 comments

Podcast: The past, present, and future of Linux systems management

The scale and dynamism of cloud computing is changing the way in which systems need to be managed. Red Hat Satellite product managers Todd Warner and David Caplan talk about how these changes are being manifested in both current and upcoming Red Hat products and associated open source communities, such as Puppet and Foreman.

Listen to MP3 (0:24:09)
Listen to OGG (0:24:09)

[Transcript]

Gordon Haff:  Hi, everyone. This is Gordon Haff, Cloud Evangelist with Red Hat, and I'm down here in Raleigh, in the shiny new Red Hat Tower, talking with our two Red Hat Systems Management Product Managers, Todd Warner and David Caplan. Welcome, Todd and David. Todd, briefly introduce yourself, and then I will have David do the same thing.
Todd Warner:  Hello Gordon, my name's Todd Warner. I've been with Red Hat for approximately 10, 11 years now, and I've been associated with the Red Hat Satellite product for the majority of that time. Recently, I had the benefit of sharing that duty with David Caplan, who's now my co‑captain on the product. David?
David Caplan:  My name's David Caplan, and I'm a Principal Product Manager for the next generation Satellite. Todd has been working diligently on Satellite 5, and I'm picking up the reins on Satellite 6.
Gordon:  We'll get into a little bit about the actual products and the road maps later on in this conversation, but let's take things up a level to start with. Maybe you could talk to our listeners about what some of the big trends that you're seeing in systems management today, and how that's driving product change. Maybe we can start with you, Todd.
Todd:  Let me talk about where the industry has been for some time, and then I'm going to turn it over to David to tell us where he sees it going in the future, and how systems management needs to address that, and then also how Satellite intends to address it directly for our customers. For the longest time now, physical systems have been the primary platform for customers to build workloads on top of, and often for small shops that's been a handful of machines. For larger shops, it might be some BladeCenters or even a data center. Satellite was built with that premise in mind, the concept of the data center.
Satellite was introduced in 2002 as a means to patch systems, provision systems, and build standard operating environments associated to systems, but times are changing. Virtualization used to be a very specialized world in the old days in computing, which was just a decade ago. Now it's very prevalent, relatively cheap, and taking over the industry.
Satellite has addressed that, but with recent trends in computing, Satellite has to be more nimble, more adaptable. With that, we're talking about cloud, hyperscale, things like that. Let me turn it over to David to talk a little bit more about that.
David:  Thanks, Todd. The world of IT, in general, and systems management in particular, has certainly grown increasingly complex. The challenges of standing up servers on bare metal has now expanded to include problems like virtualization, as Todd mentioned, but not just one flavor of virtualization. There are competing standards, including RHEV/KVM on the Red Hat side. There's VMware, of course. EC2. There's OpenStack, and then there's all the variations on OpenStack that are beginning to emerge. Systems management has to evolve alongside so that it can be nimble and basically handle these different provisioning requirements, but still keep its eye on bare metal. Bare metal is also going through some radical transformation, as we go to data center densities of 10,000 servers to hundreds of thousands of servers, as we get into the world of hyperscale. That is something that we are watching here at Red Hat.
Open source is another emerging trend that has captured the imaginations of information technology [people]. It promises, with do‑it‑yourself techniques, to take care of some of the more daunting problems of configuration, drift management, and whatnot.
What we're trying to do at Red Hat, in systems management, is to take all the benefits of open source, all the innovation, and envelop it in workflows and structure, and really allow customers to derive the full value of open source innovation, but not be burdened by all the do‑it‑yourself vagaries and cul‑de‑sacs.
Gordon:  One of the interesting things I find talking about systems management and open source is I think, arguably, systems management was a relatively late field for open source. There have been some projects around for a while, mostly in the monitoring area, but I think systems management has tended to be such a large surface area type of application that it's been really hard for open source to tackle. Today, there really are some open source technologies, including some that we are using, that have really started to have a pretty major impact.
David:  Absolutely, Gordon. For many decades, systems management was controlled by a set of well‑recognized brands. The legacy providers who built very high‑quality stuff, but the iteration and improvement tended to be very slow once these things were deployed. On the other hand, open‑source moves at a very, very fast cadence, tends to be less encumbered by what's been done before, and can take a fresh look at solving problems. An example is Puppet, something that has in many respects exceeded all expectations, and certainly capabilities beyond what the legacy suppliers have been able to do, the proprietary shops.
Todd:  I did want to add to what David was saying, is that systems management's changed, not only technologically, but also in processes, workflow, sophistication of how people model their end systems. Just 5, 10 years ago, it was only common the big houses, the big IT firms, where you would model systems and layers, develop well‑designed SOEs, and have teams that owned each little piece of the layer of the application that went out the door. That's becoming more common all the way down to even the smaller IT groups, because it's more accessible, and the tooling is much better. Open‑source has really been a cost‑effective way of getting that level of technology, that reach of technology, in the hands of people that can't afford the million‑dollar systems management deployment.
The open‑source community is really outmaneuvering the larger systems management houses, because they can focus on the small tasks, doing very well. Then, folks like Red Hat helped tie all this together and build a better, larger system.
Gordon:  Let's make things a little more concrete here, and let's talk about some of the specific things that are going on at Red Hat. By way of context here, what we're discussing in this podcast is, specifically, the systems management area. It's certainly not the only thing Red Hat's doing in management. We acquired a company called ManageIQ last December, which we're combining with some of our in‑house developed open source technology into an overall open hybrid cloud management product.
What we're going to talk about here, what we're really going to dive down here, rather than this CloudForms hybrid cloud management, is specifically the systems management side of things.
I thought maybe you could start out by describing what Red Hat means when it says "systems management," and where we are and where we're going with our systems management product that we're shipping today.
Todd:  According to Red Hat, systems management really is building that flow of defining the system, deploying that system, managing that system over time, and then recycling that system. Managing many of those, many of those definitions, many of those systems, and being able to manage that at scale. Being able to build policies surrounding that, for example security, patching, configuration management, and things like that. To Red Hat, that's really Red Hat's definition of systems management. When we associate that to where we did the ManageIQ acquisition, the driving technology behind CloudForms, Red Hat needs to, as David was talking about, the future trends that we're adapting to are systems management technologies, in this case Satellite, have to adapt to that, so that we can leverage those technologies. Like, for example, the cloud technologies that ManageIQ is bringing to bear.
With Satellite, we have the systems management, that defining systems, managing systems, recycling those systems over time, at scale, physical, virtual, and now cloud.
Gordon:  David, talk about where we're going.
David:  That was an excellent description, Todd, of Red Hat and systems management, and we're building on that in our next generation Satellite, known as Satellite 6. If I could summarize in one sentence what we're attempting to achieve in Satellite 6, it would be bare metal to cloud in a single workflow. It's the recognition that most of the work today happens in the cloudy domain, whether that's hybrid cloud, whether that's private cloud, or in the public cloud. But getting there and automating the steps from new bare metal to the new world of abstracted resource is not easy for most IT customers.
Satellite 6 is designed to begin the process with bare metal discovery, in ways that previous systems could not achieve. Primarily because Satellite 6 is unaffiliated with any one supplier of hardware.
It has to do this bare metal for all of our partners and even hardware we've never seen before. Once we have things discovered, we need to register it, we need to provision it, and we do that with end‑to‑end automation. The next step is configuration, and we do configuration with recipe‑based solutions.
The current system built today for this is Puppet. We are leveraging all the power of Puppet, and building Satellite 6 around the Puppet ecosystem, by introducing something called an "external node classifier." The combination of Puppet, of Satellite 6, and the legacy that we have built on for managing content and entitlements, should provide customers with a very, very capable solution. Not just for now, but hopefully for another decade.
Gordon:  Todd, maybe you could tell our listeners specifically, whether they're current Satellite customers or someone who's interested in doing Linux systems management, what they can get from Red Hat today.
Todd:  Thank you, Gordon. Red Hat Satellite, we just released. I shouldn't say "just." At the end of 2012, we released Satellite 5.5, and 2012, that was our 10‑year anniversary for Satellite. It's a very mature product. With that release, we were really focused on modernization and compliance features. For example, "modernization" meaning keeping up with the times, IPv6, in particular, in this case. "Compliance" meaning I want to take a policy that another group gives me, apply it to a system, and report back if that system's compliant or not. Additionally, we had some content management improvements in that release, and some generalized scalability and network bridging technologies added within that release.
Satellite 5.5 was an incremental release for us that was released in October 2012. Coming this year, we have Red Hat Satellite 5.6 coming in this fall. We're really excited about this release, in that one of the key things we're bringing to bear in this release is that we're adding improved reporting. Our customers want to be able to understand how they are consuming product from Red Hat, and operating system resources better than they have in the past, and we hope to bring that coming this fall in 5.6.
Additionally, lots of improvements as far as manageability of the product. Hot backup support, being able to split the product into two so you can scale it out better when you install it. This is the server‑side piece of Satellite that you can split into two. We want to improve the way it scales.
We also are improving our ability to do client‑side introspection, as far as troubleshooting. A core operating system service goes down, Red Hat Enterprise Linux will send the details of that crash to Satellite so that, in one console, the administrator can see why that system had issues.
We're also giving some options as far as the database that Satellite actually utilizes. We're introducing PostgreSQL as an option in Satellite 5.6.
We have some coming releases in the next couple years after that. We have currently planned out two more releases after that, into 2015. I don't really want to go into details surrounding those. They're still nebulous and in motion, but we do have, currently, 5.6 planned for this fall, a 5.7, and a 5.8, all the way out to 2015.
What's more exciting, in my opinion, Satellite 5 continues to grow and mature, and it's going to be a supported platform for many years to come. In parallel to that, we're developing our next technology Satellite, which we're calling Satellite 6, and David will talk more about that and where we're going with that.
Gordon:  David, maybe you could share a little bit of detail about what people should expect, when they should expect it, what the use cases are that Satellite 6 might make the most sense for.
David:  Certainly. Thanks, Gordon. Thanks, Todd. Satellite 6 is currently under development now, and it's a system that was built from the ground up. A lot of the capabilities of Satellite 6 are derived from Satellite 5, and would be familiar to Satellite 5 customers, and familiar to Red Hat customers who are new to systems management, because the problems that it solves are very familiar. Satellite 6 is really broken up into two major components. There's a content and entitlement piece, which takes its cue, to a large degree, from Satellite 5. Some of the changes are the introduction of our customer portal as the main access for Red Hat content. Our previous access point, Red Hat Network, served Red Hat well for many, many years, but our own success has basically caused bottlenecks in getting customer access to content when they want it and where they are located.
Satellite 6 uses a worldwide content distribution network, and it plumbs to the points of presence of that network that are closest to where our customers are. It syncs content very efficiently into a kind of a common content mirror.
From there, there are exciting new capabilities, where customers are able to create very special content containers, which are called "content views," that are similar in concept to the channels that Satellite 5 supported, but are much more performant than what we are able to do with channels, so newer technology there in delivering these things.
The other part of Satellite 6 is entitlement management. Entitlement management is very important to our customers, so a lot of effort has been put into really superb and granular reporting of subscription consumption.
The other half of Satellite 6 is concerned with provisioning and configuration. Where Satellite 5 did managed kickstarts and then used configuration channels for files and other configuration information, Satellite 6 is built on an upstream project called The Foreman. It wraps itself around Puppet in a way that simplifies the construction and manipulation of kickstart files, the introduction of special, late‑binding override parameters, and a smooth and seamless handoff to Puppet.
Where the two systems are integrated seamlessly is in the content delivery part. When Puppet runs and extracts content, when it's doing its work of provisioning a server, that content is coming from the content management system that I described previously. It's a very tightly controlled set of processes. When systems are up and running, they register for errata and can be repurposed or re‑provisioned at any time. That's basically the 10,000‑foot view of Satellite 6.
Gordon:  Great. Thanks, David. Can you maybe talk a little bit about the managed design program?
David:  Of course. The timetable for Satellite 6 is roughly a year from now, so we're talking about June of 2014. It's a big program, and it has a lot of button and knobs. What we are hoping to do is to deliver something at that time that is ready to go, and is familiar, and useful to the largest number of Red Hat customers. The way we're getting to that, or achieving that goal, is with a special program called the "Managed Design Program." The idea of this program is to pre‑release Satellite 6 at three different stages between now and GA, and let our customers experiment with the software, exercise the workflows, and then give us feedback about what could be better, what they love about the product, we're hoping to have a lot of about what they love about it.
It also allows us to have a much closer relationship with those customers who have a real stake in open source systems management and Red Hat.
This program is a big win for our effort to build the right product at the right time, but it's also a win for our customers, because they will be able to see many of their great ideas and innovations incorporated into something that they can subscribe to when the product becomes generally available.
The first MDP drop will happen this summer, and then the others in three‑month cadence.
Gordon:  Briefly, what would a typical customer getting involved in the managed designed program look like?
David:  The ideal MDP participant would have Satellite 5, be using Satellite 5, and exercising many of its complex workflows. Whether it's patching, provisioning, configuration, content management. Those types of flows will be covered in the MDP version of Satellite 6, so that would be very important to us. Equally important are customers that are already ahead of the curve, and are using Foreman and Puppet today, and are looking for ways of tying it back into their content and content versioning. That would be the second candidate.
Lucky for us, as we've gone out and talked to many of our customers, we find that there are many, many Satellite 5 customers are already using Puppet today. They're experimenting with Puppet, and they're using it the do‑it‑yourself way, and looking to Red Hat for guidance, and looking to Red Hat for enterprise expertise in tying these things together. That basically describes the ideal candidate, Gordon.
Gordon:  I think one of the takeaways from here, and I think Todd, you may have been the one that used the term, "hyperscale," is as we look to how cloud computing is developing, one of the hallmarks is you're talking about orders of magnitude difference in terms of the increase in the number of running instances under management. Trying to handle that kind of thing manually certainly hasn't been, necessarily, a very good idea for many years now. When you're talking thousands of instances in even a moderate‑sized organization, that simply can't be done manually or you're just asking for runaway administration costs, and for that matter, runaway compliance problems.
David:  Right, Gordon. In the past decade at Red Hat, we've seen our customers expand from tens of machines to 100 machines, to thousands of machines, to tens of thousands, to hundreds of thousands of machines being managed. They're able to do this because A, the technology is getting cheaper at the client side, and B, the workflow and processes in order to manage those systems has been improving over time.
Satellite's been right there with them the whole time, but we are seeing the challenge with the boom in number of systems represented by virtualization and cloud, that Satellite has to be able to adapt to extreme large numbers of systems. How does a customer deal with this issue? The fact that they have workloads that are spanning many, many systems.
One way is to expand the number of actual human beings working on those problems, but that's not realistic. Satellite exists today and it does expand to thousands of systems, but we're working towards Satellite 6 being positioned to better manage many, many, thousands of systems with a reasonably small team of administrators. That's the challenge today. That many systems, and make sense of it, to a reasonable number of people.
Gordon:  Fundamentally, this is just one facet of everything has to be automated in a consistent way. Thank you David and Todd. Anything you'd like to add?
David:  No, only that it's a pleasure doing this podcast, I hope our customers find this interesting. Satellite is a tremendous product, it has tremendous success with our customers, and I hope you're encouraged that we're not standing still. Satellite is improving, and we're making investments in new technologies in the open source world with existing product today, and where we're going in the future.
Todd:  We're very excited about what will be coming out through our MDP program and then through our general availability of Satellite 6. Pay attention, check out redhat.com and see what's going on with the latest systems management, and thank you very much.
Gordon:  Thank you David, thank you Todd.
Posted by Gordon Haff at 10:37 AM 0 comments

Wednesday, June 05, 2013

Stop data silos

Apropos of the post I put up earlier today, this sponsored post on GigaOm notes that:

Data silos and integration challenges, more than security, are the biggest barriers to cloud adoption. Siloed data in cloud apps and data centers is costing companies millions annually due to inconsistency, inaccuracy and inefficiency across the business. And the enterprise software market is crossing the threshold of another transformation, now that cloud computing has shifted the center of gravity for data.

The MIT Media Lab's Sandy Pentland noted something similar at the MIT Sloan CIO Symposium a few weeks back when he said that "About 20% of big data is getting the data out of the silos and transforming it." 

That's why at Red Hat we're so interested in the idea of open hybrid cloud--which includes open hybrid storage. (Starting with Red Hat Storage based on GlusterFS.) This isn't to minimize the important, and really difficult, role of organizational change in breaking down the silos. But there's at least technology available to help break down silos rather than aid in their creation.

Posted by Gordon Haff at 2:26 PM 0 comments

Links for 06-05-2013

Posted by Gordon Haff at 2:07 PM 0 comments

Why we should probably retire "Big Data"

Data is hugely important. Long has been of course. It's often argued that applications are the longest-lived IT asset. Arguably the data that those long-lived applications create and access sticks around for at least as long. Data is only going to become more important.

Is there a lot of hype around data today? Sure. Raw data isn't information. And information doesn't necessarily lead to useful action if organizations (or their customers and users) aren't willing to change behavior based on new information. Just because pricing mechanisms can be used to reduce traffic congestion based on sensor data--just one of the ideas discussed under the "Smart Cities" term--doesn't mean that even basic congestion pricing is necessarily politically viable. 

Furthermore, I strongly suspect that the lots of data hammer will turn out to be a rather unsuitable tool for certain (perhaps many) classes of problems--the enthusiasms of the "End of Theory" crowd notwithstanding. For example, it's unclear to what degree more data will really help companies design better products or target their ads better. (To be clear, there's long been lots of market research in the consumer space. It's just that it hasn't been terribly effective in creating the killer ad or the killer product.)

But it would be a mistake to think that today's data trends are 1990s-style Data Warehousing with just a fresh coat of paint. Whatever the jokes about the local weatherman, weather forecasting has improved. Self-driving cars will happen, though they may take a while to come into mainstream use. DNA sequencing is now commonplace--although, in a common theme, we're still in the early days of figuring out what we can (and should) do with the information obtained. And we're well on the way to sensors of all sorts becoming pervasive. 

Which makes the "Big Data" term somewhat unfortunate in my view. I realize that may seem a bit of a contradiction given what I wrote above. Let me explain.

My first problem is that "Big Data" is too narrow. This is true even if we use the term in the broader sense of data that is atypical in some respect--not necessarily in its volume. The four Vs is a common shorthand. (For a less precise, but possibly more accurate description, I like the "Difficult Data" term that I heard from the University of Washington's Bill Howe.)

But an emerging culture of data doesn't have to be about big or even difficult. Discussions about data at the MIT Sloan CIO Symposium last month included big data examples, but it was also, in no small part, about cultures of data and the breaking down of silos. Just as with IT broadly and cloud computing, data and storage have to increasingly be based on a hybrid model in which data can be accessed when and where it is needed and not locked up in one department or even one organization. Governments and others are increasingly making open data available as well. 

It's also worth remembering Nate Silver made headlines for calling the last US presidential election correctly, not because he did big data stuff or even because he applied particularly innovative or sophisticated analysis to polling data, but mostly because he used data and not his gut.

The second issue I have with "Big Data" isn't really that term's fault. Rather, it's that "Big Data," today, is so frequently conflated with Hadoop.

Based on Google's MapReduce concept, Hadoop divides data into many small chunks, each of which may be executed or re-executed on any node in a cluster of servers. Per Wkipedia: "A MapReduce program comprises a Map() procedure that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a Reduce() procedure that performs a summary operation (such as counting the number of students in each queue, yielding name frequencies)."

Hadoop also provides a distributed file system that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. (The standard file system is HDFS, but other filesystems, such as Gluster, can be substituted for higher scalability or other desirable characteristics. 

Hadoop is often a useful tool. If you can split up data sets and work on them with some degree of autonomy, Hadoop workloads can scale very large. It also allows data to be operated in-situ without being loaded and transformed into a database, which can greatly decrease overhead for certain types of jobs. (This presentation by Sam Madden at MIT CSAIL offers some benchmarks as well as some pros and cons of Hadoop relative to RDBMS systems.)

However, data can be processed and analyzed using a wide variety of tools, including NoSQL databases of various kinds, "NewSQL" databases, and even traditional RDBMs like PostgreSQL (which can still scale sufficiently to handle a great many types of data analysis and transformation tasks). In fact, we even see something of a trend with some of the new-style databases adding back in traditional RDBMS features that had been assumed to be unnecessary. 

Even high volume data doesn't begin and end with Hadoop. As Dan Woods writes for CITO Research: "The Obama campaign did have Hadoop running in the background, doing the noble work of aggregating huge amounts of data, but the biggest win came from good old SQL on a Vertica data warehouse and from providing access to data to dozens of analytics staffers who could follow their own curiosity and distill and analyze data as they needed."

Hadoop is an important tool in the kit when the amount of data is large. But there are lots of other options for that kit bad too. And never forget that it's not just about the bigness of the data but whether you share it with those who need it and whether you do anything with it.

Posted by Gordon Haff at 12:49 PM 0 comments

Monday, June 03, 2013

Links for 06-03-2013

Posted by Gordon Haff at 3:17 PM 0 comments

Friday, May 31, 2013

Links for 05-31-2013

Posted by Gordon Haff at 9:03 AM 0 comments

Thursday, May 23, 2013

Data in, hunches out

"Going with your gut is out." That line, from Russell Reynolds Associates managing director Shawn Banerji, neatly summed up a big chunk of yesterday's MIT Sloan CIO Symposium. There was discussion of the computing side of things as well. I especially liked EMC CTO John Roese's description of public clouds evolving to a "chaotic" (as in heterogeneous/hybrid) pool of resources in which special purpose clouds would have specific functions. But the majority of the day revolved around data.

Not necessarily "big data" by the way. One panelist—Jack Norris of MapR—even remarked that the "big data term is probably short lived." But, rather, the pervasive use of data, in whatever form and at whatever speed, to drive decision-making. As Annabelle Bexiga, the CIO of financial services firm TIAA-CREF put it: "Big data is just a richer set of data. [It's a] natural evolution of where data is going."

Erik Brynjolfsson

Erik Brynjolfsson (above) is the Director of the MIT Center for Digital Business. He spoke to how the data explosion was a revolution of technology but was also (and required) a revolution in management. 

He offered the example of publishing, described as historically a "culture of lunches"—which is to say a culture of hunches and people networks. But Amazon brought a culture of numbers to the industry. And things haven't been the same since.

A 2011 paper, which Brynjolfsson co-authored with Lorin Hitt and Heekyung Hellen Kim found that data-driven decision making at firms resulted in 5 percent higher productivity than at firms which weren't so data oriented. 

Brynjolfsson also discussed what might be called ambient data, data collected almost incidentally from sources such as cell phone records. He offered the example of streetbump in Boston, which uses an iPhone app to find potholes. At the same time, he observed that the app did best at finding potholes in the Back Bay, Beacon Hill, and other relatively upscale Boston locales. Why? Because that's where iPhones predominate. As Sloan's Andrew McAfee would elaborate on in his closing keynote (paraphrasing science fiction author William Gibson), "the future is already here. It's just unevenly distributed."

The panel which Brynjolfsson moderated also touched on some of the privacy issues associated with this ambient data. The MIT Media Lab's Sandy Pentland told how a big data commons, created by French telco Orange, was used to reduce commuting times in the Ivory Coast by 10 percent by rearranging bus routes using location-based data from mobile phones. Pentland went on to note, with more of a bit of understatement, that this sort of thing is politically controversial in places like the UK.

Andrew McAfee

For his closing keynote, Andrew McAfee (above) took as his jumping off point a fascinating graph that appears in Ian Morris' Why the West Rules—For Now. At the risk of offering up spoilers, the central thesis of the book is that, viewed from the perspective of today, the level of worldwide social development prior to the industrial revolution is effectively in the noise.

(Although McAfee didn't get into this, the answer to the book title's question is basically that the West was better positioned to create and take advantage of the industrial revolution when the factors making it possible came together. It's a great read. If nothing else, it's a good history of the world from the perspective of the Western and Eastern core.)

The industrial revolution had such an impact because it overcame the limitations of human muscles. (I suppose, given farm animals, it would be more accurate to say it overcame the limits of human and domesticated mammal muscles generally.) In any case, though, McAfee's thesis that that today we're starting to overcome the limitations of our individual minds.

He laid out four elements to this:

Cyborgs—as in new combinations of people and machines.

Open—which will define successful organizations in a variety of ways. (If you want to delve more deeply into this thought, I point you to a presentation I gave at ProductCamp Boston a few weeks back.)

Data-driven—because for the first time we have data-driven visibility in all sectors of the economy.

Evolving—for which McAfee offered the example of the car rental industry which evolved only incrementally since it was founded after the Second World War but has seen the introduction of radically new services made possible by the Web and mobile phones from Zipcar to Lyft.

So it's more than data. But data—along with the compute needed to operate on it and the networks needed to move things around and tie them together—is a common thread. Big challenges lie in gaining access to the right data, even within single organizations. Cutting across data silos was also a theme heard more than once throughout the day. Asking the right questions matters too. As McKinsey's Michael Chui summed up that thought: "Be data driven. But don't suck at it."

Posted by Gordon Haff at 10:52 AM 0 comments

Tuesday, May 21, 2013

Links for 05-21-2013

Posted by Gordon Haff at 12:39 PM 0 comments

Thursday, May 16, 2013

My first MOOC (Massively Open Online Course)

Small icon hover

I recently completed my first Massively Online Online Course (MOOC), a term that presumably is at least passingly inspired by MMORGs, an online gaming genre that's most popularly represented by World of Warcraft. 

The class was on Gamification and was well-taught by Wharton prof Kevin Werbach. But my focus here isn't to review or critique this particular class but, rather, to offer more general reactions to the instructional method. To reflect on what these courses seem to do well or at least handle relatively naturally, and what they struggle at. It's just a sample size of one—well, 1.5 actually as I'm currently taking a Data Science course that offers some additional insights—but I think it nonetheless exposes certain patterns.

I also encourage those interested in the topic to read Nathan Heller's "Is College Moving Online?" in the New Yorker, a thorough examination of the state of MOOCs and their potential effects on education—both for good and ill. 

The format

The format for Gamification—which it seems is fairly typical—is built around a series of lectures. These consist of fairly typical Powerpoint slides with video of the instructor superimposed or off to the side in a small window. The production values are generally high and the combination of slideware and video is engaging.

There's a syllabus with links to various articles and other (free) materials. Prof. Werbach actually has a book on the topic of Gamification but it wasn't required for the course. None of the Coursera courses I've taken a look at had much if any in the way of stuff to buy.

The course then had a series of multiple-choice quizzes and a multiple-choice final exam—plus three written assignments of increasing length and scoring weight. My current Data Science course likewise has a series of lectures. But, in this case, the score comes from a series of programming and other assignments that relate to lecture topics although they are more hands-on and practical.

Type of schedule

Gamification, like the course I'm currently taking, comes from Coursera, which has a large course catalog from a wide range of schools. It was started by two Stanford professors and has received $16 million in funding from Kleiner Perkins Caufield & Byers.

Coursera's model, like that of the non-profit edX, is to offer classes in more of less real-time—by which I mean, an eight week course has a fixed start date followed by an end date about eight weeks later. Depending on the class, there may be more or less flexibility in how closely students have to hew to a weekly schedule for assignments, quizzes, and the like. But, fundamentally, the class is on a calendar and you can't dip in and out on the basis of work or family obligations, travel schedules, and inclination. 

The downsides to this approach are obvious. There are a couple of classes I've considered but opted against because they overlapped periods when I would't have been able to devote much time to them.

On the other hand, after taking a class, I better appreciate why one might want to run a class to a schedule. Discussion boards, assignments (especially peer-graded ones—more on those in a bit), the availability of staff to answer course questions or address problems, and just getting forced into the "flow" of a class all require or at least greatly benefit from a schedule and associated incentive structure. (Education has a lot in common with a gamification system.)

In a related vein, I also better understand why you might not want to break a course into overly granular chunks, i.e. one or two week classes. I'm not just talking about any particular administrative overheads associated with putting a course in a catalog, but a broader set of transaction costs borne by all the participating actors such as figuring out how the class is run and understanding or dealing with prerequisites or tools. In many cases, it seems these would collectively just add too much overhead to a short course unless that course were more intensive than most people with a full-time job could undertake.

Does the fact that these Coursera courses so reflect the form and content of traditional university classes simply reflect tradition and the fact that much of the content was originally developed for such classes? Perhaps to a degree. On the other hand, whatever issues higher education may have today, it's also likely that not everything about traditional class instruction is wrong.

Another form of MOOC largely goes self-paced, even if related lectures still largely parallel the content of a semester of classes. Machine-graded quizzes can still exist, as can other types of computer- or self-evaluated assignments. Udacity, another VC-funded MOOC, follows this model today. 

Other types of online instruction that increasingly diverge from a true MOOC model include iTunes University and the many instructional videos on YouTube. More focused sites, such as Code Academy, teach specific skills.

I think both more- and less-structured forms will have their place. Many of us have a limited ability to schedule courses that follow a fixed schedule and find the flexibility of watch-when-you-can attractive. At the same time, I appreciate the relative discipline and other potential benefits provided by a more formal course structure.

Grading and certification

Coursera, like edX, follows another aspect of most traditional university courses; it grades you. In Coursera's case, this takes the form of a Certificate of Completion based on your performance in various assignments and quizzes as determined by the individual class—70 percent in the case of Gamification. (Coursera is also starting to offer a "Verified" version for certain classes.) 

I suspect that, at this point, a Coursera certificate is more of a gamification element, i.e. a motivator, than something that's especially useful outside of Coursera. However, it's also true that Coursera's business model will ultimately depend on being able to sell the ability to gain meaningful certifications—which means that they need to be able to grade.

Schools have, of course, been grading forever. But remember what the "M" in MOOC stands for? It's "massive," indicating that it's not at all unusual to have 50,000 people sign up for a MOOC. (Although far fewer will complete it.) It's obviously not practical for a professor and a few TAs to grade at that scale. In the future, I wouldn't be surprised to see hybrid models in which something "MOOC-like" is augmented with one-on-one and one-to-few interaction with professors and TAs for a fee—indeed we're starting to see examples of such—but let's stick to the subject at hand for today, given that what's actually being paid for starts to become a complicated question.

A couple observations about the state of grading in MOOCs.

Multiple choice works well, subject to the limitations of multiple choice. Given well-written questions, there's no more ambiguity than with any other multiple-choice exam. It's easy and instantaneously graded by computer. And modest feedback can be made available with respect to the answers. 

Computers also do well at grading freeform but well-bounded and unambiguous answers, such as a numerical solution with only one correct response. It's "42" or it's wrong.

However, start talking more open-ended problems, even in quantitative topics like programming, and automated graders can start failing answers in unexpected ways. Without going into all the gory details, the autograder for the first assignment in my current data science course was highly sensitive to, for example, how the programmer chose to parse text fields and to any quirks in the output's format. While not fatal flaws, it's indicative of how automated grading challenges magnify exponentially the more creativity is allowed in solving an open-ended problem.

Also problematic is peer review. On the one hand, this offers a way for massive scale evaluation of freeform text in a way that it's hard for imagine computers tackling anytime soon. On the other hand, across a huge class with students of all ages, skills, language abilities, and interest, it's not hard to imagine that the evaluations can be… quirky—even given a relatively detailed scoring rubric. 

I didn't personally have a huge problem with my results. But it was obvious from the discussion boards that a lot of people took low evaluations made without comment and evaluations made with obvious disregard for the scoring instructions very personally. And it's worth observing that, given a large class, statistics suggests that some will have simple "bad luck" with those they draw to evaluate their written assignments—even given multiple graders, multiple assignments, and algorithms to screen bad actors.

At the current stage of MOOCs, I personally find it easy enough to shrug my shoulders and get on with things. I have lots of diploma and certificate things gathering dust somewhere. But say a class has a written assignment contributing say, 33 percent of the grade. To the degree this grade has meaningful consequences for the student (for a resume or for tuition reimbursement if MOOCs start charging in some form), that's a problem. It's also fair to say that, real world significance aside, grades can have a motivating factor for many as well.

This hasn't been intended as an exhaustive look at the "grading issue" but it's been evident to me it's something that will have to be at least improved on—not that students are ever wholly happy about their grades—as MOOCs look to start collecting money from people and offering meaningful certifications.

Overall

I got a lot out of this course and it looks as if there's a lot of quality content out there—more certainly than I'll have time to dig into. I'm also happy to see both for-profit and non-profit initiatives probing at ways to make higher-education better and more efficient. 

What I don't have a real opinion on is what the effect of Coursera, edX, and their ilk will be. The effect will certainly be uneven although one wonders if some aspects of MOOCs can replace elements of classes even at elite institutions. (Though I'd note that we've had the technology to replace 500 student freshman lectures for at least a decade.) I do suspect, or at least hope, that MOOCs—or at least MOOC methodologies—can replace low-value-add broadcast education in many situations.

One of the reasons that this particular crystal ball is cloudy is that higher education is often this odd hybrid of credentials, socialization, and learning that can be impersonal, highly personalized, solitary, involving lots of peer interaction, or some combination all of those. MOOCs clearly don't address all those modes but it arguably can do a subset rather well.

 

Posted by Gordon Haff at 1:52 PM 0 comments
Subscribe to: Posts (Atom)