WAN Virtualization

Bet on the long term winners

On Wednesday, I was on the Breakthrough Network Technologies session at Interop New York that Jim Metzler moderated. The panel that Jim assembled had an interesting commonality across a number of us. Namely, leveraging the power of the MIPS available from Intel and the large base of open source code available on Linux.  You can see this in the next generation technology available from Arista Networks, Vyatta and Talari.

Arista is building solutions for large data center and high performance computing environments.  Vyatta is leveraging open source to be the “Red Hat of networking”, and enable customers to take advantage of commodity hardware in more and more parts of their network.

The advent of Linux-on-Intel for networking makes innovation more possible than ever before. Unless perhaps you're building a router for the core of the Internet, the x86 architecture today has plenty of horsepower for networking applications. This lowers costs for customers versus having to pay up for proprietary hardware R&D. Building on top of Linux and open source means that start-ups can begin from a much higher base, and take less time and money to deliver innovation to the market. Besides the obvious opportunity and lower costs for we start-up vendors, for enterprise customers it means that the value (and associated lock-in) of going with only large vendors for your networking purchases is lower than it's been in over a decade. 

The enterprise WAN buyer has not been able to take advantage of Moore’s Law and the ever-continuing price/performance improvements that most of the rest of technology – including the public Internet – have over the last 15 years.  For the reasons for this, see my last post.  Consequently, the last mile of the enterprise WAN has been the biggest bottleneck in networking and networked application deployment for years now.

The WAN Optimization folks like Peribit Networks (acquired by Juniper) and Riverbed Technology at the beginning of this decade had the bright idea to leverage CPU power – and gains in hard disks and DRAM – to alleviate this last mile thin pipe bottleneck.  They have successfully leveraged the power of the x86 architecture, and, I believe, the breadth of Unix/Linux as well, to add value to existing enterprise WANs, reducing bandwidth consumed on these expensive, very thin WAN links.

With Adaptive Private Networking, Talari is leveraging the power of Linux-on-Intel to build our heavy duty measurement,  intelligent forwarding,  bandwidth aggregation and bandwidth management technology which enables enterprises to take advantage of the Internet ecosystem economics and diversity (which has of course benefited from Moore’s Law advances) without sacrificing the reliability or performance predictability of their private WANs.

x86, Ethernet and TCP/IP became the ubiquitous winners of the last 15 years and vanquished their direct competitor technologies due to snowball economics.  Now, x86 and Linux (the rising star of this decade), will allow the Internet – the ultimate public network – to participate in, and in all likelihood, eventually subsume, the high margin proprietary private WAN business.

 

x86.  Ethernet.  TCP/IP.  Linux.  The Internet. 

 

Winners worth betting on.

 

[Note: I’m posting this from my Virgin America flight home to San Francisco, using Gogo Inflight Internet via WiFi (and free right now – thanks, Google!).  While APN will allow enterprises to take advantage of 3G and 4G networks as auxiliary capacity and diversity for their wired WANs, I don’t think Talari will be harnessing this particular WAN service any time soon, beyond its obvious powers to enable blogging!]

 

WAN Service History and Oligopoly Margins (or, Why does MPLS cost so much more than Internet connectivity?)

by Talari Blogger-in-Chief Andy Gottlieb...

MPLS is really expensive.  Internet connectivity is really inexpensive.

Why do private WANs cost so much?  More specifically, why are enterprise WAN services (MPLS, Frame Relay) priced so high?

Let’s take a look at the history of why private WAN services are priced the way they are.

I double majored in economics as an undergraduate, and I'm a firm believer that the market sets prices, and those prices are based on supply and demand.

Why are private WANs priced as high as they are?  It’s clearly not because of cost-based pricing.  Let’s assume that in the early 1990s the customer price for Frame Relay service quite likely was largely a function of the cost of providing the service.  Back then, in fact, Frame Relay was ground breaking price/performance, far better than the X.25 and leased lines it replaced.

In the roughly 12-15 years since that service became widespread, the cost/bit of the equipment to operate a carrier data network has gone down dramatically.  And while the ongoing costs of an “over-engineered” IP network, and of the headcount needed to provide better MTTRs in enterprise SLAs for last mile failures, mean that the cost/bit of running a private WAN is somewhat higher than for “plain old Internet service”, those ongoing costs are certainly not more than about, say, twice the cost/bit of providing Internet service.

I noted in an earlier post that MPLS is priced at $450 - $1,500 / Mbps per month for the copper connectivity typically deployed at all but the very largest enterprise locations, while the monthly price of broadband connectivity is ~$3 - $15 / Mbps per month. 

Fiber MPLS connectivity is typically in the $100 - $300 / Mbps per month range, while fiber Internet connectivity at a customer location is typically half that, and high bandwidth Internet connectivity at a colo facility is typically down around $10 - $20 / Mbps per month.

There is simply no way that the cost of providing private WAN connectivity is 30 - 100 times the cost of providing public Internet connectivity.  And so the pricing is clearly based on what the market will bear.  Which is surely the right of the carriers – and should be – in a free market economy.  But a look deeper into why the pricing is what it is can be quite revealing.

 

Broadband Internet service costs roughly $25 - $200 per month, and the bandwidth available is in the range of 3 Mbps to 50 Mbps.  One can argue that the telecom SPs were dragged into pricing this way because the cable companies – which had no telecom service market to protect – priced their product somewhat aggressively to target the consumer market, and have continued to increase the bandwidth available (e.g. 50 Mbps downstream for $189.95 / month from Comcast: http://www.comcast.com/About/PressRelease/PressReleaseDetail.ashx?PRID=814)

Whatever the initial reason for the pricing strategy, the fact remains that broadband is a growth business for the telecom SPs, and outside of wireless, broadband is one of their biggest investment areas.  And while monthly service prices for broadband seem to have more or less stabilized, price/bit continues to improve as the amount of bandwidth available per connection increases.  The price for Frame Relay and MPLS services (MPLS has always been priced under the Frame Relay umbrella, and is almost always within ~25% of the price of FR service of equivalent bandwidth) has come down somewhat in the last few years after being fairly flat for the first half of this decade, but has still not come down with Moore’s Law as with, oh... everything else in technology! 

Why is this?  I’d submit that it’s primarily for the following reasons:

• The last mile RBOC/PTT monopoly  “Backhoes don’t obey Moore’s Law” is the old saying, and with the Telcos owning the last mile TDM links – especially the copper connections to the overwhelming majority of business locations – they have been able to continue charging monopoly rents for these links.  Other than at colo facilities and in portions of some “NFL cities”, the same has largely been true for higher speed fiber links to business locations as well.

• Oligopoly of small number of vendors as credible national (global) providers of FR, MPLS  The oligopoly of 2 ½ vendors (AT&T, Verizon [formerly MCI], plus Sprint as the ½) in the U.S.,  and similar monopolies or duopolies in most other countries, limited enterprise’s choice of providers, especially since there has never been meaningful interoperability of FR or MPLS networks.  With fewer choices comes higher pricing.

• WAN buyers are correctly risk averse  Given limited IT staffs at most remote locations, it makes sense to be risk averse when it comes to the WAN, as the yearly savings of going with a cheaper but less reliable WAN service can sometimes be eaten up in handling even a single incident.  And given limited staffs, paying lower WAN costs and handling connectivity problems yourself when they arise is not even an option for some businesses with large numbers of locations.

• …and so they are not hugely price elastic  Unlike consumers, and many other parts of the computing or networking business, where a price reduction of 2x can sometimes result in 10 times the sales, service providers found that lowering private WAN prices did not result in higher revenues.  So they didn’t lower prices!

• The Internet on its own never got reliable enough  While public Internet connections have improved from the ~1 nine (90% - 95%) reliability they had 10 years ago, they are still only about 2 nines (99%) reliable – where reliability means the union of simple connection availability plus the ability to get your packets through to their destination without being lost or excessively delayed – unlike the 3½ to 4 nines reliability available from FR and MPLS.

• …and unlikely ever will The reasons for this have little to do with technology, and everything to do with economics and billing.  The carriers simply don’t get paid extra for making the public Internet – a network of networks – 4 nines reliable, and so it doesn’t pay for them to over-engineer their networks, and the network interconnections, and supporting network management, troubleshooting, etc. to do so.  The billing issues would be incredibly complex as well.  The billing issue alone is a key reason why technologies like multicast and IntServ over the public Internet never really “happened” either.

• Carriers are milking the FR/MPLS cash cow  While the telecom SPs have indeed aggressively moved customers from FR to MPLS, under the auspices of the lower cost of operating a single data network and ostensibly to deliver the QoS needed for real time applications like VoIP, they haven’t been investing all that much in their enterprise data networks, but instead have focused their investments in wireless, consumer broadband and perhaps the Internet core.

Given the above factors, absent a technology like Adaptive Private Networking (APN), I’d argue that Enterprise WAN managers – even those generally aggressive in rolling out new IT for competitive advantage or to lower costs – have done exactly the right thing being conservative with their WANs for the past 10 years.

But just as RAID enabled businesses to leverage the PC hard disk for their enterprise-class storage needs, APN, by solving the reliability and performance predictability issues associated with using public Internet transport, will enable enterprises to finally take advantage of the carrier's huge and ongoing investments in the public Internet – and indeed, in 4G wireless technology – for their corporate Intranets.

…and finally gain leverage over their telecom SPs and do something about those extremely high prices.

 The market does work!