WAN Virtualization

APN's Innovations to Improve WAN Application Performance

In my first post in this trilogy, we looked at the factors which affect application performance on the WAN. In the second, we enumerated the various techniques vendors have implemented historically to address these issues.  Here, we'll take a look at what Talari’s Adaptive Private Networking (APN) solution brings to the table, and how these new techniques complement the existing solutions that the WAN Optimization, Virtual Desktop and WAFS/caching vendors deliver today.  

In addition to doing bandwidth management/QoS for outbound traffic essentially as any modern middlebox does, and offering TCP termination to address the bandwidth-delay product issue as many existing solutions do, APN innovates in a number of areas in addressing WAN application performance.

Most obviously, of course, APN's Resilient Multipath Connectivity enables link aggregation of diverse IP WAN connections while still delivering predictable and reliable performance.  Inexpensive Internet connections can now be used to reliably augment and/or replace expensive WAN connections like MPLS.

APN also offers innovation in a number of other application performance and performance predictability areas as well.  APN’s techniques for reliable packet delivery mitigate the effects of loss on TCP applications, buffering and retransmitting packets to make the WAN look like a zero-loss network, with occasional bouts of jitter.  Of course, the jitter introduced is no worse than a TCP application on the WAN has to be able to handle in case a packet is stuck in a queue in some WAN router behind a congested WAN link.  This enables TCP window sizes to stay at maximum, without negative impact on the WAN itself.  Further, in the face of more than moderate loss, or in the face of bursts of jitter, APN will move traffic, especially timing sensitive traffic, to a better path with lower loss, and/or latency.  By making this move away from “flaky” bandwidth sub-second (< ~250 ms across the continental U.S, < ~600 ms even for traffic going halfway around the world), APN delivers reliable predictable application performance even while using the “pretty good but not business quality" public Internet for its sources of bandwidth.

For real-time traffic like VoIP or videoconferencing applications (or even low-bandwidth, time-sensitive interactive applications like Citrix), APN will first put the real-time traffic on the best path with the lowest loss and lowest jitter to the destination.  As per above, if loss or congestion (jitter) is detected, it will switch the traffic, sub-second,  to a better path.  But for “platinum” quality voice – or for that matter, platinum quality videoconferencing, if you’ve got the bandwidth – it will send a replica of the packet stream on a different paths, as unrelated in terms of local and remote WAN links as possible; the replica is discarded at the receiving APN appliance.  Routing, of course, has always taken care of up/down network availability issues.  With this technique, even if what is usually the best performing path suddenly starts experiencing 68% loss or even a 150 ms burst of jitter, the same packet will show up at the destination along the other path, a few milliseconds later, and the application will never miss a beat.  APN counts on there being jitter buffers in the receiving hosts for this scheme to work, and in fact this is exactly how VoIP and videoconferencing work.

On the bandwidth mgmt/QoS front, APN delivers superior results for outbound traffic over existing solutions because, as a two-ended solution which continuously knows the state of all the network paths to the destination, it gets priority packets not just out the local link first – which is all existing solutions in practice do, since RSVP and IntServ never really “happened” – but to the other end of the WAN first, which is what you really want.

For inbound WAN bandwidth mgmt, in addition to doing a cruder version of what Packeteer PacketShapers have always done for limiting generic public Internet traffic on links shared with APN connections, it does far more sophisticated inbound management than any prior IP forwarding devices for inbound traffic coming from other APN sites, avoiding most last mile-based loss and congestion in the first place.  I.e. unlike TCP, which, as one sees with the famous TCP sawtooth, is actually designed to cause congestion and loss before backing off, APN will sometime temporarily attempt to send 3.1 or 3.2 Mbps into a 3 Mbps link, say, but will never try to offer 5 or 6 Mbps of load to the link, as standard TCP will habitually do, especially if there is inbound traffic from two or more locations.

WAN Optimization products, as a whole, combat the effects of limited bandwidth and high average latency.  WAN Opt typically delivers a 2x – 4x bandwidth benefit by freeing up bandwidth with its compression/caching techniques.  WAN Opt addresses high average latency with its generic TCP termination capabilities, and with its application-specific chattiness reduction optimizations for CIFS and HTTP.  WAFS and caching products generally combat the speed-of-light latency problem as well, in a less general, more focused fashion, but have historically achieved less success in the marketplace, in large part because of the extremely limited bandwidth available at most smaller locations.

Adaptive Private Networking is quite complementary to WAN Optimization, since it not only cost effectively enables much more bandwidth, but excels at combating the effects of packet loss and jitter (i.e. worst case latency), as well as frequently preventing loss and high jitter from occurring at the last-mile in the first place.  APN is also complementary – indeed, even an enabling technology – for distributed, replicated file services by enabling the provision of large amounts of inexpensive bandwidth, a solution not heretofore cost effective.

WAN Opt, WAFS, caching, Citrix and other application-specific solutions help some for bandwidth and are great at addressing speed-of-light latency issues; APN, while doing little to address speed-of-light issues, is unique in its approach to addressing the effects of WAN packet loss and congestion-based latency/jitter, as well as in preventing much of that performance-killing packet loss and congestion from occurring in the first place.

Traditional Techniques to Address WAN Application Performance

In my last post, we looked at the various networking-specific factors which affect application performance on the WAN. While an exhaustive review of all the possibilities would take far more than a single blog entry, let’s take a look now at the techniques vendors have implemented over the last 10 - 15 years to address these factors.

Last time, we saw that latency, packet loss and bandwidth are the 3 key factors which drive application performance.  Latency has a relatively fixed component based primarily on the speed of light for the distance a packet must travel on the one hand, and a variable component from the queuing congestion experienced at packet forwarding devices like routers on the other.  Packet loss is most of the time a function of queuing congestion, and partly a result of bit errors on unreliable links. Bandwidth, of course, is typically limited on the WAN simply because it has a monthly charge, and it’s expensive. 

Three other major factors affecting WAN application performance, which themselves are hugely impacted by latency and packet loss, are the bandwidth-delay product, TCP’s congestion avoidance mechanisms, including additive-increase-multiplicative-decrease (AIMD) scheme, and the “chattiness” of certain applications or protocols, especially Microsoft’s CIFS protocol and HTTP.

We know that for best WAN performance, we want to avoid packet loss and high latency as much as possible.  Successful techniques will either address bandwidth limitations or application chattiness, or will hide loss or high latency from the application, or will avoid the loss or high latency (i.e. jitter) occurring in the first place.  The best solutions, of course, will address multiple of these.

 

Link Aggregation – e.g. MultiLink PPP – has been around for a number of years.  Bonding multiple identical WAN connections together, essentially at layer 1, link aggregation allows greater overall bandwidth by combining multiple similar circuits.  Examples include bonded ISDN, and bonded T1 or E1 services.  [In the LAN, bonded Ethernet works similarly].  What these solutions have in common is that each of the links is identical, and has extremely low loss and extremely low jitter.  In such cases, simple layer 1 bonding techniques and MLPPP can work very well to deliver greater bandwidth.  This usually comes, of course, at a near-proportional increase in WAN service cost.

Bandwidth management, also sometimes loosely referred to as “QoS” (Quality of Service), is a way to deal with limited bandwidth to ensure that loss-sensitive or latency-sensitive applications like VoIP, videoconferencing or Citrix get “favored” use of bandwidth, essentially prioritizing certain traffic over others.  QoS/bandwidth management has been available in data networks “forever”. This essential technique for running real-time applications like VoIP can be sufficient if a single provider owns the WAN infrastructure, and can often improve performance for interactive applications like web pages when sharing links simultaneously with file transfers.

Bandwidth management / prioritization usually refers only to the outbound direction of packet forwarding.  All WAN forwarding devices – routers, WAN Optimization products, VPN products, Adaptive Private Networking appliances, etc. – available today do this type of bandwidth management and prioritization.

In the late 1990s, Packeteer (now part of Blue Coat) pioneered a technique known as TCP rate control to improve the use of inbound connections, especially public Internet connections.  By messing with the TCP window size and when it released TCP acknowledgement packets for a given TCP flow, Packeteer was able to reduce congestion and enable better performance of real-time and interactive applications coming into a WAN link.

 

Desktop virtualization – think Citrix remote desktop product, now known as XenDesktop – solves the problem a different way.  By running the entire application remotely, performance issues related to bandwidth and the chattiness of the underlying/original application are rendered moot, as the application now runs entirely on a LAN – or sometimes entirely in a virtual machine – at a data center.  Only the GUI, mouse clicks, key clicks, and screen results are transmitted across the network. Note, however, that desktop virtualization is both a solution to WAN app performance, as well as an “application” itself for which performance over the WAN – especially lossy or very high latency WANs – must be solved/addressed.

 

TCP termination in a networking device solves the bandwidth-delay product issue. Specifically, TCP termination solves the problem where TCP transfers are limited by the 32KB – 64KB maximum TCP window sizes on typical Windows or Linux OS hosts, which impede the ability to use large WAN links for transfers over long distances.  This obviously solves problems with high latency.  Less obviously, it also improves performance in the face of small amounts of loss, since TCP’s AIDM algorithm reduces the TCP window size by half upon a single lost packet.  Under small amounts of loss, the window size of a terminated flow will typically be much larger than the window size of the unterminated flow, and so performance is maintained at a higher rate using TCP termination in this case. [Note that under high packet loss, this benefit is muted, as the window size of the flow using TCP termination is unlikely to be much or even at all larger than the window size for an unterminated flow.]

Caching, e.g. as Akamai is well known for for public Internet applications, can solve the speed of light issues involved in getting objects/data across long distances, and can sometimes help a bit with bandwidth limitation issues, depending on whether the cache is at the local site versus whether it is (as for Akamai) in the network cloud but at a point very close in terms of latency to the client.  Such caching historically is an application-specific solution.

 

WAN Optimization products from companies like Peribit (acquired by Juniper) and  Riverbed Technology  came along early in the 2000s to focus on the issues of WAN application performance.  They first attacked the limited bandwidth available on WANs, freeing up bandwidth by doing data compression.  Initially, the compression was completely memory-based, then Riverbed introduced disk-based compression/caching, improving compression ratios substantially in some cases.  Compression has been around forever, but where in the ’80s and ’90s it might have yielded an average of 40% bandwidth improvement – and at a prohibitive cost in terms of CPU consumption at the time – memory-based compression delivers an average of 2x bandwidth improvement now, and disk-based compression typically delivers improvements in the range of 2x – 4x.

By combining compression with CIFS-specific optimization, WAN Optimization not only delivers a bandwidth benefit on average (with the occasional huge benefit when the same file is transferred then 2^nd, 3^rd, 4^th, etc. time), but also solves CIFS’ notorious chattiness problem.

WAN Optimization products also typically do HTTP-specific optimization, reducing the chattiness problems with that protocol.

Wide Area File Services (WAFS), or more generally distributed files services which do replication, such as Microsoft’s DFS (Distributed File Service) with Replication, are a specific implementation of caching/replication for files.  They deliver true LAN-speed performance for files which have been distributed to remote locations and stored on a hard disk on a server device there.   In the era of very thin WAN pipes that has been the case for the last decade plus, WAFS/DFS was appropriately beaten in the market by WAN Optimization products which not only solved more problems than just file access, but solved them better where downstream bandwidth is at such a premium that pre-positioning of files in a distributed fashion is not practical.

In my next post, we'll look at the innovations Talari is bringing to the table with Adaptive Private Networking technology to address application performance on the WAN.

The Whole is Greater than the Sum of the Parts

In Peter Sevcik's and Rebecca Wetzel's recent report on Flex-IT Networking: An Emerging Trend, they illustrate how the the confluence of some key technologies and capabilities will transform the way wide area networks will be built in the future.

Obviously, my interest in this area is directly related to the fact that one of the capabilities is to build a reliable private WAN out of a 'mosaic of network transport options', to create an 'Integrated Transport Network (ITN)'. This is exactly what APN technology was designed to do.

Server consolidation accelerated by server virtualization continues to be a hot trend and provides real OpEx and CapEx savings for IT departments. However, as computing resources become centralized and more distant from the end user, the bandwidth and latency demands on the enterprise WAN have increased significantly.  This paper points out that that colocation facilities combined with an ITN are primed to become an integral part of your WAN design to help solve this problem.

Colocation facilities, that have traditionally been used to house Internet facing web services and for disaster recovery, will play a broader role with an ITN approach.  Now they can be used to inexpensively consolidate branch services such as an email server, firewall and intrusion detection system within a geographical region close to the users without the need for an expensive private WAN connection. Connected with plentiful and cheap bandwidth they also provide an answer to the age old question of whether to back-haul your Internet traffic or not.

The resulting solution will provide the branches with both more Intranet and Internet bandwidth which is great for the users, and for the IT department it reduces costs and the number of locations that need to be managed.

So when you add it all up, cheap bandwidth allows you to move services to where they are more cost effective to manage, virtualization allows you to rent less rack space in those facilities and an ITN powered by APN technology allows all of the inexpensive network resources you can leverage to run even more reliably than the MPLS you thought you needed.

The Factors Affecting WAN Application Performance

Let’s take a look at the various networking-specific factors which affect application performance on the WAN.  I.e. We're not attempting to cover here things such as server capacity, or computational complexity, which are independent of “the network”. Said another way, what are the issues which cause application performance on a WAN to be so much worse than performance of the same application run entirely on a LAN?

In future posts, we’ll look more at which networking and computing techniques – Adaptive Private Networking, WAN Optimization, as well as other techniques – address each of these factors.  Before doing any kind of technology deployment to “make the network work better”, it’s useful to understand what the factors impacting network application performance and predictability are.

For those of you familiar with what WAN Optimization products like those from Riverbed Technology, Blue Coat Systems and Silver Peak Systems do, a lot of the below will be old hat; hopefully, you’ll still find some of the detail beneficial.

 

I contend that WAN-specific application performance is driven entirely by 3 factors: latency, packet loss and bandwidth.  While networking geeks might say “duh!”, some people familiar with WAN issues might say “huh? That’s not right”.  These people will argue that there are other factors as well.  I will show below that these other factors – which in fact are very important – matter precisely because of the impact of loss, high latency and/or limited bandwidth.

 

Definitions (from Wikipedia): “Latency is a measure of time delay experienced in a system, the precise definition of which depends on the system and the time being measured.  Latency in a packet-switched network is measured either one-way (the time from the source sending a packet to the destination receiving it), or round-trip (the one-way latency from source to destination plus the one-way latency from the destination back to the source). Round-trip latency [a.k.a. RTT - Round Trip Time] is more often quoted, because it can be measured from a single point.”

"Packet loss occurs when one or more packets of data traveling across a computer network fail to reach their destination.”

“Bandwidth is a measure of available or consumed data communication resources expressed in bit/s or multiples of it (Kbps, Mbps etc).”

Bandwidth is a pretty obvious limiting factor on transfers of large amounts of data, of course.  While it is most definitely not the only reason for poor application performance on the WAN, and in some cases has little or nothing to do with application performance, just as in the LAN, having more bandwidth will make many applications run better and more predictably, and in particular will make the network manager’s life easier. 

We’ll get into the reasons for latency and packet loss problems in WANs in just a second, but before going on, it is worth noting the three other major factors affecting WAN application performance, which themselves are hugely impacted by latency and packet loss.

The "bandwidth-delay product refers to the product of a data link's capacity (in bits per second) and its end-to-end-delay (in seconds). The result, an amount of data measured in bits (or bytes), is equivalent to the maximum amount of data on the network circuit at any given time, i.e. data that has been transmitted but not yet received.” The bandwidth-delay product, which essentially has no effect on LAN performance, is a well-known limit on how fast data transfers can occur over high delay Wide Area Networks.

A closely related issue is the manner by which TCP does congestion control: “TCP uses a network congestion avoidance algorithm that includes various aspects of an additive-increase-multiplicative-decrease (AIMD) scheme, with other schemes such as slow-start in order to achieve congestion avoidance.”  TCP’s congestion control algorithm, and AIMD in particular, is the primary reason why the Internet has not “collapsed” from the weight of everyone using it, and is the amazingly elegant way TCP’s designers came up with to efficiently use available bandwidth, and provide fairness “on average”.  For an individual application’s performance, however, this means that performance suffers notably with packet loss, and for interactive or real-time applications, can frequently have particularly bad performance when packet loss rates exceed ~1%.

Finally, there is the issue of the “chattiness” of certain applications or protocols.  Essentially, chattiness refers to how many multiple round trip communications – largely serialized – between client and server are required to perform a given application function.  A fantastic explanation of this issue can be found here in this NetForecast paper explaining Web performance over the Internet. 

Two common protocols which are very chatty are Microsoft’s CIFS protocol, and HTTP, the dominant protocol used for web applications.  Much like the bandwidth-delay product issue, “chattiness”, which doesn’t hurt performance much on a LAN, can have major consequences on a WAN facing packet loss and/or high latency.  For public Internet applications, including but not limited to web apps, large numbers of DNS (Domain Name System) requests are a form of application chattiness.

While there are other, innumerable application-specific factors, it’s not much of an oversimplification to suggest that in the end, their impact is quite similar to the “chattiness” issue noted above.

Ok, so if all network-specific performance issues can be traced back to bandwidth, latency, and packet loss, let’s look a bit deeper into the causes of latency and packet loss in IP WANs.

[Some sharp-eyed folks might be wondering about now “why hasn’t this guy mentioned jitter yet??  We know jitter is a huge issue in real-time application performance”.  In fact, jitter is a measure of the variability over time of the packet latency across a network.” – i.e. a component of latency!]

If we break down WAN latency into its constituent parts, we see both “fixed” components and variable ones.  The fixed components of WAN latency relate to the number of route miles a packet must travel between source and destination – and thus limited by the speed of light – with a smaller component based on the number of routers the packet must go through, with the small, fixed amount of time it takes to transit the router even when the links are lightly loaded.  The typical one-way latency across the continental U.S. is ~40 ms, meaning a typical RTT across the country and back is ~80 ms.

The variable component of WAN latency – the jitter, in other words – is caused by queuing congestion at the routers (or other IP forwarding devices) anywhere along the way.  Queuing congestion is caused when there is more data entering a device (router) trying to go out a given link than the bandwidth available on that link.  In typical IP WAN routers, queuing congestion at any given router can add up to 100 - 200 ms of latency.  Beyond that amount of delay, packets will typically be dropped – causing packet loss.

Overflowing queues in forwarding devices, as just noted, are the primary reason for packet loss.  Some routers use a technique called WRED to drop a lower percentage of packets when their queues are beginning to fill up, to avoid excessive jitter and better promote “fairness” across flows.  Finally, while less common than in the past, bit errors can also be a cause of packet loss.  Fairly rare on wired networks these days (beyond the occasional flaky DSL connection), bit errors are sometimes responsible for a moderate amount of packet loss on wireless networks.

High latency causes obvious problems in application performance.  Packet loss usually has an even bigger negative impact on WAN app performance.  Why is this?  Given the TCP bandwidth-delay product, loss rates above ~1% mean that the application can only use a very small amount of bandwidth on a WAN, not matter how much is available.  Over longer WAN distances, throughput is lower still.  Because TCP is a windowed protocol, forwarding of additional packets from the source will quickly come to a halt until the lost packet is retransmitted and acknowledged.  Even for applications where the bandwidth-delay product per se is not an issue, the “chattiness” problem has much the same – and often worse – effect.  In the face of packet loss, all data transmission halts until the packet is retransmitted and acknowledged.

Whew!  A lot to digest here, and we’re really only scratching the surface. We know that we want to avoid packet loss and high latency as much as possible.  In my next post, we’ll look at which techniques address which of these factors, combating the negative effects of loss and latency directly and indirectly.

Bet on the long term winners

On Wednesday, I was on the Breakthrough Network Technologies session at Interop New York that Jim Metzler moderated. The panel that Jim assembled had an interesting commonality across a number of us. Namely, leveraging the power of the MIPS available from Intel and the large base of open source code available on Linux.  You can see this in the next generation technology available from Arista Networks, Vyatta and Talari.

Arista is building solutions for large data center and high performance computing environments.  Vyatta is leveraging open source to be the “Red Hat of networking”, and enable customers to take advantage of commodity hardware in more and more parts of their network.

The advent of Linux-on-Intel for networking makes innovation more possible than ever before. Unless perhaps you're building a router for the core of the Internet, the x86 architecture today has plenty of horsepower for networking applications. This lowers costs for customers versus having to pay up for proprietary hardware R&D. Building on top of Linux and open source means that start-ups can begin from a much higher base, and take less time and money to deliver innovation to the market. Besides the obvious opportunity and lower costs for we start-up vendors, for enterprise customers it means that the value (and associated lock-in) of going with only large vendors for your networking purchases is lower than it's been in over a decade. 

The enterprise WAN buyer has not been able to take advantage of Moore’s Law and the ever-continuing price/performance improvements that most of the rest of technology – including the public Internet – have over the last 15 years.  For the reasons for this, see my last post.  Consequently, the last mile of the enterprise WAN has been the biggest bottleneck in networking and networked application deployment for years now.

The WAN Optimization folks like Peribit Networks (acquired by Juniper) and Riverbed Technology at the beginning of this decade had the bright idea to leverage CPU power – and gains in hard disks and DRAM – to alleviate this last mile thin pipe bottleneck.  They have successfully leveraged the power of the x86 architecture, and, I believe, the breadth of Unix/Linux as well, to add value to existing enterprise WANs, reducing bandwidth consumed on these expensive, very thin WAN links.

With Adaptive Private Networking, Talari is leveraging the power of Linux-on-Intel to build our heavy duty measurement,  intelligent forwarding,  bandwidth aggregation and bandwidth management technology which enables enterprises to take advantage of the Internet ecosystem economics and diversity (which has of course benefited from Moore’s Law advances) without sacrificing the reliability or performance predictability of their private WANs.

x86, Ethernet and TCP/IP became the ubiquitous winners of the last 15 years and vanquished their direct competitor technologies due to snowball economics.  Now, x86 and Linux (the rising star of this decade), will allow the Internet – the ultimate public network – to participate in, and in all likelihood, eventually subsume, the high margin proprietary private WAN business.

 

x86.  Ethernet.  TCP/IP.  Linux.  The Internet. 

 

Winners worth betting on.

 

[Note: I’m posting this from my Virgin America flight home to San Francisco, using Gogo Inflight Internet via WiFi (and free right now – thanks, Google!).  While APN will allow enterprises to take advantage of 3G and 4G networks as auxiliary capacity and diversity for their wired WANs, I don’t think Talari will be harnessing this particular WAN service any time soon, beyond its obvious powers to enable blogging!]

 

WAN Service History and Oligopoly Margins (or, Why does MPLS cost so much more than Internet connectivity?)

by Talari Blogger-in-Chief Andy Gottlieb...

MPLS is really expensive.  Internet connectivity is really inexpensive.

Why do private WANs cost so much?  More specifically, why are enterprise WAN services (MPLS, Frame Relay) priced so high?

Let’s take a look at the history of why private WAN services are priced the way they are.

I double majored in economics as an undergraduate, and I'm a firm believer that the market sets prices, and those prices are based on supply and demand.

Why are private WANs priced as high as they are?  It’s clearly not because of cost-based pricing.  Let’s assume that in the early 1990s the customer price for Frame Relay service quite likely was largely a function of the cost of providing the service.  Back then, in fact, Frame Relay was ground breaking price/performance, far better than the X.25 and leased lines it replaced.

In the roughly 12-15 years since that service became widespread, the cost/bit of the equipment to operate a carrier data network has gone down dramatically.  And while the ongoing costs of an “over-engineered” IP network, and of the headcount needed to provide better MTTRs in enterprise SLAs for last mile failures, mean that the cost/bit of running a private WAN is somewhat higher than for “plain old Internet service”, those ongoing costs are certainly not more than about, say, twice the cost/bit of providing Internet service.

I noted in an earlier post that MPLS is priced at $450 - $1,500 / Mbps per month for the copper connectivity typically deployed at all but the very largest enterprise locations, while the monthly price of broadband connectivity is ~$3 - $15 / Mbps per month. 

Fiber MPLS connectivity is typically in the $100 - $300 / Mbps per month range, while fiber Internet connectivity at a customer location is typically half that, and high bandwidth Internet connectivity at a colo facility is typically down around $10 - $20 / Mbps per month.

There is simply no way that the cost of providing private WAN connectivity is 30 - 100 times the cost of providing public Internet connectivity.  And so the pricing is clearly based on what the market will bear.  Which is surely the right of the carriers – and should be – in a free market economy.  But a look deeper into why the pricing is what it is can be quite revealing.

 

Broadband Internet service costs roughly $25 - $200 per month, and the bandwidth available is in the range of 3 Mbps to 50 Mbps.  One can argue that the telecom SPs were dragged into pricing this way because the cable companies – which had no telecom service market to protect – priced their product somewhat aggressively to target the consumer market, and have continued to increase the bandwidth available (e.g. 50 Mbps downstream for $189.95 / month from Comcast: http://www.comcast.com/About/PressRelease/PressReleaseDetail.ashx?PRID=814)

Whatever the initial reason for the pricing strategy, the fact remains that broadband is a growth business for the telecom SPs, and outside of wireless, broadband is one of their biggest investment areas.  And while monthly service prices for broadband seem to have more or less stabilized, price/bit continues to improve as the amount of bandwidth available per connection increases.  The price for Frame Relay and MPLS services (MPLS has always been priced under the Frame Relay umbrella, and is almost always within ~25% of the price of FR service of equivalent bandwidth) has come down somewhat in the last few years after being fairly flat for the first half of this decade, but has still not come down with Moore’s Law as with, oh... everything else in technology! 

Why is this?  I’d submit that it’s primarily for the following reasons:

• The last mile RBOC/PTT monopoly  “Backhoes don’t obey Moore’s Law” is the old saying, and with the Telcos owning the last mile TDM links – especially the copper connections to the overwhelming majority of business locations – they have been able to continue charging monopoly rents for these links.  Other than at colo facilities and in portions of some “NFL cities”, the same has largely been true for higher speed fiber links to business locations as well.

• Oligopoly of small number of vendors as credible national (global) providers of FR, MPLS  The oligopoly of 2 ½ vendors (AT&T, Verizon [formerly MCI], plus Sprint as the ½) in the U.S.,  and similar monopolies or duopolies in most other countries, limited enterprise’s choice of providers, especially since there has never been meaningful interoperability of FR or MPLS networks.  With fewer choices comes higher pricing.

• WAN buyers are correctly risk averse  Given limited IT staffs at most remote locations, it makes sense to be risk averse when it comes to the WAN, as the yearly savings of going with a cheaper but less reliable WAN service can sometimes be eaten up in handling even a single incident.  And given limited staffs, paying lower WAN costs and handling connectivity problems yourself when they arise is not even an option for some businesses with large numbers of locations.

• …and so they are not hugely price elastic  Unlike consumers, and many other parts of the computing or networking business, where a price reduction of 2x can sometimes result in 10 times the sales, service providers found that lowering private WAN prices did not result in higher revenues.  So they didn’t lower prices!

• The Internet on its own never got reliable enough  While public Internet connections have improved from the ~1 nine (90% - 95%) reliability they had 10 years ago, they are still only about 2 nines (99%) reliable – where reliability means the union of simple connection availability plus the ability to get your packets through to their destination without being lost or excessively delayed – unlike the 3½ to 4 nines reliability available from FR and MPLS.

• …and unlikely ever will The reasons for this have little to do with technology, and everything to do with economics and billing.  The carriers simply don’t get paid extra for making the public Internet – a network of networks – 4 nines reliable, and so it doesn’t pay for them to over-engineer their networks, and the network interconnections, and supporting network management, troubleshooting, etc. to do so.  The billing issues would be incredibly complex as well.  The billing issue alone is a key reason why technologies like multicast and IntServ over the public Internet never really “happened” either.

• Carriers are milking the FR/MPLS cash cow  While the telecom SPs have indeed aggressively moved customers from FR to MPLS, under the auspices of the lower cost of operating a single data network and ostensibly to deliver the QoS needed for real time applications like VoIP, they haven’t been investing all that much in their enterprise data networks, but instead have focused their investments in wireless, consumer broadband and perhaps the Internet core.

Given the above factors, absent a technology like Adaptive Private Networking (APN), I’d argue that Enterprise WAN managers – even those generally aggressive in rolling out new IT for competitive advantage or to lower costs – have done exactly the right thing being conservative with their WANs for the past 10 years.

But just as RAID enabled businesses to leverage the PC hard disk for their enterprise-class storage needs, APN, by solving the reliability and performance predictability issues associated with using public Internet transport, will enable enterprises to finally take advantage of the carrier's huge and ongoing investments in the public Internet – and indeed, in 4G wireless technology – for their corporate Intranets.

…and finally gain leverage over their telecom SPs and do something about those extremely high prices.

 The market does work!

 

APN and RAID: the Technical Commonalities

by Talari Blogger-in-Chief Andy Gottlieb...

As I note whenever I’m asked what it is that Talari is bringing to Enterprise networks, Adaptive Private Networking is doing for WANs what RAID did for storage. 

In a previous post, we looked at the analogy to RAID at the business benefits level.  Here, we examine the parallels on the technical level.

Wrapping hardware and intelligent software around core enabling technology

Seagate hard disk technology was originally designed for and targeted at the nascent but fast growing market for personal computers. Each hard disk was not nearly as reliable, having nowhere near the MTBF or seek times, as the existing mainframe/minicomputer disks.  But RAID - Redundant Array of Inexpensive Disks - took advantage of the massively better price/bit of the Seagate hard disk to revolutionize the storage market. By combining a layer of hardware and intelligent software with multiple of these inexpensive disks, RAID delivered a system with higher capacity, lower cost, competitive – and ultimately superior – access times, and greater reliability than the older generation storage solutions.

For APN, the analogous enabling technology is the public Internet.  By wrapping a two-ended system of appliance-based hardware running intelligent software around multiple WAN connections – most or all of which are Internet connections – APN creates an enterprise WAN which is lower cost, massively better cost/bit, higher capacity, and – critically – more reliable than the best single vendor MPLS WAN.

Redundancy to deliver application continuity

The basic idea behind RAID – and behind APN as well – is that while two devices operating in series which each have 99% reliability will deliver a system with only .99 *.99 = 98% reliability, a properly designed system with the same two devices operating in parallel  will deliver 1 – (1- .99) * (1 - .99) = 99.99% reliability.

The key phrase, of course, is proper design.

The first premise of RAID is that the loss of any single disk ensures not only that no data is lost, but that the application – data reads and writes – continues to function normally, without meaningful performance degradation.  This is essentially what RAID Level 1 delivered.

Existing traditional routed networks with appropriate link and device redundancy at each location, of course, provide network availability in the face of any hard single link failure or router failure.  But this "no loss of connectivity" is merely the equivalent of "no data loss" in the storage world.  Even a no-single-point-of-failure routed network does not provide for application continuity in all cases, as a routed network can take upwards of 30 seconds at times for router convergence in the face of a given failure.  More importantly, routing does not handle the case where packet loss or excessive latency causes significant problems with application performance.  Yet these “soft failures”, due to congestion on shared IP networks, especially Wide Area Networks, occur with far greater frequency than hard link or device failures.  

In fact, it is precisely because of the congestion-based packet loss and jitter which occurs at Internet peering points between ISPs that the public Internet has earned its "works pretty well most of the time" reputation.  Of course, "pretty well most of the time" just ain't good enough for your Enterprise WAN.

Handling "soft failures" - and doing it quickly!

APN, by leveraging multiple paths across the network between locations, ensures that no single hard or soft failure of a network link, device or peering point in the middle of the Internet will cause a loss of connectivity or application predictability.

APN does its equivalent of RAID Level 1 by continuous measurement of network path performance (loss, jitter, latency, bandwidth) and sub-second response to problems with any network path. In typically less than 3 round-trip times (RTTs), APN will move traffic off of a path experiencing high loss or excessive jitter.

For TCP applications, APN also delivers this predictability by buffering packets from flows and retransmitting them in the face of loss. 

For real-time application flows, APN can replicate the traffic on multiple paths, suppressing the duplicate packets at the receiving appliance, providing almost an exact equivalent mechanism to what RAID 1 does.  Application predictability, by avoiding loss and minimizing jitter, is much more important than efficient capacity utilization for real-time apps like VoIP.  By using Internet connections, the cost of that extra bandwidth is minimal.  For VoIP, it’s actually trivial.  Even for videoconferencing, it’s fairly small where the capacity exists.  [Note that APN also allows “conditional” replication of such flows, based on availability of bandwidth at the time.]

Striping = greater throughput

RAID Levels 2 – 5 (and up) allow improved data access performance by doing bit, byte and/or block level striping across multiple disks, allowing simultaneous disk seeks.  With APN, an individual TCP flow can be striped across multiple paths/links to deliver greater aggregate throughput.  The buffering and retransmission of packets, in conjunction with buffering and packet reordering at the receiving APN appliance and packet forwarding logic armed with the relative latencies of each of the network paths involved, enables high throughput even in the face of packet loss.

Automating the solution to MTBF and MTTR issues 

Network availability and predictability are quite naturally the number one concern of any enterprise WAN manager.  Just as RAID changed how storage systems were designed, an APN-based WAN turns the historic importance of, and so emphasis on, certain metrics on their head.

Before RAID, the MTBF of a disk subsystem was a critical factor in designing a highly reliable and available IT system.  With RAID, this concern pretty much vanished. Before RAID, MTTR of the storage system was a big deal as well. RAID solutions allowed much faster system MTTR (just swap out the hard disk and insert a relatively cheap replacement disk). The defective disk itself is almost never actually “repaired” any longer.  And with proper redundant system-wide design and automatic synchronization and backup processes in place, if in fact the MTTR of even making that disk swap ends up being 24 or 48 hours, no one is particularly bothered.

Similarly, prior to APN, the union of the network availability and the predictability of packet delivery of the private WAN, usually somewhere in the 99.95% - 99.99% range, has quite correctly been of huge importance to the enterprise network manager.  And the private WAN provider's SLA for a 4 hour MTTR, say, when a WAN problem does occur has also been extremely important.

With APN, however, the need for each WAN “subsystem” to be 99.9%+ predictable and reliable is greatly reduced. Given multiple diverse WAN connections combined with sub-second switchover in the case of packet delivery problems with any of them, the need for a 4 hour MTTR guarantee goes away as well.  Who cares if the MTTR is 24 or even 48 hours for a broadband connection at a branch office, say, if the network continues to work, and users notice no loss in connectivity and minimal loss in application performance and predictability?  And if delivering application predictability even during those rare times when a given link is down completely is important for your application or location, by having 3 diverse connections at that location, rather than 2, you will have superior application performance as well as network and application availability versus a private WAN actually delivering 99.99% uptime with an SLA committing to a 1 hour MTTR!

 

RAID revolutionized storage economics, improving storage reliability and capacity while radically reducing costs.  In a very similar way, APN is revolutionizing enterprise WAN economics, improving WAN reliability and enabling substantial capacity increases while simultaneously enabling network managers for the first time in more than a decade to not just have control over their WAN costs and their WAN providers, but to radically reduce that huge budget item as well.

Report: The Compelling ROI of Adaptive Private Networking

In his new white paper, The Compelling ROI of Adaptive Private Networking, well-known industry analyst and Network World columnist Dr. Jim Metzler of Ashton, Metzler & Associates analyzes data from two companies and finds that APN delivers:

• 45% – 86% monthly WAN expense reduction
• Payback in 3 – 7 months
• Bandwidth to each site increased between two and twelve times versus their current network
• End-to-End QoS functionality
• Greater availability and reliability than typical Frame Relay or MPLS services

Check out The Compelling ROI of Adaptive Private Networking.

The RAID Analogy

by Talari Blogger-in-Chief Andy Gottlieb...

When I’m asked for the “elevator pitch” on what Adaptive Private Networking is, I always go back to the RAID analogy – “APN is doing for WANs what RAID did for storage”.  Since we’re not on an elevator, let’s examine this analogy in some detail.

The analogy works both at the business benefits level, and at the technical level.  We’ll look primarily at the benefits part in this post, and the technical parallels in a future post.

Before RAID, in the 1970s and the 1980s there was a large market for expensive mainframe attached storage systems, which IBM dominated, and similar markets for minicomputer storage.  IBM’s margins were extremely high, and even the vendors of “plug compatible” storage systems had high gross margins as well.  They competed on the benefits of their “proprietary disk drive technology”, and only within a narrow range (a factor of 2 or 2 ½ at most) on cost.

Along came Seagate hard disk technology, targeted at the nascent but fast growing market for personal computers. Each hard disk was not nearly as reliable, having nowhere near the MTBF or seek times, of the mainframe/mini disks, but were built in high volumes with massively better price/bit.

RAID - Redundant Array of Inexpensive Disks took advantage of that Seagate disk price/bit to revolutionize the storage market. RAID solved the reliability and capacity limitations of the PC hard disk, and enabled them to be leveraged for data center business quality storage systems.  With RAID technology, companies like EMC built bigger, faster, cheaper and more reliable storage subsystems.  Within a very short time, no one tried to build big, fast reliable proprietary single disk storage systems any longer.

For APN, the existing high margin market is that for private WANs, which today means MPLS and Frame Relay.  Where IBM was the single dominant storage vendor before RAID, today with private WANs, the oligopoly of AT&T and Verizon (and to a much smaller extent, Sprint) in the United States, and government-sponsored telecom monopolies or near monopolies in other countries dominate the market, with 1 or 2 vendors in each company controlling 70% or more market share for these services.

Pricing of private WAN service, which was somewhat aggressive and clearly related to the cost of the service when Frame Relay was introduced in the early 1990s, now bears no correlation whatsoever to the cost of providing the service.  Rather, due to the last mile access (copper and fiber) monopoly by the RBOCs and the oligopoly of nationwide / global private WAN service providers, the price / bit of these services has barely come down in the last 10 years, even as the price of almost everything else has followed the classic Moore’s Law curve downward.

The irony here is that the analogous enabling technology for APN is the public Internet – much of it provided by these very same service providers!  Just as with the PC and the PC hard disk in the 1980s, massive investment in the Internet and broadband has been targeted at individuals and small business, and given the combination of high volumes and competition in that market, the price/bit here has continued to come down with Moore’s Law.  Internet access now costs $3 - $15 / Mbps per month, versus the $450 - $1500 / Mbps per month which the telecom SPs charge for MPLS.

While Talari’s APN technology is pretty sophisticated in terms of how we do real-time per packet measurements and sub-second response – more on that next time – just as with RAID, the price/performance comes from providing the surrounding reliability to enable leverage of Internet economics.  Where RAID wrapped a layer of hardware and intelligent software around multiple PC hard disks, APN’s two-ended, appliance-based solution does something similar with multiple WAN connections - existing private WANs and high speed Internet connections (T3, OC3, Metro Ethernet, etc.) at data centers and large sites, as well as existing connections and any type of broadband Internet links for branch/smaller locations.

RAID leveraged the PC hard disk technology to revolutionize business storage cost, capacity and reliability.  APN leverages the most powerful, ubiquitous, low cost communications network ever created – the public Internet – to deliver Enterprise WANs that are far higher bandwidth, far lower cost and more reliable than the best proprietary single service provider WANs available today.

It may well take a bit longer for APN to change the networking world as completely as RAID changed the storage world, but the parallels in terms of surrounding superior economics with business-class reliability are striking.

What would you do with 20 times the branch bandwidth?

by Talari Blogger-in-Chief Andy Gottlieb...

Fish don’t notice water.  It’s just a given of their environment.

Similarly, scarcity of enterprise WAN bandwidth is a "fact" that IT managers have long taken as a given.  Despite 10 Gbps LAN backbone and server links today, and a minimum of a switched 100 Mbps Fast Ethernet link to every desktop, with Gigabit Ethernet desktop connections more and more frequent, network managers – and application developers, too! – live with the reality that a 1.5 Mbps T1 WAN connection shared across 5 to 50 people is frequently all that’s available.  Even at larger sites, average WAN bandwidth per person will often be similar (or even lower). 

This 3 or 4 order-of-magnitude difference in the availability of bandwidth on the WAN has major implications for network design, server deployment and applications which can be run on the corporate WAN.  And in tough economic times like these, spending twice the amount on your WAN to get more bandwidth to run applications is just not an option.  Heck, spending the same amount you did last year is tough for some companies to do.

When we talk to customers about Adaptive Private Networking (APN) and its benefits, some customers pick up on this idea that they can now afford to have 20 or 30 times the WAN bandwidth they’ve been forced to live with, but for many, it’s an idea that’s so foreign they’ve not though much about the possibilities.

A great example of an application APN enables is videoconferencing.  “Gee, I’d never run ‘real’ videoconferences over the public Internet” is what any network manager worth his salt knows, after all.  That’s what QoS on MPLS is for. 

At Interop Las Vegas 2009, we’ll be demonstrating how having a lot of bandwidth changes the way networks are designed and applications are run.  While almost all techniques from WAN Optimization vendors like Riverbed and Blue Coat are about conserving bandwidth, APN actually takes the opposite approach, focusing, much as RAID did with “cheap” PC hard disks for storage, on making inexpensive bits business quality. 

Because of the inexpensive nature of Internet bandwidth, not only does APN make it affordable to have enough bandwidth to run high definition videoconferencing to the smallest of branch offices simultaneously on your now “converged” network with the rest of your data traffic, it actually duplicates videoconferencing and VoIP sessions to enable very high quality even over “pretty good but by no means perfect” public Internet connections. [Note: such duplication is administrator configurable, and can also be done dependent on the availability of bandwidth.]

So if you’re in Las Vegas next week, please stop by booth 1567 and see for yourself the video quality APN enables even in the face of network conditions which would ruin a simple voice call.

And start thinking about the applications you could run, and the centralization redesigns which would be possible, if only you had 20 times the bandwidth on your WAN. 

Because now you can afford to.