The Great Debate

Which model is best? It depends on your priorities. Let the debate begin! 

The Case for Building Your Own VNOC
By Anonymous

When do you in-source a Video Network Operation Center and when do you outsource it? Telepresence Options Magazine put the question to a VNOC operator who  runs conferencing for a global Fortune 500 firm and manages 500+ multi-screen, multi-codec telepresence systems and traditional videoconferencing systems. He preferred to stay anonymous, but he had a lot to say.

What's the case for building a VNOC yourself or going to a managed service provider to get the job done?

There's a case to be made for both. It comes down to customer service excellence, volume and scale. How many video connections do you do a day? How many video systems do you have and how much money are you willing to throw at it? You weigh that against customer service excellence and individual business needs  and you've got your formula.

I believe that an internally hosted VNOC has the ability to deliver the most optimal customer experience to meet its business' needs and culture. Whereas the managed services model, even the "white gloves providers," has to be built and designed to manage multiple clients and has to scale the customer's needs against their ability to produce a profit.

When you look at volume, if you have five systems and they're in 15 conferences a day, and that's all you have, then it makes sense to avoid investing in the infrastructure resources you need to manage that and go to a service provider, white glove/concierge or a virtual meet-me-cloud. If you have 250 systems and they're in 1200+ conferences a day, managing this internally is going to take a lot-video infrastructure, network, necessary staff, and physical real-estate investments--but it may justify itself, since a service provider may never be able meet to your internal customer's specific needs.

The other reason companies would take this on themselves is for security. Some firms are simply not comfortable with their sensitive internal business communications transiting equipment that they do not have control over.

There is a financial breakeven and gain, especially when you have lots of multiscreen immersive rooms. Those are costly to have run by a managed service provider. The specific breakeven is one that would have to be scoped separately for each enterprise.

Money aside, what are the reasons someone would want to bring an operation like this in-house?

For me, I would say the number one reason is to drive adoption that improves the overall ROI. Number two is to improve the customer experience. If those aren't priorities then you might as well buy an "out-of-box solution" and let it do its thing on its own and be moderately successful. But for us it comes down to being able to manage the experience to our internal customers' exceedingly high standards.

Business-to-business needs are another reason why you may want the privacy and security of managing the conversations and interactions internally. The risk of an unwanted leak could cost your company its reputation or have financial implications. Health care, legal and other fields that use video collaboration to manage mergers and acquisitions or other private financial matters may want to consider insourcing. That said, almost every cloud model offers NDAs as part of their solution.

What are you specifically doing to drive adoption over what a Managed Service Provider can do?

First thing is our video schedule is published to digital signs outside each room before every meeting so end-users know they are in the right place and how long the room is available. If the room isn't being used then people can see the availability, which improves the ability to hold ad-hoc meetings. When our customers walk in the room the meeting's ready to go, there's nothing they have to do. The only thing they really need to know is where the mute button is. Everything else we manage for them. So if it's a 28-seat multi-camera, multi-screen immersive room or a single screen room with a PTZ camera, the VNOC will selfframe that room. They will zoom in on the person and capture a two-thirds view so that it's equal proportions to all the other sites. Service providers will not do that part, and if they do they charge you an astronomical fee.

So it's how we work in the support model. We know who the hosts of the meetings are. In every one of our meetings we IM our hosts as they're sitting down to let them know which operators are assigned to their meeting to ensure they have the best experience possible. A service provider is not going to know that the person running the meeting is the global CFO; participants are just talking heads to them. For us, we take customer service very seriously--it's the highest thing we focus on. So we're really looking to see if the host has arrived and IMing him/her: "I'm Bob, your operator today. If you have any problems, you can contact me and I'll make it right for you." We manage the customer experience to a finite level.

The final thing we focus on is the need for every meeting to start on time without incident. Our customers want to walk into the room, sit down, and focus on their agenda, which is frequently a revenue producing opportunity for the company. The last thing that one of our internal customers wants to do is to walk into the room worried about the technology. That is why the industry has failed for so long and adoption is low overall--failed meetings cause lost productivity, wasted time and hard money.

What are the economics of bringing a VNOC in house?

It's got to be scaled on a one-off basis. I've got millions of capital invested in building the VNOC for our organization, and that includes bridges, infrastructure, operator equipment and real estate. In addition, I spend about million a year in operating costs to maintain everything and staff it. It's not cheap, nor is it for the faint of heart. It requires a commitment and dedicated follow-through.

NOC Photo Image Credit: flickr user Docklandsboy licensed under creative commons

Is that more or less than an MSP would cost?

I would say it's probably more.

How much more?

If it was truly full service, we would be looking at each multicamera, multi-codec telepresence rooms costing around $40,000 a year with an MSP. You add on our requirements for white glove service and it would probably be closer to $65,000 a room. We manage all our rooms for around $8,000, regardless of room type and are able to provide truly white glove service for every meeting.

What are the economics for small or medium firms. Where's the break-even on those?

Folks with just video-conference endpoints with a smattering of multi-screen, multi-codec systems. They should outsource the whole thing. I don't think it even makes sense until you have over 100 endpoints. It goes back to adoption, and it goes back to automation. If your rooms are being used 80 percent of the time then that could be a lot of connections and a lot of management. It would still make more sense to outsource. You'd just get more bang for the buck.

What kind of gear and personnel are needed for success?

There are four levels of staff. You have your operational staff and your engineering staff, which you can break into two groups. For engineering you need an engineer and an architect. You can also leverage a provider's architect if you want. You definitely want an engineer who knows what he's doing and can speak knowledgeably to what's being implemented and how it's being implemented. From the VNOC operations side, you need three groups of resources. You need customer service agents, the people who manage the phones, the IMs, the emails, the bookings, all that. The next level is you need VNOC producers, the people who set up meetings, tear down meetings, monitor meetings, and constantly look for problems, errors issues, whatever it may be. Then you need the incident  management team, the team that's opening tickets, managing tickets, dispatching vendors and triaging meetings that have incidents.

Did you get your reservation system off the shelf?

We bought it off the shelf and paid the vendor to customize it and it sucks. It's still the number one pain for me and the industry as a whole. This needs to be a focus area for the suppliers.

What do you need to run a basic VNOC five days a week, eight hours a day and 50 weeks a year?

Not knowing the exact number of sites or connections, this could differ. I would look at what's the volume you are currently running and where you think you will grow to. Realistically you could run a basic VNOC with three to five people, if all they do is connect, triage and tear down meetings.

What advice would you give to an organization contemplating bringing VNOC in-house?

Don't do it if you don't have to. Get an out-of-the-box solution or a managed service provider. The winner's going to be the one who automates it. That's your trigger point. When you get into integrating unified communications into your video conferences, that should be when you really start to look at in-sourcing VNOCs. Because there's not a service provider out there that's figured out how to integrate Microsoft Lync and IBM Same Time and some of these other environments seamlessly into their meetings.

How did you do it?

We integrated all of our MCUs directly into our unified communications environment. But I own all the MCUs, I own all the staff and it's a manual process.

What about companies looking to improve their business to-business communications?

They need to focus on reliable interconnections on the network side. The most important thing is figuring out the network piece. Most of the codecs transcode now. There's not this huge interoperability issue there was a year ago. If you're going to leverage the Internet to get anything outside your firewall or usea B2B exchange, you've got to have reliable inter-connections there. You also need to plan for at least three MBps of bandwidth for each high-definition call/connection to navigate through the firewalls properly.

What else do you think is important on this topic?

The one thing I would do before even engaging in this territory is to hire a vendor-neutral expert to help make the right decisions for your individual business. They'll help you evaluate your individual business and figure it out. And they can't be related to any of the vendors you would be working with, because you don't want a biased opinion. You want someone that's a reliable thirdparty looking at how your business works, how you operate, how you connect internally, how you connect externally. Are you driving revenue from this? Are you not? They'll consider all these people and parts and then take a step back and give you an analysis of what technologies it will take, based on your business model and how you do business today. Have them create a financial model looking at costs and service levels of the various MSPs you are interested in versus the cost of building a VNOC yourself and then make an informed decision. TPO

The Case for Managed Service
By Telepresence Options Magazine

The other side of the debate is outsourcing videoconferencing operations to a video managed service provider (MSP) that specializes in telepresence, videoconferencing, streaming and other video-centric technologies. Video managed service providers offer access to shared video network infrastructure that splits up the cost over many different customers. Many providers are staffed for 24x7x365 global operations.

Let's take a look at the business case:

Enterprises implement videoconferencing and telepresence because they improve employee productivity while minimizing expenses such as business travel. The big catch is that capital and operating expenses (underestimated or simply overlooked by CIOs and IT managers) frequently undermine all those bottomline benefits.

In fact, cost and complexity are two major reasons why so many enterprises--particularly small and mediumsize organizations--have either avoided implementing videoconferencing or minimized it to keep CapEx and OpEx at points they can live with. Some examples:

• Although videoconferencing and telepresence systems grow increasingly intuitive and user-friendly, they still often require a lot of employee hand-holding. That means enterprises have to add staff for tasks such as troubleshooting, training and room set-up.
• Multi-vendor interoperability remains a challenge, creating additional hardware and support-staff costs. Cutting corner on interoperability backfires because it limits the platform's ability to communicate and thus undermines its ultimate ROI.
• Laptops, desktops, smartphones and tablets are increasingly common endpoints, as is the use of consumer-grade video services such as Skype. Enabling videoconferences with such a wide variety of endpoints and services adds another layer of complexity and cost.

There's also another factor at play: The videoconferencing/telepresence market has become increasingly competitive over the past few years, a trend that's shrunk hardware and software margins. As a result, many AV integrators, carriers, and nontraditional service providers are now offering telepresence and videoconferencing as a managed service, which provides better margins, recurring revenue and the kind of long-term customer relationships that lead to upsale opportunities.

That trend is good news for enterprises because the competition between these new managed service providers enables them to get all of videoconferencing's bottom-line benefits without the ROI-busting support costs and other ongoing expenses.

There are several reasons so many enterprises and other organizations are going the managed-services route:

• Reduced Technology Risk: Telepresence and videoconferencing technologies are constantly changing. In the past decade we've seen the emergence of multi-screen, multi-codec telepresence systems, new videoconferencing standards like H.264 SVC, and an explosion of consumer and prosumer video including Skype, ooVoo and Google Talk. Managing this constantly changing landscape is a full-time job ... for dozens!

Scott AllendeVaux, SVP of MSP Architecture at telepresence and videoconferencing provider AVI-SPL explains: "Even for large-scale enterprises, the technology risk is substantial, especially trying to manage interoperability between disparate platforms and connecting to hundreds of carrier and enterprise networks for high-quality inter-company telepresence and videoconferencing. We have a staff of dozens of professionals across multiple disciplines immersed in the industry and technology on a full-time basis. I can't imagine trying to keep up as a side line."

• Predictable Costs: Instead of getting blindsided by the cost of upgrading codecs and hardware to maintain multivendor and multi-standard interoperability, enterprises can have those rolled into their monthly fee. That arrangement provides the kind of total-cost-of-ownership (TCO) predictability that's been lacking in videoconferencing. Enterprise upfront and startup costs - typically at least $1 million in hardware and software - also become more affordable with managed services. "By going to the cloud, you're saving on the entire back-end video infrastructure, [such as] the MCUs, the recording servers and the registration servers," says Anil Balani, Glowpoint senior vice president of technology. "We recently announced a new service offering where you can bundle in the endpoints as part of the service as-well."
• On-Demand Expertise: Managed services provide the kind of hands-on expertise critical for resolving problems quickly - or avoiding them altogether. Instead  of training staff to ferret out dead endpoints and anticipate firewall-traversal problems, enterprises can rely on their managed services provider's experience. And for multinational enterprises, a managed service provider also can offer 24/7 support.
• Scalability, Flexibility, and Usability: A managed service provider can help clients quickly and cost-effectively expand their room-based systems to devices such as PCs and tablets. After all, when more employees can use videoconferencing, the investment has a greater ROI. Meanwhile, the managed service provider also has the expertise to ensure the expansion goes smoothly, which is key to ensuring employees actually use what their company has paid for. "What we've been seeing is that if they choose a managed service provider, the adoption of video is a lot greater in the enterprise," Balani says. "It's all about consistency. When you go into the room, you know it's going to work. We have a couple of clients that have a dedicated person on staff for each conference to make sure that everything is working properly. That doesn't scale."

AT&T's Andy Adams chimes in: "The biggest trend that leverages Cloud/Managed Services is the "personalization" of video - and proliferation of single-codec endpoints and soft clients (desktop and mobile) with the concurrent need to scale "on demand" network and MCU resources to accommodate the transition from Scheduled to "Ad-Hoc" reservation-less meetings. Video is becoming less a replacement for a trip and more as an "enhancement" to and everyday audio/web meeting.

Organizations that may have 100 video or Telepresence rooms are looking to scale thousands or more personal users in the next 18-24 months. That puts too much of a burden on dedicated (internal) IT staff to scale (volumes) and geography (global). AT&T sees the break-even point at around 15% utilization - beyond that it's overwhelmingly in favor of a service providers for Day 2 support.

Little or No In-House Staff Required: Videoconferencing has unique requirements - including on the network side - that affect the user experience. Some enterprises learn that the hard way. "They find out after installing it that they don't have the expertise in house," Balani says. But there's a chronic shortage of videoconferencing talent, which means they command premium salaries and benefits. Some enterprises try to save money by sending their IT staff out for videoconferencing training. Outsourcing videoconferencing frees enterprises from the hassle and expense of finding and keeping experts in VNOC operations, network architecture and other must-have roles. Meanwhile, the enterprise's IT staff can focus on other tasks, instead of additional videoconferencing-related work such as troubleshooting and maintaining video infrastructure such as bridges.

ROI-Enhancing Synergies: Managed service providers often have expertise in other AV and IT systems, so they can suggest ways to maximize a videoconferencing system's ROI that CIOs or IT managers may not identify on their own. For example, at a law firm, a managed service provider might recommend and have experience in integrating videoconferencing and billing systems so that each conference is automatically charged to the right client.

Rapid Rollouts: The ideal managed service provider has the geographic reach, partnerships and staff that, together, enable even national and multinational companies to implement videoconferencing throughout their organizations, including remote and home offices. They can enable those rollouts faster and more cost-effectively because they've done them hundreds of times, unlike an enterprise IT department that has to feel its way through its first videoconferencing deployment. The faster the rollout, the quicker the investment starts driving bottomline benefits.

AT&T's Global Network Operations Center

Cap and Grow: Some enterprises start out by handling videoconferencing entirely in house. Eventually they reach a point where it's clear they can't afford to continue that approach. Maybe usage has reached the point that adding support staff dedicated to videoconferencing isn't financially viable. Or maybe their infrastructure is almost maxed out and adding ports and other gear is prohibitively expensive. In those cases, it often makes sense to bring in a managed service provider. The provider can take over operation of the existing infrastructure and give access to the additional hardware and software necessary for growth. "We manage many enterprises' own infrastructure," Balani says. "They're using the cloud as a capital augmentation to their own infrastructure."

Future Functionality: The equipment and services that you buy/ build today are not necessarily those that you will want and need in the future. The MSP community continues to innovate. One example is the growing trend of Virtual Meeting Rooms where individual employees have their own video room dial-in "number" similar to a reservationless conference call bridge where a service providers can scale "on-demand" network and MCU resources under a subscriber model. All the MSPs we interviewed for this article (AT&T, AVI-SPL, Glowpoint, and Teliris are offering or are planning on offering Virtual Meeting Room services. AT&T's Andy Adams discusses the advantages: " Customer implementing internal infrastructure and head-end network will forever be playing a "guessing game" of how many MCU resource (and what type) or how much bandwidth is needed to be provisioned for concurrent demand. This is inefficient, costly, and administratively difficult. AT&T's VMR's will be somewhat unique because of the ability to integrate with AT&T pervasive global MPLS fabric, third party VPNs, and public network access gateways. This provides the best quality of service and scale across an Enterprise (as well as outside of it). TPO