Virtual Testing Trims Real World Costs and Delivery Times
"We have to provide better and faster services to our customers every week and every month. We mostly had problems on issues such as the accessibility, authorization, downtime, and private data for reaching the other third party's infrastructures," said Hasan Yukselten, test and release manager for TTNET. "So we needed virtualization on our test systems and we needed automation for getting fast deployment."
When you're a top Internet service provider in Turkey, and you deploy scores of applications each year, you have to find an efficient testing method that ensures the apps are working properly across all infrastructures.
TTNET, with six million subscribers, found that service virtualization (SV) was the best solution. The ISP found that by using this form of app testing combined with automation, it was able to cut costs and time to delivery. The business side of the systems saw improvements, and end users were provided with better experiences.
Hasan Yukselten is the test and release manager for TTNET, a subsidiary of Turk Telekom that is based in Istanbul. In this podcast, Yukselten talks about how TTNET used advanced service virtualization solutions to automate end-to-end testing, which helped pave the way for integrated functional testing.
The discussion is led by Dana Gardner, principal analyst at Interarbor Solutions.
Download the podcast (23:58) or use the player:
Here are some excerpts:
Dana Gardner: Before we get into this discussion of how you've used SV in your testing, what was the situation there before you became more automated and before you started to use more software tools? What was the process before that?
Hasan Yükselten: Before SV, we had to use the other party's test infrastructures in our test cases. We're the leading ISP company in Turkey. We deploy more than 200 applications per year and we have to provide better and faster services to our customers every week and every month.
We mostly had problems on issues such as the accessibility, authorization, downtime, and private data for reaching the other third party's infrastructures. So, we needed virtualization on our test systems and we needed automation for getting fast deployment to make the release time shorter for greater virtualization. And of course, we needed to reduce our cost. So, we decided to solve the problems of the company by implementing SV.
Gardner: What did you do to begin this process of getting closer to a faster and automated approach? Did you do away with scripts? Did you replace them? How did you move from where you were to where you wanted to be?
Yükselten: Before SV, we couldn't do automation, since the other parties are in discrete locations and it was difficult to reach the other systems. We could automate functional test cases, but for end-to-end test cases, it was impossible to do automation.
First, we implemented SV for virtualizing the other systems, and we put SV between our infrastructure and the third party infrastructure. We learned the requests and responses and then could use SV instead of the other party infrastructure.
After this, we could also use automation tools. We managed to use automation tools via integrating Unified Functional Testing (UFT) and SV tools, and now we can run automation test cases and end-to-end test cases on SV.
Gardner: Was there anything about this that allowed you to have better collaboration between the developers and the testers? I know that in many companies, this is a linear progression, where they develop and then test, and it can be something that there's not a lot of communication on. Was there anything about what you've done that's improved how developers and testers have been able to coordinate and collaborate?
Yükselten: We started to use SV in our test systems first. When we saw the success, we decided to implement SV for the development systems also. But, we've just implemented SV in the development site, so I can't give results yet. We have to wait and see, for maybe one month, before I can reply to this question.
Gardner: Tell me about the types of applications that you're using here as a large Internet service provider. Are these internal apps for your organization? Are they facing out to the customers for billing, service procurement, and provisioning? Give me a sense of the type of applications we're talking about.
Yükselten: We are mostly working on customer relationship management (CRM) applications. We deploy more than 200 applications per year and we have more than six million customers. We have to offer new campaigns and make some transformations for new customers, etc.
We have to save all the informations, and while saving the information, we also interact the other systems, for example the National Identity System, through telecom systems, public switched telephone network (PSTN) systems.
We have to ask information and we need to make some requests to the other systems. So, we need to use all the other systems in our CRM systems. And we also have Internet protocol television (IPTV) products, value-added services products, and the company products. But basically, we're using CRM systems for our development and for our systems.
Gardner: So clearly, these are mission-critical applications essential to your business, your growth, and your ability to compete in your market.
Yükselten: If there is a mistake, a big error in our system, the next day we cannot sell anything. We cannot do anything all over Turkey.
Gardner: Let's talk a bit about the adoption of your SV. Tell me about some of the products you're using and some of the technologies, and then we'll get into what this has done for you. But, let's talk about what you actually have in place so far.
Yükselten: Actually, it was very easy to adopt these products into our system, because including proof of concept (PoC), we could use this tool in six weeks. We spent first two weeks for the PoC and after four weeks, we managed to use the tool.
For the first six weeks, we could use SV for 45 percent of end-to-end test cases. In 10 weeks, 95 percent of our test cases could be run on SV. It was very easy to implement. After that, we also implemented two other SVs in our other systems. So, we're now using three SV systems. One is for development, one is just for the campaigns, and one is for the E2E tests.
Gardner: Tell me how your relationship with HP Software has been. How has it been working with HP Software to attain this so rapidly?
Yükselten: HP Software helped us so much, especially R&D. HP Turkey helped us, because we were also using application lifecycle management (ALM) tools before SV. We were using QTP LoadRunners, QC, etc., so we had a good relation with HP Software.
Since SV is a new tool, we needed a lot of customization for our needs, and HP Software was always with us. They were very quick to answer our questions and to return for our development needs. We managed to use the tool in six weeks, because of HP's Rapid Solutions.
Gardner: Let's talk a little bit about the scale here. My understanding is that you have something on the order of 150 services. You use 50 regularly, but you're able to then spin up and use others on a more ad-hoc basis. Why is it important for you to have that kind of flexibility and agility?
Yükselten: As you say, we virtualized more than 150 services, but we use 48 of them actively. We use these portions of the service because we virtualized our third-party infrastructures for our needs. For example, we virtualized all the other CRM systems, but we don't need all of them. In gateway remote, you can simulate all the other Web services totally. So, we virtualized all the Web services, but we use just what we need in our test cases.
Gardner: And this must be a major basis for your savings when you only use what you need. The utilization rate goes up, but your costs can go down. Tell us a little bit about how this has been an investment that's paid back for you.
Yükselten: In three months we got the investment back, actually, maybe shorter than three months. It could have been two and half months. For example, for the campaign test cases, we gained 100 percent of efficiency. Before HP, we could run just seven campaigns in a month, but after HP, we managed to run 14 campaigns in a month.
We gained 100 percent efficiency and three man-months in this way, because three test engineers were working on campaigns like this. For another example, last month we got the metrics and we saw that we had a total blockage for seven days, so that was 21 working days for March. We saved 33 percent of our manpower with SV and there are 20 test engineers working on it. We gained 140 man-months last month.
For our basic test scenarios, we could run all test cases in 112 hours. After SV, we managed to run it in 54 hours. So we gained 100 percent efficiency in that area and also managed to do automation for the campaign test cases. We managed to automate 52 percent of our campaign test cases, and this meant a very big efficiency for us. Totally, we saved more than $50,000 per month.
Gardner: That's very impressive and that was in a relatively short period of time. Do you expect now to be able to take this to a larger set of applications, maybe beyond your organization, more generally across Türk Telekom?
Yükselten: Yes. Türk Telekom licenses these tools and started to use these tools in their test service to get this efficiency for those systems. We have a branch company called AVEA, and they also want to use this tool. After our getting this efficiency, many companies want to use this virtualization. Eight companies visited us in Turkey to get our experiences on this tool. Many companies want this and want to use this tool in their test systems.
Gardner: Do you have any advice for other organizations like those you've been describing, now that you have done this? Any recommendations on what you would advise others that might help them improve on how they do it?
Yükselten: Companies must know their needs first. For example, in our company, we have three blockage systems for third parties and the other systems don't change everyday. So it was easy to implement SV in our systems and virtualize the other systems. We don't need to do virtualization day by day, because the other systems don't change every day.