This year has been a great year for the cloud, for Microsoft, and for SSW customers. We’ve delivered many solutions built on top of Azure, CRM Online, SharePoint Online, and TFS Online – all running on Microsoft’s cloud infrastructure. These solutions work across devices. They are easy to manage – you are set up in virtually no time and they are enterprise-ready, and we’re able to offer substantially more for much smaller costs.
On the consumer front, Microsoft will look back and see 2013 as the start of the fight for the phone and the tablet.
Building on the Microsoft Stack: I’ve just come back from a couple of weeks at the MVP summit in Seattle (I love this event – I was MVP of the year in 2011). As a Microsoft Regional Director and ALM MVP, as well as Chief Architect of SSW, I enjoy keeping my finger on the pulse of Microsoft and its many product developments. Since my main job is helping companies adopt and deploy Microsoft technologies, my roles at Microsoft are very useful for my customers and staff. Therefore with all the new releases in 2013 it has been a busy year.
Internally at SSW we have implemented the latest upgrades across the board: all our PCs now happily run on Windows 8.1, our phones now use Skype and Lync 2013, the developers are making good use of Visual Studio 2013 and TFS 2013. Daily we run the latest Exchange Server 2013 and SharePoint 2013. Next on our to-do list is to upgrade to CRM 2013, which has just been released.
We’ve definitely seen a shift by our clients from SharePoint development to stand alone MVC solutions that have some integration points with SharePoint, CRM and other applications via REST APIs. Our developers love using ASP.MVC 4 and 5 with either SQL Server 2012 or Azure. One of these combinations power 90% of our client projects.
There were 2 standout projects this year. Both embraced the technology, both used Scrum, and both brought complicated solutions to life. The first one was Toll Transport, led by Mark Liu, and the second was PwC, led by Eric Phan. There were many other projects also where I saw exemplary work built on the Microsoft stack.
My final project starting for 2013 is one I’m very excited about: FireBootCamp. FireBootCamp is an intensive training course that will cover technologies such as .NET MVC,
even a starter developer to a .NET hero in just 9 weeks! http://firebootcamp.com
Building SSW TV:
We have continued to work hard to make more and more presentations available to view on SSW TV. This video site, for which I am the Product Owner, is focused exclusively
on promoting Microsoft platforms and interviewing Microsoft experts from around the world, continuing the SSW tradition of sharing knowledge to help our industry create
better quality software. In the last 12 months I have personally created over 30 videos on SSW TV. During this period, our total views have jumped from 44,000 views to more
than 218, 000 and our subscriber numbers have more than quintupled from 295 to more than 1500.
I also continue to run the Sydney and Canberra .NET User Groups every month to ensure best practices are identified and shared with the community. This year we’ve covered
a wide range of subjects from business law to Scrum, and have had some exciting international guest speakers such as Glenn Block from Microsoft and Jason Beres from
Infragistics. The final User Group of the year will be a crescendo featuring the handsome man Scott Hanselman.
Through SSW TV we make each Sydney .NET User Group a global event by live-streaming the meeting each month and then publishing the video online. No interested party is left out!
The TV team have matured and grown our channel extraordinarily this year.
In addition to client work, throughout the year I have spoken around Australia and at interesting international events. I also presented at my old favorite,
TechEd Australia and TechEds abroad, on topics such as:
“I feel the love again – Using Access 2013 and SharePoint 2013 together”
I know many of you always want to know what’s new with Microsoft and what’s cool in the industry so I will be putting together new rules and new presentations in 2014. Now’s the time to sing out and tell me.
I would like to thank our clients and all those who have sent me a terrific amount of feedback. Keep telling me what we could do better – my email address is adamcogan at ssw.com.au
I’m proud to be one of the technology leaders in Australia and I will continue helping teams work better together by being a Scrum Master, Scrum Trainer and a .NET
architect, working with TFS and implementing the whole ALM lifecycle.
I am proud to announce the launch of the best training course we have ever done. It takes a beginner .NET developer to a *Job Ready* world class junior developer in 9 weeks.
Long story short, they get thrown into the fire, continuously mentored, while building a real world app. They will successfully complete 4 sprints. I called it FireBootCamp.
I’ve approached this very differently from all the training SSW has done for many years for companies around Australia. It has aspects of the best practices we’ve learnt presenting at conferences around the world and lecturing at the University of Technology Sydney for many years. We all learn better by doing, and working with others. The successful applicants become “FireBootCampers” spending 40-60 hours per week for 9 weeks. They will learn the right way to do .NET enterprise applications using a series of generally accepted patterns and practices. Most teams will build an ASP .NET MVC app all the way from concept to deployment.
No one will get stuck. No one will go down the wrong path. They will be mentored by an SSW Solution Architect each day during the program and we believe this is one of the keys to success. Having an experienced architect help them get over the hurdles when they get stuck, teaching them on best practices and pushing them to work hard and work smart, will turn these new recruits into job ready coders.
Employers: At the end of the program we welcome you to join our Gala night where the students will show off the apps they have built. You will have the first opportunity to add a “FireBootCamper” to your team. So if you are looking to hire a passionate junior world class developer, then register at http://www.firebootcamp.com/employers/ and be a part of the Gala night on 15th February 2013 at 2-6pm in Sydney, Australia. (If the demand is there, we may live stream the event.)
Job Seekers: If you you want to be part of the action, build a great app and get great experience, become a “FireBootCamper” for the next 3 months.
The first boot camp starts 16th December 2013 in Sydney. Get in early for a special half price introductory fee.
Quizzes: We wanted to provide more value to the viewer and give them an opportunity to reinforce the key points of the video. Now you can take a short quiz. Go ahead and test your knowledge!
TV Format: We have finally ditched the blog format. See how the content is now more accessible in our new TV-friendly format with the playlist down the right, showing the newest 16 releases. I preferred the scroll stopped when it got to #16, but I lost that argument and it now revolves endlessly.
Navigate by Technology: Say you want to see all the SharePoint videos? Now just click “SharePoint” in the new section on the bottom right “Navigate by Topic”
Figure: New SSW TV Video-style Design
With this new look and feel, you will notice that we have moved away from the old blog format (below).
Figure: Old SSW TV Blog-style Design was just a blog
At my Teched talks this month (Australia and New Zealand) I talked about Product Owners. After my session there were lots of discussion around the Product Owner Scrum role, and what makes a good one. I loved hearing the stories of dysfunction about Product Owners who have not done their job well.
The client is generally the Product Owner (PO). They should read the Scrum Guide, watch my new Product Owner video and communicate an understanding of their role. It is so important to the success of their project.
Order the Product Backlog. The important things will be done first, in order to maximize the ROI as the budget will run out one day.
Be available, at least remotely, to unblock a developer if he has questions/impediments. A good PO has a feeling of urgency.
Ideally listen in on Daily Scrums. This is optional but means that the PO will have daily insight into the team’s progress.
Understand Product Backlog Items (PBIs) and be able to explain what they want using Acceptance Criteria. This is the main way that developers and POs sync their understanding of what needs to be done.
Agree on a Sprint Goal for each sprint.
Not influence (or anchor) developer estimates with comments like “this one will be easy” and allow the team to come up with converged estimates.
Respect the Sprint Goal. Understand that the team will only work on things in the Sprint Backlog and don’t expect other things to be done as well. Most things can wait for the next sprint.
Who should be the Product Owner?
It’s hard to give guidance on who in the company would make a good PO. The usual candidate is often extremely busy. It should be:
1. someone with a personal stake in the success of the project.
2. someone who is available
3. someone with a clear vision of the product,
4. someone who has authority with budget
e.g. They could authorise adding a designer to a sprint for a couple of days
5. someone who has read the Scrum Guide and watched the Product Owner video and understands the role.
It’s possible to outsource the role of PO to someone in the development consulting company, but this is not recommended. Don’t put the fox in charge of the chickens.
During my session I said:
“Most dysfunction I see in Scrum teams is caused by a bad Product Owner”
This statement is based on my experience. I wonder if it rings true of what *you* are seeing? Let me know in the comments…
Using the right indexes is one of the most basic concepts when performance tuning SQL Server. Everybody knows you cannot touch the TFS SQL database directly, so they incorrectly assume that they can’t performance tune the indexes.
Before making any changes we measured the execution time 3 times to get an average.
We tested with a load of 100 concurrent users, plus 1 user setup with a collector to do the measuring.
We set the IIS setting for ‘Maximum Worker Processes’ (see yellow highlighting on image below) to “7″. (Use the calculation in Lesson 6 below)
There were 2 custom fields that the users were searching often. After adding the 2 indexes, the performance improved about 3 times.
‘Maximum Worker Processes’
Test 1 (s)
Test 2 (s)
Test 3 (s)
TFS field index = off
TFS field index = on
Figure: The performance improvement by enabling TFS field indexes was huge
We enabled a field index by executing the following command:
Lesson 4: Avoid virtualization in an *extremely* demanding production system
Hyper-V is very useful for optimizing your server infrastructure, however we’ve learnt you should avoid using Hyper-V on the data tier, because the Disk I/O overhead it introduced affected our performance significantly.
In this project, the 1st step we did to boost the performance was to optimize the data by moving the data tier to dedicated hardware that exceeded the recommended system requirements for maximizing SQL performance. The gain we got was roughly double.
Note: We first tested Hyper-V Pass-through disks prior to deciding that dedicated hardware was substantially better. That said, Pass-through disks performed better than VHDs (fixed or dynamic), but were still not good enough.
One of clients requirements was for one of their large busy systems, to call TFS and create work items. The 2 systems needed to stay in sync.
IIS by default has limited number of concurrent requests it can handle, the rest are put in a queue. Many long running transactions mean many things in the queue.
“Web gardens was designed for one *single* reason – Offering applications that are not CPU-bound but execute long running requests the ability to scale and not use up all threads available in the worker process.”
Why is the TFS application different from most web applications?
We are not sure why it falls into the 1%, but my understanding is TFS uses a dynamic data schema (Under the covers, they have a long, narrow table that stores dynamic column values as rows that then need to be pivoted to create a short and wide dataset), which makes the SQL queries very expensive, especially when you customize your process template a lot, e.g. adding many custom fields like we did.
Figure: This is IIS under a heavy load. These are queued requests in the IIS worker process (notice they are all in the ExecuteRequest state). Anything that’s over 50 is long and we have one that is an incredible 125 ms. Therefore long running transactions like this benefit from Web Gardens.
There are another 2 conditions you need to meet before you enable the Web Garden:
You are not using any InProcess sessions
Your service is totally stateless (this is usually the same as 1, but not always)
Fortunately, the TFS WebApp meets these conditions. And this is why we can use NLB (Network Local Balancing) on the AT (Application Tier) of TFS without any extra configuration.
The last question is, how many processes should I allow?
This is a tricky question and I wish we could give you a simple answer. In our tests, we found it depends on many factors, such as physical hardware, the application and even the request patterns. As a general rule, you should never set it above the number of CPU cores that you have on your box. This is because the overhead of switching between CPU cores kills any performance gains.
Therefore our IIS AppPool configuration was set at maxProcesses = (number of cores) – 1… in our case that equalled “7″ (as we had 8 cores)
Figure: We found increasing the setting of IIS’s Maximum Worker Processes to “7″ gave us the best results.
Lesson 7: In your integration code, find the performance bottleneck using Visual Studio Profiling (on the client side)
You may notice until now, we haven’t really touched the application yet, either the TFS WebApp or the application we developed.
The general rule when you are trying to improve performance is to focus on what gives the best ROI.
In this case, tuning indexes can improve an application sometimes more than 10x, but you cannot buy 10x hardware. So you should always tune your index first, then hardware and the last thing you want to touch is your code.
We used the Visual Studio Performance Wizard to analyse our code. It allowed us to identify the bottleneck quickly…
Figure: Run the Visual Studio Performance Wizard (in VS 2010 and VS 2012)
Figure: Or choose the “Performance and Diagnostics” menu in VS 2013
Here is the 1st report we saw:
Figure: Our 1st performance profiling report identifies the bottleneck is NewWorkItem()
You can clearly see the most expensive call in the chart is NewWorkItem(), so we dug into this method and found out it it’s actually the object initialization that costs the most. However, we cannot optimize this object initialization because that’s a core object. So we decided to implement an object pool to cache enough pre-initialized objects when the program starts, so then it’s just consuming it from the pool. Once the code was changed, this reduced the time by 22%.
Lesson 8: In your integration code, find the most expensive Web Service calls to the server (enable TFS tracing)
In lesson 6, we measured the TFS Object Model APIs, but we still needed more optimizations. So we dug deeper and found the most expensive calls to the server.
We enabled tracing on the client side and generated the following log.
Figure: Tracing TFS Web Service calls show the most accessed methods, which was our starting point
We found out the most expensive Web Service offender was GetMetadataEx2(WorkItemTracking) which is related to NewWorkItem() method, and this proved some of our suspicions before (in lesson 7). The TFS Client Object Model is reliant on some local cache to store the work item schema information, so it doesn’t have to retrieve this information every time. This confirmed our object pooling solution (in lesson 7) as a way to great benefits.
More info: the configuration to enable this tracing is below:
Object pooling on the TFS Object Model API (as per lesson 7)
Directly invoke the TFS Web Service endpoint
And they got similar improvements. After some debate, we chose Object Pooling as it was the safer choice since it is based on the documented API.
So in the end TFS holds up well under a massive load. The success of this project depended on going through a logical process of divide and conquer. We confirmed that the maximum benefit, for the least cost was still Index Tuning. But in very large systems there is much for you can do to squeeze out significant performance improvements. All in all, I am very proud of the results and I can say the project was a success, the client was happy and our SSW team learned heaps.
Recently Lei Xu and I completed a hair-raising TFS 2012 project. We hit some snags trying to optimize Work Items with 20,000,000 records. Let me tell you the story…
It was completed shortly after arriving home from the MVP Summit in Redmond. It was lucky we were full of information from Brian Harry and his team. This job turned out to be one of the most challenging that I’ve ever done, pushing the performance limits of Team Foundation Server 2012 (these tips apply to TFS 2013 and 2010 as well).
Figure: Brian Harry and Lei Xu @ MVP Summit
First some background: the client runs one of the biggest development teams in the world. They have over 20,000 developers and have a lot of experience gathering, analyzing and acting on performance metrics acquired while testing software prior to wide scale deployment. The system we needed to implement and customize had to cope with a massive number of concurrent requests, of course in very timely fashion. We used the TFS Integration Platform and the TFS Object Model to implement most of the functionality required.
However, like every story, things will never run as you expect. Once the coding was done, with all the data access, business logic and interface implementations on top of TFS Object Model, it was time for the 1st performance tests.
The initial results were disappointing:
(Before tuning) Red if target missed
Figure: The initial performance results were disappointing – some being 5-6 times worse than the target
It was powered by a great beefy SQL Server, so even though the TFS collection database had 20 million work items in it, I was shocked.
But also like every story, there is a happy ending, so here is the result after our tuning:
2500 ms (single server) or 1500 ms (NLB)
Figure: The client was happy with the results, we made our target in each case. That said, I think we were pushing TFS limits
There were many lessons that we learned and many people who helped. Let me summarize the lessons.
Lesson 1: Team Work
I put team work as the top one, as great software development is never a one man job, especially when you are dealing with a complex system. This system has many moving parts, including lots of performance tuning to TFS 2012, SQL Server, Windows Server 2012, IIS 8, the majority of TFS web services and the TFS Object Model. We were lucky enough to have experts for each of these parts and when put together, we achieved our goal.
During this project I got help from guys at Microsoft, my colleagues at SSW and couple of MVPs around the world. These included Brian Harry (Microsoft Technical Fellow, father of TFS), Aaron Hallberg (TFS DevTeam), Tiago Pascoal (ALM MVP), Ramesh Rajagopal (DevDiv from MS Dev Center), Julia Liuson (Manager of TFS DevTeam), Yongming Yi (MS Technical Specialist) … and more.
In short, if you want to do the job right, you need the right people. Having such a great team was essential for the end result.
Lesson 2: Performance testing should be done as early as possible
We used Scrum for this project and we built in unit tests and load tests from the very 1st sprint. One large impediment we had was the hardware. We didn’t get the right hardware until end of the project.
The results from the initial performance tests were poor. Thankfully this was not a too big of surprise for the client because one benefit of Agile methodology like Scrum is being transparent. This transparency led to understanding from the client.
The other main benefit of implementing performance testing early was that we had enough time to contact helpful people to gain support.
As you see the first 2 lessons are really not technical lessons. My next blog post will cover the technical lessons…
Diagnosing issues with your web app can be one of the most difficult things for developers. Sometimes, the browser just doesn’t show you what you expect. Visual Studio is the best possible IDE to help developers see what’s going on with their code during development, but what about what’s happening in the ASP.NET framework? And what about after it’s been deployed?
Brian Harry recently blogged about some great new features coming in Visual Studio 2013 and Team Foundation Server 2013 and I want to address the three aspects that affect our customers the most: TFS support for Epics, the new Team Explorer window in Visual Studio, and Cloud Load Testing.
1) TFS Support for Epics
Brian Harry called this “Agile Portfolio Management” but it’s really about supporting large pieces of work that represent what many agile guys call Epics (of course Scrum white-robes don’t have Epics, they just have large PBIs:) ). They could be big collections of User Stories or even high-level goals.
In the past, some of our guys have changed Team Project Templates for our clients to support these larger groups of work, so it’s great to see this make it into the product.
Figure: Epics in TFS 2013
2) New Team Explorer
Now for some developer love. Microsoft has improved the UI of the Team Explorer window in Visual Studio 2013. It’s a much cleaner view that matches the Windows 8 “Modern UI” style. You also get a list of all the solutions in your current workspace, making it very fast to open your projects. You’ll be pleased to know there’s a bit more color as well.
Figure: The new Team Explorer in VS 2013
3) Cloud Load Testing
As many of you know, SSW is very big on continuous deployment and of course we care about load testing. Web Access now has a Test tab that lets you manage test plans in the browser (without needing to open up MTM). This is great, but the really exciting part of this announcement is the cloud load testing service. Instead of setting up your own infrastructure to run load testing, you’ll soon be able to point your load test at the cloud and let it run!
For our high traffic sites, we currently use LoadStorm. We expect many of our customers to switch to this.
Figure: TFS 2013 Cloud Load Testing
There are plenty more features in this announcement, so have a look at Brian Harry’s blog post for more information.
It feels like only last year I was showing off Visual Studio 2012 at Tech Ed. Oh that’s right, it was last year! Microsoft are getting faster at shipping, not slower.
I’d like to congratulate Brian Harry’s Visual Studio ALM team for not only giving us two updates packed full of goodies for VS 2012, but having the bandwidth to deliver us Visual Studio 2013 as well! They’re on a very fast release cycle these days which is really impressive.
Yesterday I told you about the lightening fast, dead sexy movie site, ‘Event Cinemas’ that went live. Today I have a real gem from SSW TV. This video shows a little of how that sausage was made and hopefully you will get a great takeaway, that you can use in your own Scrum team.
One thing that is practically universal in our industry is that we let Product Owners get away with far too much. Product Owners are typically extremely busy people and I believe that most dysfunction that happens in Scrum teams comes down to the Product Owner. Letting a Product Owner skip estimating the “Business Value” does *not* make developers work better together.
So how do we estimate Business Value?
Well, we know that developers love the Fibonacci sequence (1,2,3,5,8,13,21), but the last thing you want to do, is teach Product Owners about how hip Fibonacci is, because it is slightly complicated and unnatural to those non-mathematically minded.
Therefore these days I much prefer the “Doubling” method (1,2,4,8,16,32) as it is far simpler for Product Owners to understand. Even better, I prefer that the developers switch to estimating their “effort” using the same scale, then things are much easier all around.
So come on and lets get this happening in all Scrum teams!
I propose the following:
1) Let’s make ‘Business Value’ *not* optional, and explain to Product Owners that if it’s good enough for a developer to estimate “Effort”, then it’s good enough for the Product Owner to estimate “Business Value”.
3) Once you have had a good experience using it, then tell Ken Schwaber and Jeff Sutherland that “Business Value” is really important and we want it in our beloved The Scrum Guide. And it wouldn’t hurt if it was also in the Terminology section of their book, Software in 30 Days.
Of course, many Scrum teams don’t bother with “Business Value”, so why should you?
There are plenty of benefits, starting with:
Helping you order your backlog
Finding the low-hanging fruit (easy tasks that give high value) and
If you have comments on why you like, or dislike, “Business Value”, let me know.
A big thank you to Robin Maes, Andrew Turner and the whole Event Cinemas team who worked tirelessly with the SSW’s guys to deliver a great looking, super fast, site that was on time and to the fixed budget.
Thanks to the Telerik Sitefinity team, our favourite Bulgarians, for giving us the updates with the fixes we needed.
Thanks to the SSW guys too, Adam Stephensen, Eric Phan, Daniel Hyles, Gerard Beggerlegs and Drew Robson who put in the hard yards.