Lean and Agile at Sony Ericsson

Combining Kanban and Scrum

2008-12-07T20:47:00.008+01:00

Background

Some time ago the organization that my section belongs to was re-organized. We lost some resources and got some new ones. We also took over the ownership of a big internal tool that was previously belonging to another section. The other section had not been working according to agile and lean principles before.

We created two new teams. One was responsible for the new big system and the other team for the big test automation system that my section was managing before. We created the team so they both included some people that was used to working according to Scrum from before the re-organization.

Initial cleanup

Although we had been working to Scrum before we still had a huge backlog of old issues in our defect management system DMS. The new system also had a big amount of work in progress in the form of DMS issues. In total we had more than 600 issues than were not closed. The majority of them were in the first “submitted” state but we also had a lot in other states such as “postponed”, “rejected”, “investigating” etc. Our first priority was to clean up this into a more manageable amount. Doing this by Scrum would take a long time, so we decided to try a Kanban approach. We also felt that all the planning and time estimation used in Scrum would be a big waste in this case, since no one was interested in how long it would take. It was task that simply needed to be done.

We divided a whiteboard in the different states an issue can have in DMS. We created one whiteboard for each team. We then printed out all the issues through Excel as small paper slips. The slip only contained the ID, title and state of the issue. We then put all these slips into an unsorted queue in a box. We took 50 white board magnets for each whiteboard that we used as kanbans. We took the first 50 issues and put them on the board in the corresponding state. Then we got our hands dirty and started working. The only goal was to get the issues on the board to the closed state so we could free a kanban and put up a new item from the queue on the whiteboard.

We continued with lean and agile principles such as daily morning meetings, self organizing teams etc. The product owners were also heavily involved during this work, providing feedback regarding importance of issues etc.
It turned out that a lot of issues could be closed without fixing them. A lot of the submitters had long since left the company, the issue was connected to a project closed a long time ago, the issues was obviously not important, the issues was already fixed but not yet verified etc.
In the closed box we regularly stapled the slips representing closed issues into bunches of ten. We then made small packets of 50 slips of these bunches of ten. There was an immediate satisfaction when physically closing the issues by bunching them together and seeing the number of closed issues constantly growing.

The speed was amazingly fast. In six days we were down to 280 open issues. It felt like we had closed the majority of the no-brainer issues and it was time to really start prioritizing and reducing the amount of work in progress. It was time for the product owners to really make an effort. We divided the issues into our two main systems (let’s call them A and B) and a third group for our other small systems (that we will refer to as C). System A had 76 issues, B had 115 and C had 19. The product owner for A should reduce them to 30 issues, the product owner for B to 40 and I as a section manager should reduce the C issues to 10. The deadline was two days, so the only way of reducing was by prioritizing and closing non-important issues without fixing them. Basically telling the submitters that there issues were not important and never would be fixed.

The result of this cleanup was amazing. The two new teams started to work together very quickly as self organizing teams. We went from more than 600 to 80 issues in approximately two weeks. We fixed some of them but the majority was simply rejected and closed. The most amazing part was that we only got 2 customer complaints about closing their issues. This is a very good indication that our huge backlog actually was nothing but rubbish, and that we did a good job identifying the important issues and closing the rest.

Selecting between Kanban or Scrum

Excited by the speed and result of our Kanban cleanup effort I was ready to skip Scrum and start using Kanban fully for our development. Most of our work is maintenance so I thought it was a good idea. My teams and product owners didn’t agree and the developers working with Scrum previously was not keen at all on skipping Scrum. The main reasons were;

The teams really appreciated the iterative analysis and discussions for each issue. All doubts and uncertainties had been settled once the issues had gone through poker planning and sprint planning and when the developer started working with the task in the sprint they knew exactly what to do. They felt the iterative group analysis was lost with the kanban approach.
The majority of the team members appreciated the time box and deadline in a Sprint. They felt that having a manageable work package that should be delivered towards a clear deadline made them perform better.
The product owners liked the fact that they only had to spend one or two intensive days with the teams during poker planning and sprint planning and that the team then knew what to do and could work more independently. The product owners felt that the kanban approach made it necessary for the team to interrupt them constantly with ad-hoc questions for each issue.

Since the main idea with agile and lean development is that I as manager have full confidence that the team themselves know how to work in the most efficient way we decided to continue with Scrum. The only problem was that I wasn’t ready to let go of Kanban easily since I had seen how it really solved the problem with issues ending up in buffers in different postpone state, investigating state etc. I really liked the idea of having all non-closed issues on whiteboard where everyone could see them.

Combining Kanban and Scrum

The solution we found was to combine the two techniques. We use Kanban in order to control the value chain of our issues from submitted to closed. The team uses Scrum to organize their work once the issues are ready to be implemented. In detail it means that we have the following states in our Kanban system:

Submitted: Once an issues is put into our DMS system it gets the submitted state. This is the same as being in the Scrum backlog. The product owner prioritizes all the issues in the submitted state and might mix between DMS issues or other items that are supposed to enter the next sprint. We only use Kanban for the DMS issues. The rest is handled by Scrum.
Assigned: The issue gets the assigned state once they are in a sprint.
Test OK: When the issue fulfills the “Software DONE” criteria it gets the “Test OK” state.
Integrated: Means that the issue has been released in an official version of the system. It is now up to the submitter to verify that the fix is according to their whishes.
Verified. An issue ends up in this state when the submitter has verified it. We then close it.

There are also two additional “side-track” states.

Investigating: If the product owner needs more information from the submitter it is returned in the investigating state.
Rejected: If the product owner founds the issue irrelevant or erroneous it ends up in the rejected state and gets closed after a couple of days.

All the states have a maximal number of allowed issues, including the submitted state. If someone submits more issues the product owner needs to start to prioritize and reject. This makes sure that the amount of work in progress is manageable. Since the Integrated state has maximal number of issues that are almost similar to the Test OK state it and assigned state it means that we can’t start a new sprint if there are issues from the previous sprint that still haven’t been verified by the submitters. This guarantees that we don’t flood the organization with more fixes than they can assimilate. The rejected state doesn’t have a maximal number. The issues are instead closed after five days if the submitter doesn’t press the “Disagree” button in our issue management system.

The maximal number of issues is higher than in a pure kanban system (for example 20 for the submitted state and 15 for the assigned state) and the pace is slower, since the pace represents the sprint time.

We feel that this system combines the advantages of Scrum with the control mechanisms of Kanban that makes sure that the issues are really converted into business benefits. Before using Kanban on top of Scrum we had a tendency to only implement the issues and then forgetting to make sure that they were really deployed, verified by the submitter etc. Our combined system also makes sure that our backlog doesn’t grow out of control again. Basically the system helps us to avoid ending up with 600 junk issues again.

Handling external dependencies

One of our systems uses resources that are not a part of the team. We created separate states in their kanban flow for each of these external resources in order to make sure that the tasks doesn’t get stuck there.

Other positive side effects

Our support team got a bit jealous and started to use some of the lean and agile principles as well. They are not using kanban or Scrum but they are having daily status meetings and a whiteboard with ongoing tasks, difficult support issues that haven’t been solved yet etc.

Conclusions

Using a Kanban approach to clean up old messes is really efficient. It can be used for almost anything and the approach to draw up the flow of the issues to handled, printing out each issue as a paper slip and then using white board magnets as kanbans worked really well. Combining this with other good lean and agile principles such as self organizing teams, kaizen activites and daily short status meetings worked very well. It can be used for clean up any kind of old and messed up backlog.

Combining Kanban and Scrum gives us the best of two worlds. Scrum helps the teams to work in an effective way and deliver real business value and kanban helps the product owner and the rest of the organization to control the flow through the value chain to make sure that what the team has developed is really being used.

The seven wastes

2008-10-10T09:47:00.013+02:00

After having read the book "Lean Software Development, an Agile Toolkit" I decided to spend our following section meetings on discussing each waste and finding out how we could reduce them.

Partially done work

Our by far biggest heap of partially done work is our big backlog of not completed DMS issues. (DMS is Sony Ericsson's defect management system). We currently have 324 not closed issues for the tools that we are responsible for. We will try reduce that heavily, hopefully by a kanban approach similar to what David J. Andersson describes in his blog, although I think that we will try to combine it with our normal way of working with Scrum.

Extra process

By using Scrum for a long time we actually don't have a lot of extra processes. The team came to the conclusion that what we had was pretty necessary. We have some improvement possibilities in the following areas:

Cleanup of our intranet pages. They have some irrelevant and out-of-date processes described.

Remove our script delivery process. Our group previously had the responsibility to handle configuration management for test scripts used by on of our tools, but that responsibility has now been moved to our different test teams, so this process is totally redundant.

Extra features

We have a history and culture of developing to much nice-to-have features. We will try to solve that by including earlier customer feedback a lot sooner by mock ups, prototyping, workshops together with the end-users etc.

Task Switching

We found the following time consuming task switching:

Part time scrum master. Our Scrum Master also takes part in one development team.

Mixed Sprints. We sometimes have sprints with mixed tools and goals since we have many small tools.

Ad hoc tasks with high priority, for example support issues. Our development team handles the support mailbox. Although we plan sprint time for it this still means task switching.

Helping ccolleagues

Meetings

E-mails

Introducing FxCop into existing systems

2008-10-03T14:52:00.009+02:00

Why code review?

I consider code review one of the most cost effective quality increases available for your code. It is easy, cheap and increases the communication within the team as well as each team members knowledge about other parts of the system. It also gives a feeling of joint responsibility of the code and creates a lot of creative and good conflicts between the developers.

The problem is that the simple code review, such as controlling that everyone follows the coding guideline, writes correct headers for the file etc. is boring, repetitive and not a task suited for an intelligent human being. Having a completely automated code review is every developers dream, especially if they have a manager like me forcing them to do code review. It would allow them to use their intelligence to focus on reviewing the design, functionality and architecture of the code, instead of the syntax, naming conventions etc.

Introducing FxCop in existing systems

We decided to use FxCop for our automated code review because it is free and you have the possibility to write your own custom rules. We were facing the same problem as many development organizations, that is a number of old and already existing systems in which we would have to spend a couple of thousand man-years fixing all the FxCop rules violations if we would turn on all the rules. That would hardly add any value for our customers.

We decided to make a review of all the existing FxCop rules in order to select the ones that we found applicable for our systems. We felt that we could skip a lot of them, especially a lot of the security rules since we write systems that are only used within the Sony Ericsson Intranet. We also defined some custom rules that we wanted to create, such as naming convention rules. We wanted to do this for two reasons; getting rid of the most repetitive naming convention checks during code review and to get knowledge about how to write custom rules.

We then created a list of all these rules and made a plan to turn them on, a couple at a time, for each sprint. We felt that would give a good trade-off between increasing the code quality without affecting the value-add for our customers too much. It also fit very well into our continuous integration concept, since we could use a copy of the FxCop configuration file for our different systems, having all the rules already defined, only turning them on a couple at a time.

Turning on rules in systems currently not under development

Another problem we faced was how to handle systems that was not currently under development, but never the less were part of our continuous integration setup. We didn't want to spend time on messing around with working code just to fix FxCop violations if no other work was supposed to be done on the tools, but on the other hand we didn't want to get mismatches between the quality levels of our different systems, meaning we didn't want to have a set of well maintained up-to-date systems that didn't violate FxCop rules and another set of old shabby systems that never been submitted to FxCop. The solution we found was;

Each time a rule is turned on as a part of a sprint of an active system, the rule is also turned on temporarily for all the other systems in order to see how many violations that rule would cause in the in-active system.

An error report is submitted for that system in our issue management system (DMS) just stating the rule and how many violations it caused.

As soon as any development is being done on one of the in-active systems the FxCop issues for that system are fixed as a part of that sprint.

This solution makes it possible to plan for the FxCop rule fixing as a part of the sprint and we get a continuous quality improvement for all our systems.

FxCop rules for new systems

Just recently we had the opportunity to start from scratch with a totally new system, which is not a very common situation for us. We often spend our time on maintaining old already existing systems. We were all very exited about the possibility to try our new agile way of working from scratch, but we found some interesting surprises. Our initial idea was to be very rigid and do full FxCop, unit test and continuous integration from the very start, but it turned out to be very slow and troublesome for the team. In the early stages, when they were doing a lot of prototyping, deleting files, creating projects etc. the continuous integration server was screaming constantly. If you have read my previous blog postings you already know that we have a speaker system connected to the server, so I really mean screaming. The developers ended up spending way to much time on fixing FxCop violations for code that was never going to be used in the first place. During the sprint the developers decided to turn of FxCop during the first early trial-and-error phase of a new system. During the sprint retrospective they decided to make this a rule and always start brand new systems with a grace period without using FxCop

Lean and Agile Tools

2008-10-02T15:56:00.007+02:00

This artikel is about what tools we use.

ScrumWorks Basic

We use ScrumWorks Basic For our product backlog, sprintplanning and burndown charts. The burn down charts of our two teams are displayed on a central monitor.

CruiseControlNet

We use CC.Net for our continous integration. We use continous integration for all our tools and projects. We currently have 12 projects that are being integrated and built continously.

ClearCase

We use clear case as our version handling tool, connected to CruiseControl.Net

DefectManagementSystem

Sony Ericsson has their own defect management system, called DMS, which is based on ClearQuest.

FxCop

We have a constantly increasing number of FxCop rules that are triggered by the continous integration.

NUnit

We use NUnit for our unit testing. It is also triggered for each build by Cruise Control.

All our tools are connected to a build server. A subwoofer is connected to the server and makes a horrifying scream in our office space whener a build, unit test or FxCop rule fails.

I will post other articles on this blog going through each tool more in detail and how we work with them

Scrum Failure modes vs. Five dysfunctions of a team

2008-09-26T10:40:00.005+02:00

My ScrumMaster was attending a certified scrum master course at the same time as I was reading the book "Five dysfunctions of a team" as a part of a management development program I was attending. My ScrumMaster got a list of scrum failure modes during his course and I think that the connection to the five dysfunctions was very obvious, so I decided to make a picture that shows the connection

I interpret the fact that the two theories describes the same thing in two ways:

Failing with implementing Scrum is often due to other reasons then Scrum tools or processes. It is a lot more likely that it is due to other team/organization issues
Scrum it-self will never solve basic team dysfunctions. If there is no trust between the team members or a fear of confligt these issues will have to be addressed at the same time as Scrum is being brougth into the organization

Introduction

2008-09-25T10:44:00.002+02:00

This is the first post on this blog and the intention is to give a short introduction to me and why I'm writing this blog.

I work as a section manager for a development section at Sony Ericsson. We develop internal tools used mainly by the software development organization within the company, such as test automation tools, test management systems and issue managament systems. We develop mainly using C# .NET.

I got in touch with Scrum about one year ago and we have worked according to that ever since. We had our share of problems in the beginning, but sending our ScrumMaster to a course solved a lot of them. I have recently started to put more attention to the lean and agile parts of our development after having read the book "Lean and Agile Software Development". We are right know going through the seven wastes at each weekly section meeting.

My intention with this blog is to share our progress and experiences as we become more effective and efficient in our work