Cosmos versus chaos. Brief guide to development methodologies that improve efficiency.
Software development is rarely linear. As the project moves along, the customer may want new features, or have new functionality requirements that conflict with the existing application architecture. If the team is unable to successfully manage those processes, it could cause chaos during development, and the customer will face sharp increase in development time and poor quality of work.
Companies that have streamlined internal processes solve this problem by using top organizational practices such as Scrum, XP, Lean, Kanban, Kaizen. As a result, planning the workload becomes easier, and the development work itself becomes more transparent and adaptable for both the team and the customer.
Next, I will expand on those methodologies and explain how to use them to overcome chaos.
Scrum
When you hear the word ‘Agile’, the first thing that pops into your head is, of course, SCRUM. Over the last few years Scrum became one of the most popular methodologies in the IT crowd. This project management framework is founded on clear and simple principles — the product is developed step by step, beginning with the easiest and progressing to the more complex tasks.
To organize all the requirements and features requested by the customer, the project has a backlog, which is reviewed on a regular basis. The development work consists of iterations, called sprints, and proceeds in stages. The goal of each iteration is to implement a specified set of features.
Plenty has been written about scrum; however, let’s review the key aspects.
Sprint
All iterations must begin with planning. Planning is necessary in order for the team to develop a collective understanding of the sprint goal. That helps define the effort and give it a group estimate that will be fair for the whole team.
The point of sprint planning is to complete all the non-development tasks during the initial stage. That includes setting deadlines, prioritization, evaluating the features and creating specifications. At the end of the sprint planning session all developers should clearly understand their tasks within this sprint. This is also the time to ask the customer any questions.
The rest of the time developers are implementing features without being distracted by organizational matters. It’s a simple equation: higher concentration = faster work + higher efficiency = fewer bugs.
The tasks in the sprint are always sorted by priority so the entire team has clear visibility into which tasks are critical and which aren’t. This is also a benefit for the customers — now they just prioritize the tasks rather than try to figure out who will complete them, and how.
Occasionally there are circumstances that need to be handled separately. In the event of a failed release or if a hotfix needs to be deployed, the sprint has to be adjusted. But if those situations don’t happen often, they can be resolved as part of the sprint.
Decomposition and effort estimation
One of the key factors of successful planning is competent estimation of the tasks based on time and resource cost. As a rule, large stories need to be decomposed — broken down into smaller pieces that are much easier to estimate.
If completing those smaller tasks takes longer than estimated, it is not a big deal: it is far better than having large unwieldy epics that are initially estimated at three days of work, but end up taking two weeks to complete.
Estimation challenges
The biggest problem with planning is to accurately estimate how much time and resources will actually be required for each task. Some developers tend to underestimate the effort, while others — overestimate it.
There are several approaches to planning and estimation. Scrum teams often use planning poker, a group discussion of features followed by each participant making independent estimates, not influenced by other members. As a result of planning poker, the developers reach consensus and any deviation from the estimated effort will not be significant.
For example, a strong developer with solid understanding of the domain will develop a feature estimated at two hours in an hour and a half. A developer that needs extra time to figure things out will complete it in two and a half hours. That is a minor discrepancy and it will be compensated for by the other tasks in the sprint.
Another way to estimate development tasks is by using so-called story points. Using that approach, the team estimates the effort not abstractly but in relation to some known unit of measurement — a story point. The goal of that is to minimize the errors in time estimates. One developer may estimate a task at two hours, another — at four hours. But both will estimate the task at 3 story points because they have a defined scale of those story points.
That estimation approach is useful not only for planning individual sprints, but also for long-term estimates.
Personally I like the concept of adaptive planning, described in the book “Jedi Techniques”. The planning steps for this approach are:
1. Identify the workload implemented in a certain period of time
2. Calculate the number of story points
3. Build a graph for each iteration
Then the workload that was added during each iteration is written down, and the graph of added workload is plotted. The graphs of added workload and initial tasks for each iteration are overlaid.
Straight lines are drawn on each graph, and if these straight lines intersect, the work effort is heading towards completion. The intersection of these lines will signify the point of completion.
Of course, this estimate is approximate, not final. But in practice it turns out to be more realistic than hour estimates.
Overall, decomposition, planning and regular retrospectives (reflection on the past sprint) allow, within the Scrum framework, to not only flexibly plan and estimate the development process, but also to detect issues before they become impossible to resolve.
Scrum Limitations
Despite the hype, it is important to realize that Scrum, like any methodology, has its limitations. Sometimes utilizing Scrum is difficult, impractical, or unlikely to noticeably improve the efficiency and transparency of the development process.
First, there is a significant barrier to entry. To implement the Scrum methodology, the team needs a scrum master responsible for organizing all the processes. Without the scrum master, it is unlikely that the process can be implemented with no “casualties”. Additionally, the fate of the entire project rests on the scrum master’s shoulders, which is a very high degree of responsibility.
This style of development work may not be the right fit for some teams. Occasionally developers may find themselves spending too much time on operational activities, at the expense of the actual development work.
Sometimes the nature of the project itself suggests the use of other methodologies. If there is a need for something concrete with pre-defined requirements, it is better to use Waterfall. A classic example is medical equipment, which has clear specifications and work standards. In that case, flexibility is not needed. Alternatively, if the tasks are constantly created by different sources and there is no defined product owner, it is likely better to use Kanban.
XP (Extreme Programming)
XP is somewhat similar to Scrum — it also has regular, but shorter, meetings and iterations. All XP tools are intended for collective code ownership and rapid deployment.
Testing and continuous integration
The key concept of extreme programming is that development must be test-driven, and integration must be continuous. As part of the XP methodology, each contributor works off of the latest version of the code. Before writing any new functional code, developers write unit tests and run the existing unit tests.
Test driven development allows to
- understand the functionality of the existing code (due to explicitly defined inputs and outputs)
- verify that after modifications the code has retained its functionality in cases that should not have changed.
Thus, regression errors when using XP are virtually non-existent, which is a huge advantage of this approach. After implementation, each feature can be deployed without manual and regression testing. Since there are no conflicts, very few bugs occur, and creating test cases for each bug eliminates any chance of regression.
As a result of this approach, best coding practices are defined. The core principles of XP help create out-of-the-box uniform coding styles and ensure stable code performance.
Another advantage of XP is rapid customer feedback. At almost any given moment, the customer can view the current working version and provide comments. Even if the customer is far removed from the development team (for example, if it is the end user leaving a comment in AppStore), due to the short release cycle the feature will be quickly implemented and deployed to production.
Pair Programming
Another XP concept is pair programming — two developers working together on the same set of code. This approach helps avoid errors caused by lack of attention. It is also effective if the feature that needs to be implemented is complex and too difficult for a single person to wrap their head around. Pair programming is also useful if one of the developers needs to be brought up to speed.
However, due to difficulties of implementation, this technique is mainly used for training.
XP Disadvantages
Extreme programming can produce incredible results, but the approach is relatively difficult to integrate into the workflow. First, managing the entire process requires a strong team leader able to reach consensus with the team and properly organize the workflow. Second, continuous integration requires investments and infrastructure adjustments.
Besides, this approach is challenging to use in large projects with a large team, especially if the product is not split up into smaller sub-products.
***
Even though we describe Scrum and XP as agile, they are rather concrete (and thus semi-agile) implementations of higher-level principles. Therefore, after gaining an understanding of these methodologies as process improvement tools, it is time to introduce Kanban, Lean and Kaizen.
Kanban
I have already mentioned Kanban earlier. Essentially, it is a project management method that uses visualization and work in progress limits to manage development. In general, Kanban allows for more flexibility — it has no sprints and no story points. It does not require rethinking of the entire workflow and can be implemented on top of the existing processes. However, not everyone is ready for an approach that does not dictate the workflow but rather makes one think and provides tools for analysis.
When implementing Kanban, a Kanban board is created to visually depict the stages of all work items. The simplest Kanban board consists of three columns or sections (To Do, In Progress, Done) but the quantity and content may vary depending on the team or the project.
With Kanban, it is always clear what stage the project is in at any given moment. But sometimes this approach needs to be customized to assess the real state of affairs.
Example: if the team leader is the only person performing code reviews, his column on the Kanban board will always be very large. And the tasks will accumulate continuously. What is the solution? Code reviews may be assigned to strong developers, and certain restrictions created, for example, limiting the column to three tasks. Then no developer can move the task to review if three tasks are already there. The developer will have to decide how to resolve the issue, whether by kicking back other tasks, or by performing code reviews. The tasks will spend less time in one status and will move along faster. The efficiency and speed of development will increase.
So, what do we find? Essentially, Kanban is not a project management methodology, but rather one of the ways to implement Lean.
Lean
Lean is a methodology of lean manufacturing. At its core, it is a classic development approach aiming to reduce waste during the programming process, to understand the bottlenecks, and to identify processes that require optimization.
The main idea of the Lean approach is that developers should not only perform their work, but also think about optimizing common processes. For example, if developers spend a lot of time on tasks that could be automated, or if they recognize where the time is wasted (for example, testers are overloaded, causing a bottleneck in the process), then, according to Lean, the developers must analyze the situation and propose a solution to minimize that waste.
To streamline the optimization process and eliminate waste, Lean offers the concept of a “value stream” — a flow of operations required to produce a product, starting with the initial planning and ending with the delivery to a customer. Within each flow, the actions and operations that do not add additional value to the final result are eliminated.
Overall, we can say that Lean is a process of searching for an optimal methodology. If you embrace the principles of Lean and arm yourself with Kanban, then over time you may organically come to XP or Scrum, having determined the most effective duration of iterations, release process, etc.
Kaizen
Kaizen is the most comprehensive concept, philosophy aimed at continuous improvement of processes. Kaizen in development is a set of principles that need to be introduced to the team so that they are capable of working without strict supervision and the developers constantly take initiative to optimize their own work.
What does it look like in practice? Few examples from personal experience:
1. If a developer figures something out but no documentation exists — he or she will write the documentation (horizontal scaling of development skills).
2. A channel of communication with developers can be created, allowing to discuss problems they have identified in the processes, and to propose solutions.
3. I delegate a lot (of course, after training has been completed). Almost anything can be delegated. It makes development more open to modifications, and the developers are involved in the process, as opposed to just following orders.
4. Co-design is also a great tool for horizontal scaling. Additionally, it can serve as a training stage for developers prior to completely handing processes over to them.
If Kaizen is integrated correctly into base Scrum, all the developers will participate in retrospectives. Not only the team leaders and project managers, but the entire team will begin to manage their own processes and improve them. Over time, more and more issues will be resolved internally by the team. As a result, the development approach may shift to processes that are different from Scrum to fit the needs of the company.
***
In conclusion, I would like to say: always think about what you could improve. There are multiple tools and approaches that allow teams to work more efficiently. There is no need to reinvent the wheel — dig into the methodologies used by millions of companies and choose the approach that best fits your needs.
(by Boris Chernysh)
CSSSR is a large outsource team of IT-engineers working hard yet completely remotely across the world. We specialize in front-end development of high load web applications, including online banking systems, payment interfaces, sales and brokerage platforms, customer portals for airline companies, and websites for mobile network operators. Connect with us here.
