For the last 18 months our team has been working to rebuild an existing system using entirely new technologies, methodologies, and a new development team. My company, Hindawi, developed a platform in the mid 2000s to allow academics to peer review academic papers online. For 10 years, the system performed admirably, helping the company grow rapidly as a publisher of academic journals.
The platform has ballooned in complexity over time, becoming a labyrinth of tightly coupled services. More recently, we began struggling to keep pace with new technology. We passed up attractive business opportunities because of the difficulty of integrating with or adapting the aging system.
After assessing our options, we decided to start from scratch and develop a new platform.
While this felt like a daunting task, we found several ways to break things down to make it feel more manageable. I’d like to run through some of our challenges, and how we got through them along the way…
Challenges
1. Starting Point
With such a large and complex system, it was very difficult to know where and how to make a start.
Result: A traditional model of MVP development pictures a skateboard evolving through a bicycle into an automobile. When you already have a car, it can be tricky to figure out how to get your old skateboard back out. It’s also a challenge to ask users to switch back to a skateboard once they’ve got used to the car experience.
The MVP had to be a simple, but it also had to capture at least one complete end-to-end workflow for our users, otherwise it would not work as a replacement to the existing system. We wanted the MVP to feel close to feature parity for our users, even if it was far simpler under the hood.
We invested heavily in defining what the MVP should be. We asked users and stakeholders what “fancier” features they could live without. We stripped out any integrations, reminders, or automations that could be handled manually. Our starting point was anything that was necessary for the core workflow of the system – and anything else became secondary.
Learning: Understand the core of your product. MVP can mean different things for different “new” products. Once we had worked out what we needed in order to launch, priorities for development became much clearer. It was worthwhile investing upfront in defining this stage of the product.
2. Long-term vs Short-term
Why spend time and money rebuilding a thing that isn’t actually broken?
Result: This was a strategic debate, involving key stakeholders across the business. The rebuild process would be expensive and disruptive in the short-term. But to achieve our long-term strategic goals, we required a flexible system that would allow us to react to business changes, something the existing system was not able to do.
We were lucky that we started the rebuild process early, before we faced a crisis with the existing platform. This meant we were able to rely on the existing product while developing the new one. It gave us time to prove what could be built using the new technologies, rather than having to demonstrate immediate returns. It allowed us to put off migration and integration questions. It gave us the ability to test and compare the new systems and validate the improvements we were seeking.
Learning: Start the project early. Short-term needs can cloud decision-making, so it’s important to understand long-term business objectives before entering into a project of this scale. We had to make some short-term sacrifices to achieve our long-term goals. We went to great lengths to ensure that everyone understood the reasons why.
3. Rebuild or buy?
There were options for us to switch to a commercial off-the-shelf solution that would have satisfied the majority of our needs.
Result: An out-of-the-box solution, while adequate for our own needs, did not provide us with the adaptability for new business opportunities that we wanted from this system. After many years of controlling our own development roadmap, we felt it would be difficult to give up the flexibility of owning our own platform. We wanted to control our own plans, and not be reliant on an external partner to meet our requirements. One of our goals as a business was to make more of our products open source, which a commercial product wouldn’t facilitate. We also felt that in our case, the full cost of ownership for our own system was not significantly different from the cost of a commercial platform.
Learning: You have to do your research here. Consider all the costs of maintaining a platform, including salaries, development, and hosting. But also consider the costs of implementation, support, or custom changes with a commercial product. Consider the costs to the business of slower release cycles and limited input into the product roadmap. When we assessed our long-term ambitions, and looked in detail at what the other solutions offered, it became clear that rebuilding our system was the right approach.
4. Existing Team vs new Team?
Will your developers transition smoothly from working on a legacy product to a new platform?
Result: We took a step back here, and reassessed what we were trying to achieve from a technical point of view. We wanted to be part of an open source community. Once we had committed to that, that led us to look for projects in our space with momentum. We had to weigh the vision, quality, size, and stability of the community alongside the technology choice. We knew how we wanted to approach things, and ultimately we went with the best community available, rather than sticking to our legacy technology. Since we were changing technologies, the only way forward was for us to build a new team.
However, this had unexpected benefits, We agreed that the current product team would continue to maintain the existing platform without changing their roadmap. A new, external product team would start development on the new platform, working as part of the open source community.
By keeping the new team independent, we prevented them simply migrating over parts of the old platform to the new product. They avoided being trapped by legacy design decisions. The old team was able to stick to their current development roadmap, which minimised short-term disruption for users. It gave the new team the freedom to experiment and take risks.
While we knew this would be a financial and administrative challenge, we embraced the benefits of working with a team that had fresh ideas and was not tied into existing ways of working.
Learning: Have teams working in parallel. This approach is expensive, but less risky, and more flexible. The process ended up being refreshing in many ways, and made us question what we had been doing. We discovered new and efficient ways of working that now seem obvious. We also felt that approaching the problems of our industry with a viewpoint from outside the industry has helped to identify areas we could focus on to build in a competitive advantage.
5. Stakeholder Buy-in
When attempting to transition an entire business from one system to another, you’ll be under a lot of scrutiny.
Result: After a decade, we had a huge amount of knowledge of the issues with the current system and a long backlog of feature requests that had been building up over the previous 18 months. This backlog was an extremely valuable resource but it was also challenging. We knew that teams were putting a lot of faith into the idea that the new system would be the solution to all life’s problems.
We managed this by carefully going through and working with each team to help them understand the timelines and expected outcomes of the new development. We were very careful not to make any unachievable promises, and to ensure that what we were building was taking into account the existing backlog. We also worked with each stakeholder group to help them go from thinking “how could the old system to be fixed” to “how would I build this properly from scratch.”
We set up regular meetings to ensure key stakeholders were kept up to date with development plans and timelines, as well as getting input on prioritisation. We used our communication tools (Slack, in this case), to share wireframes, designs, and questions at all stages directly with stakeholders. This helped us to ensure our initial release was something everyone was comfortable with.
Learning: Communicate early and often. This sounds easy… but it’s also very easy to forget about it!
Today, we are six months past the first release of the new system, we have a working peer review system in place, and we are expanding the functionality of the new system to add back in the layers of complexity we removed. We plan to replace the old system completely by the end of the year.
The project has been successful because we set ourselves realistic targets, and we obsessed over prioritising the core features of an initial working system. This allowed us to validate the benefits of the new system, and encouraged long-term thinking and patience from stakeholders. The challenge we now face is working out whether features in the old system are useful enough to be rebuilt into the new system, or whether we should do things in new ways. The challenges, just like the system, continue to evolve.
Comments 0