Make the best use of your production or commercial resources

The typical concern with customized solutions is that it takes very long to build and might not be reliable. We have addressed these issues across our analytics projects with curated development protocols and well-trained people. In fact, preventive maintenance activities are at the core of our standardized development processes and are embraced by our team of scientists.

While we diligently collaborate with our clients to validate our solutions, our primary effort goes into avoiding bugs and mistakes. Specifically, most of our proactive activities fall into three categories:

• Implementing automated testing: The standard way of increasing the reliability of software is to write automated testing, typically unit testing and regression testing, that covers, let’s say, 95% of the code. A new release of the solution must then pass all these tests. This is not a common practice among most operation researchers and data scientists. But any serious software developer will have a ton of incontestable arguments for not skipping testing. We take those arguments very seriously.

• Using standardized components: Even when building a solution from scratch, we typically reuse several already optimized and tested substructures. Re-inventing the wheel is not only inefficient but also increases the chances of making mistakes. So we save our creativity for when it’s needed and prioritize reliability otherwise.

• Using development protocols: We have established well-defined development protocols that we are continuously improving and training our scientists to use. In other words, we also ensure reliability with our development processes, like lean production in manufacturing. In addition to reliability, development protocols lead us to build maintainable solutions and promote the day-to-day adoption of best practices among our collaborators.

Sometimes we reuse well-validated blocks of code, which speeds up the development process significantly. But quite often we build solutions almost entirely from scratch to fully address business requirements and deliver top-performance engines. Even then we can deliver end-to-end solutions in weeks because we are constantly training ourselves to apply standardized development protocols across our projects.

Using existing components and protocols does not restrict the possible outcomes and allows us to focus on what adds value to our client's businesses. For example, we don’t have to waste time choosing the best documentation approach, defining repository structures, implementing code to validate input data, or designing pipelines to test the solution. Instead, we use the same patterns and templates that we have generalized just enough and optimized over time. We get quicker and quicker as we repeat the same process project after project. As a result, we achieve:

• Standardization that promotes effective communication and efficient collaboration among scientists and IT professionals, even across projects;

• Significant development speed-ups and, still, much more reliable solutions;

• Maintainable solutions that can be further expanded to address new requirements as the business problem evolves.

On the delivery side, we have our own platform for the development of web applications, Mip Hub, which allows us to convert our Python packages (the standard way to develop professional software in Python) into custom web applications. Because of the way Mip Hub is designed, our decision scientists deploy enterprise-grade applications with minimal effort in addition to the development of the core solution engine. Effectively, our scientists can address feedback from business users and deploy new versions of a solution in minutes. This means new features and enhancements on the solution engine and user interface, not in weeks or months, but in minutes!

Mip Wise is a purpose-driven organization. Part of our purpose is to democratize technology so that even small organizations can benefit from cutting-edge support systems to make optimized decisions (strategic, tactical, and operational). This way we are contributing to minimizing waste and maximizing efficiency across industries and, as a result, helping to reduce costs for goods and services while also reducing negative impacts on our planet.

Aside from our motivation, we’ve learned that the missing ingredient to build and deliver enterprise-grade solutions to support scientific-sound decision-making is not technology but people and processes. In fact, most of the technologies we need are either open-source or very low-cost compared to how much organizations have been paying for “AI solutions”.

Processes are free of cost, except that it takes people and time to develop them. Since we already have our processes in place, people are the most important variable in our equation. That’s why the majority of our investments go into training. This means that when you hire our services you are contributing to the formation of new professionals that soon may also be contributing to your organization.

By continuously training our people and using well-defined processes, we drastically reduce the number of staff-hours in each project–with much more effective communication as a bonus. In addition, we don’t waste resources implementing features that you will never use, because we develop and deliver purpose-built solutions. Finally, we have invested almost no money in marketing because we want to continue to grow organically and word of mouth is helping the right people to know about the quality of our deliverables.

Agile is a methodology that suits many applications and is particularly adopted in software development. We are also big adopters of Agile but building decision-making support systems go beyond software development. In fact, Develop is only one of the four D’s in the D-Wise Framework (Discover, Define, Develop and Deploy). To ensure that the four functions of D-Wise synchronize together effectively and efficiently, we apply Agile in a broader scope and employ additional sub-processes.

For example, one of the most important steps in an optimization project is to write a precise representation of the business problem to be solved. That is called a mathematical formulation and falls into the Define bucket of D-wise. Next, the formulation is translated into another precise representation, but this time using a programming or modeling language so we can use computers to solve the problem. That is called an optimization model and falls into the Develop bucket of D-wise. The lack of a process that allows scientists to switch back and forth between these two representations of the business problem poses the project with a major risk of delivering wrong recommendations. This is particularly the case when the formulation and the model implementation are done by different people, and get even worse when it comes to future maintenance or expansion of the solution. Unlike traditional software development, unit testing can’t make an optimization model bulletproof, which reinforces the need for well-defined development processes.

As one of the most advanced users of mathematical optimization in the world, we know that a generalized optimization solution, such as those implemented in off-the-shelf software, will never outperform a custom solution. Custom solutions have the massive advantage of exploiting specific properties of the business problem that generalized solutions are too rigid to benefit from. The gap becomes more apparent as the business problem evolves (which it always does), and additional tweaks and twists are required to fit it into the generalized solution.

Our deep expertise in optimization and our experience with scientific processes allow us to match the modeling approach with the best algorithmic approach to strike the right balance between performance and accuracy. Following the Agile methodology, we start simple and go all the way to custom heuristics and decomposition approaches depending on the complexity of the problem.

It’s important to be skeptical about vendors that promise to solve challenging optimization problems by increasing computing power. In most cases, the challenge of solving optimization problems is due to combinatorial complexities. In this case, run time grows exponentially as the problem instance increases linearly. Therefore, increasing computational power linearly (more cores and more memory, for instance) won’t help much, only modeling and algorithms can beat the exponential complexity growth of the problem instance.

For us, challenging problems mean opportunities to make impacting contributions. For example, in one of our client’s projects, our custom algorithm was able to solve instances of an optimization problem with more than 900,000 products whereas the off-the-shelf software couldn’t even solve a simpler version of the problem with 30,000 products in an acceptable amount of time. In another client project, after a modeling tweak, we could solve in seconds with an open-source optimization solver (CBC-PuLP) the same instances of a problem that one of the best commercial optimization solvers (Gurobi) could not solve in 30 minutes before the modeling change.