A hallmark of large Enterprises is using System Integrators to deliver IT products. It’s often more efficient than having an internal team because SIs usually have the experience of having delivered a particular technology repeated times. What’s left for an Enterprise is to select and manage an SI in the most efficient manner possible.
We’ve all been here before – this can be a difficult and time consuming process – for both yourself and the SI. One of the most difficult things to pin down is how to qualify a delivery has taken place. Do you base it on delivery of a your requirements? And if so how do you check that the requirements have been delivered to your satisfaction? Often SIs will provide some approach with milestones to prove delivery. These will include artefacts like, Solution Designs, Technical Designs, Project Plans, etc.. But you still have the issue of assuring your delivery.
It’s all too common to come to the end of a delivery and either the SI has reduced functionality to meet time-scales, or you realise that what has been delivered is not exactly what you’ve expected. And this is usually because measuring the SI is usually a blunt metric like final delivery of a function that you can test manually. The only problem is that by the time it is delivered it is often too late to do anything about it.
What if there was another way of creating KPIs that you could see as the vendor is delivering. Almost like a live pulse that will show all aspects (quality, timeliness, etc.).
There are techniques in DevOps that can provide these metrics and here’s how:
As an Enterprise you should provide delivery tools that the SI can use where various aspects of the delivery can be automated. For example by providing code quality tools, automated testing tools for regression/load/smoke/integration/etc., or continuous build servers that can track code check-in and build runs.
Once these tools and processes are in place a set of very detailed metrics can be formulated (here’s a small sample)
(1) Technical Debt – this can be calculated on every build
(2) Test Coverage - the extent to which tests are automated
(3) Deployment Frequency – how frequently do you go to production, reflects throughput.
(4) Mean time to Recover (MTTR) – how well are you recovering from issues, reflects stability.
As these metrics are defined they can be negotiated with the Vendor. This gives you and the vendor very detailed and clear metrics to achieve. It may often make vendors change their estimates or pricing, but this will ultimately reflect what they can actually achieve per cost, rather than what they perceive without the details.
By defining these metrics up front it can vastly increase the probability that you will have a successful project that is within the quoted budget, time, and quality.