A hallmark of large Enterprises is using System Integrators to deliver IT products. It’s often more efficient than having an internal team because SIs usually have the experience of having delivered a particular technology repeated times. What’s left for an Enterprise is to select and manage an SI in the most efficient manner possible.

We’ve all been here before – this can be a difficult and time consuming process – for both yourself and the SI. One of the most difficult things to pin down is how to qualify a delivery has taken place. Do you base it on delivery of a your requirements? And if so how do you check that the requirements have been delivered to your satisfaction? Often SIs will provide some approach with milestones to prove delivery. These will include artefacts like, Solution Designs, Technical Designs, Project Plans, etc.. But you still have the issue of assuring your delivery.

It’s all too common to come to the end of a delivery and either the SI has reduced functionality to meet time-scales, or you realise that what has been delivered is not exactly what you’ve expected. And this is usually because measuring the SI is usually a blunt metric like final delivery of a function that you can test manually. The only problem is that by the time it is delivered it is often too late to do anything about it.

What if there was another way of creating KPIs that you could see as the vendor is delivering. Almost like a live pulse that will show all aspects (quality, timeliness, etc.).

There are techniques in DevOps that can provide these metricsĀ and here’s how:

  1. As an Enterprise you must provide delivery tools that the SI can use (or the SI provides) where various aspects of the delivery can be automated. For example by providing code quality tools, automated testing tools for regression/load/smoke/integration/etc., or continuous build servers that can track code check-in and build runs.
  2. Once these tools and processes are in place a set of very detailed metrics can be formulatedĀ (here’s a small sample)
    1. Technical Debt – this can be calculated on every build
    2. Unit test coverage – how much of the SIs code is unit tested (which will imply adherence to requirements and acceptance testing)
    3. Number of code quality violations – how well is the code being developed? Is the SI adhering to best practice for that language? Does the SI need some help in correcting some of the Quality early enough in the process before it becomes too big to handle
    4. Individual performance – it is possible to score your engineers based on adherence to your metrics, and this is not so much to name and shame but again rather to identify bottlenecks early on in the process to assure a successful delivery

As these metrics are defined they can be negotiated with the Vendor. This gives you and the vendor very detailed and clear metrics to achieve. It may often make vendors change their estimates or pricing, but this will ultimately reflect what they can actually achieve per cost, rather than what they perceive without the details.

By defining these metrics up front it can vastly increase the probability that you will have a successful project that is within the quoted budget, time, and quality.