Our approach to technology and process improvement initiatives is often fueled by hope, driven by shame, and made urgent by fearful hype.
Missing from that list?
Data.
This does little to set us up for success and leaves change efforts subject to the whims of whomever is loudest or most powerful. If we want to see change in law firms, we must prove that tech initiatives deliver results. And if our change efforts are failing, then we must correct course or cut our losses.
We can’t do either without data.
You Can’t Manage What You Don’t Measure
We introduce technology and process improvement initiatives, but rarely track their progress. We don’t know whether benefits outweigh costs—or the costs outweigh benefits.
As a result, we:
- can’t justify continuing current initiatives or launching new ones;
- don’t have the data to encourage people to continue their journey;
- don’t know how to improve what we are doing or know what needs improvement;
- can’t make a well-informed decision to correct course or cut our losses.
We must measure and track what we’re doing.
The good news is this concept is not new. Evaluation is its own discipline and it has been embraced successfully in industries worldwide. It’s time for the legal industry to apply data-driven evaluation to technology.
A Theory of Change
To understand a technology intervention and evaluate implementation, consider what the intervention requires and what it’s designed to produce. The inputs, activities, outputs, outcomes, and impacts that make a theory of change will vary with the circumstances. Within legal tech, a theory of change can usually be broken down along these lines:
Figure 1: Theory of Legal Tech Change
Looking at the diagram, if we exclude the category of impacts (which are, by their nature, difficult to measure) and activities (which don’t require quantification), every other element of this theory can be counted and measured. Data and metrics for legal tech are feasible, and fairly simple to collect.
To ensure that technology and process improvement initiatives are valuable, we must measure and record our change efforts and their results. Using metrics that track change is essential. It will allow you to accurately assess the situation, fix problems, and advance objectives. Metrics also serve as a compass throughout the change management process, which help to evaluate and adjust course.
Let’s apply these five elements of change to a legal technology implementation or intervention.
Objectives
Tracking the wrong data or creating an overly burdensome data collection process will thwart your efforts. So start by establishing and clarifying your business goals for this change effort. Then determine what conditions must be met to make it possible to meet your business goals. These are critical success factors (“CSFs”). Then implement and track your initiative. It should line up with your CSFs and your KPIs (defined below). Each element of your plan should have a change management outcome-focused metric. Adopt activity-based metrics, such as counting actions taken, training courses completed, time spent, etc. And also adopt outcome-based metrics to assess whether you are meeting your business goals. For each metric, define what success looks like at each stage of change. Assign someone to own the data collection for accountability.
Inputs
Inputs include everything that goes into selecting, implementing, and maintaining a new technology or process. It includes search, scoping, bid evaluation, and selection; piloting; acquisition cost; training and support. The time and financial cost of each input must ultimately be compared with results. Remember to include the cost of evaluating and monitoring in your inputs. Assume between 1% and 5% of the total project budget for monitoring.
Outputs
Outputs are direct metrics that can be clearly attributed to activities. For technology initiatives, you’ll want to analyze usage patterns. How much is the new technology being used? Are users embracing the technology or begrudgingly using it? Is there a trend showing that adoption is growing? Metrics might include individual log-ons, calls to tech support increasing or decreasing, compliance with new processes, and incorporation into workflow. You might also want to conduct user surveys to determine whether users are getting frustrated, confused or lost; whether users are using workarounds that introduce error; or whether users are improperly delegating work to others. These metrics help to assess whether the technology initiative is working as planned.
Outcomes
From there, you can develop your key performance indicators (“KPIs”). KPIs must be important, predictive, and measure achievement or success. They can relate to financial outcomes, client satisfaction, process improvement, or skills building, empowerment, or investment of individuals. Select no more than five KPIs. Your selection should include a combination of outcome metrics that give insight to the end result of a process and process metrics that give feedback on the performance elements of the process. Overall, KPIs should reflect progress toward a specified goal and align business activities and individual actions with strategic objectives. For law firms, one of the main KPIs is the realization rate. Realization rates reflect the real value of work to the firm and consider efficiency, experience, technology, staffing, and client satisfaction. However, if realization rates are your only KPI, you risk attributing too much to that one metric (and in the real world, metrics may change for many reasons), so always have multiple KPIs.
Monitoring, Evaluation, and Action
Monitoring is making sure you get the data. But that’s not enough. Using your data to clearly evaluate whether your intervention is working is key. Evaluation measures costs and benefits. If your measurement of outputs shows that an initiative is failing, maybe more training on the technology is required. Or maybe the people in your organization don’t understand the benefit of the change. Uncovering the reason for the failure is key to figuring out how to fix it. With data, you can find the problem early, correct course, and make the change successful.
Celebrate Small Wins
On your path to transforming your business, use metrics to track and celebrate small wins. Measurement can motivate people and give guidance toward goal achievement. Small wins give feedback and justification for the change effort and validate decisions and strategies. Small wins also build faith in the change effort and can even draw in people who hadn’t yet embraced the initiative.
Cut Your Losses
As much as we are hopeful, sometimes the best thing to do with a failing tech initiative is to end it. If four people are using an expensive piece of software bought for the entire firm, don’t continue to pay for the entire firm to use it. Sometimes IT projects don’t work out. If that’s your firm, don’t regret that–at least your firm is taking the steps in the right direction and trying. But equally, don’t continue to pour money into it. How often has the ego of a partner or IT guru led to big bills for a technology that no one wants or uses? Hanging on to a poor fit and a bad decision will hamper future efforts.
Find the Lesson
When you’ve successfully implemented technology and reaped the rewards, don’t forget to evaluate your success so it can be repeated. Look for the lessons. Did benefits outweigh costs because cost was low or because benefits were high? What other kinds of technology intervention are worth considering for your firm as a result?
Conclusion
Tracking the success of technology and process improvement initiatives is key to knowing whether they work and planning for the future. Metrics that show success can also justify the costs of ongoing technology investments and training programs, and support efforts to launch new initiatives. Capturing small wins can encourage individuals to continue their learning journey despite setbacks. And data can show whether technology initiatives are actually meeting business objectives. Collecting and evaluating data will help you to manage, measure, and motivate.
This article was originally published May 9, 2018 on Above the Law during Ivy’s time with PerfectIt, which is a consistency checker that works well with WordRake.
About the Author
Ivy B. Grey is the Chief Strategy & Growth Officer for WordRake. Prior to joining the team, she practiced bankruptcy law for ten years. In 2020, Ivy was recognized as an Influential Woman in Legal Tech by ILTA. She has also been recognized as a Fastcase 50 Honoree and included in the Women of Legal Tech list by the ABA Legal Technology Resource Center. Follow Ivy on Twitter @IvyBGrey or connect with her on LinkedIn.
