Who:

All Users

Why:

The Star Rating gives current information about the performance of Providers against a range of criteria.  It is the overall indicator of Provider performance. Star Ratings assist the Client to decide which Provider should be allocated to an order and which Providers should be invited to Quote on a Matter. The Star Rating results may also be mapped to KPIs set out in the Service Level Agreement and therefore referenced during Providers’ performance reviews.

Overview

Rating templates are defined by the Technical Contract Manager.  They enable a Provider’s performance within the LPG to be measured so that it can contribute to a rating that can be used to inform future interactions with that Provider.

The Star Rating is an overall rating out of five stars based on the Provider’s performance in three main areas in the LPG application, each of which is given a weighting to contribute to the overall Star Rating

• Compliance: The Provider must submit current Insurance documents, Compliance documents and any other required Evidentiary documents.
• KPI Results: This covers a range of activities as determined by the Client and configured within LPG. Examples of activities used as Key Performance Indicators are:
  • How long it takes for the Provider to accept an Order;
  • Timely completion of the Order (completed by the Order End Date);
  • Percentage of invoices approved without dispute.
• Performance Surveys: Surveys are completed by Clients or Matter Managers on their level of satisfaction with panel firms’ legal services and advice.

 

·      NOTE: Other areas that may contribute to a Star Rating (i.e. Policies, Evidentiary Documents) have not been configured for use in LPG.

Multiple Rating templates can be defined by the Technical Contract Manager so that different performance expectations can be accommodated for different types of contract.  Each template will contain the list of predefined performance criteria which can be switched on or off depending on their relevance to different contract types. Ratings templates are associated to Contract templates but can be modified at individual contract level, if required.

Procedure

1. To View Ratings results, on the LPG menu select Contracts > Find Contracts.

2. The List Contract screen appears. Search for the Contract you wish to check the Ratings for and click on the Contract Reference Number.

3. This opens the View Contract screen. Click on the Ratings tab to bring it into focus.

You will see a description of the Rating Template that was used for this Contract. This also shows the pre-defined percentage ranges that will decide how many Stars are awarded.

 

4. Under this, you will find the Star Rating Results.

The criteria are sorted into Criteria Groups (4a). To see a score breakdown for a Criteria Group, click the box  next to the name (4b) and the individual items that may contribute to the score are displayed.

·      NOTE: Only those Criterion Groups that have a score can be expanded to display how the score was derived.

 

5. The Cost Rating Results display an average order costs for the contract.

Tips & Troubleshooting

CONCEPTS:

Group Weighting

• Group weighting must be specified if the user enters weighting for the specific groups individual criteria, i.e., group weighting for Compliances must be entered if there is a weighting for PUBLIC LIABILITY INSURANCE.

Individual Criteria Weighting

• At least one Individual weighting must be specified if the user enters group weighting for the related group.
• The user needs to enter weighting for a criterion if they select the Mandatory field for it. If not, an error message is displayed.

In other words, if a weighting is entered for the Group, a weighting must be entered for the Individual Criteria, and vice versa. The total of all Group Weightings must equal 1.0.   If there is a value entered in a Group Weighting, the corresponding Individual Weightings must equal 1.0 for each Group. See below.

A. Values in entire Group Weighting column must sum to 1.0
B. Values in each group in the Weighting column must sum to 1.0 if there is a Group Weighting entered for the group.

CALCULATION RULES:

In general:

A batch process will sum the total points earned for each individual criterion and multiply this figure by the Group Weighting to determine the result for the Group when calculating the Star Rating.

An exception to this is if an individual criterion was marked as mandatory on the template and received 0 points, the result for the entire Group when calculating the Star Rating will be 0.  This will only apply to the Compliances section.

Below are the calculation rules used for the three criteria used for Star Rating calculations in the LPG.

1. Compliances

Score calculated (to achieve a maximum possible score of 100 points, all contributing criterion are multiplied by 100)	Rule
100 points x individual weight	For a valid compliance document submitted and verified by Customer Support and the Contract Manager For a policy Verified by Support only
100 points x individual weight	For an Compliance Document not marked as required on the Contract but has been submitted and included in the Rating Template
0 points x individual weight	For an invalid/expired For a Rejected policy (Customer Support or Contract Manager) For a policy submitted but not yet checked by Customer Support

Example:

• Compliances Group Weighting = 0.1
• Individual Criteria (total = 1.0) (All Marked as Mandatory on the Rating Template)

            Assurance of Compliance = 0.35

            Value Add Services Report = 0.35

            Professional Indemnity Insurance = 0.10

            Public Liability Insurance = 0.10

            Workers Comp Insurance  = 0.10 

Scenario: Group Weighting = 0.1

Contract Requires	Rating Weight		Scenario	Calculation
Workers Comp		0.10	Submitted and accepted by Support, but not yet verified by CCM	100 x 0.10 = 10 points
Public Liability		0.10	Valid	100 x 0.10 = 10 points
Professional Indemnity		0.10	Expired	0  x 0.10 = 0 points
Assurance of Compliance		0.35	Valid	100 x 0.35 = 35 point
Value Added Services		0.35	Not marked as required on contract but included in Rating	100 x 0.35 = 35 points

TOTAL SCORE = 10 + 10 + 0 + 35 + 35 = 90 points

TOTAL SCORE x GROUP WEIGHTING = 90 x 0.1

 

= 9 points out of a possible 10 points for Compliance

2. KPI RESULTS

·      NOTE: KPI Results to be used in the Rating Template batch process are the adjusted figures for the previous Period 2.

 

Score calculated	Rule
Adjusted KPI Result % for previous period 2 x individual weighting for KPI	Where KPI Events are being recorded for the Contract and a positive KPI is being measured
100 minus Adjusted KPI Result % for previous period 2 x individual weighting for KPI	Where KPI Events are being recorded for the Contract and negative KPI is being measured

Example:

• KPI Results Group Weighting = 0.4
• Individual Criteria (total = 1.0)

            Order Acceptance – Due  = 0.25

            Invoices submitted on Time = 0.15

            Undisputed Invoices = 0.15

            Invoice submitted within Budget = 0.15

            Work Completion Deliverable – Due = 0.15

            Work Completion Engagement - Due = 0.15

Scenario: Group Weighting = 0.4

Contract Requires	Rating Weight	Scenario        (Previous Month Results Adjusted)	Calculation
Order Acceptance – Due (Positive KPI)	0.25	85%	85% x 0.25        = 21.25 points
Invoices Submitted on Time (Positive KPI)	0.15	92%	92% x 0.15        = 13.80 points
Undisputed Invoices         (Positive KPI)	0.15	78%	78% x 0.15        = 11.70 points
Invoices within Budget    (Positive KPI)	0.15	65%	65% x 0.15       = 9.75 points
Engagements by Due Date  (Positive KPI)	0.15	100%	100% x 0.15      = 15.0 points
Deliverables by Due Date     (Positive KPI)	0.15	84%	84% x 0.15        = 12.60 points

TOTAL SCORE = 21.25 + 13.80 +11.70+9.75+15.00+12.60 = 84.10 points

TOTAL SCORE x GROUP WEIGHTING = 84.1 x 0.4

 

= 33.64 points out of a possible 40 points for KPI Results

 

3. Performance Survey results

Scores are calculated from the question level.  The question score (1-5) is normalised to a percentage and then multiplied by the question weighting.  Where sections of a survey have been marked not-applicable, the questions that have responses are re-weighted proportionately.  When the weighting is applied, a question ‘score’ is calculated and an average of scores for that question may be determined.  The average scores are then added for the Criteria Group and the group weighting is applied. 

Example:

• Survey Results Group Weighting = 0.5
• Individual Criteria (total = 1.0)

            Question 1 = 0.25

            Question 2 = 0.25

            Question 3 = 0.25

            Question 4 = 0.15

            Question 5 = 0.10

 

Each question is answered with a rating score from 1 – 5 that is converted to a percentage as follows:

 

1 = 0% (the lowest score)

2 = 25%

3 = 50% (the midpoint score)

4 = 75%

5 = 100% (the highest score)

 

If any question is regarded as ‘not applicable’, the responses to the other questions in the survey are re-weighted.  To calculate the new weighting, take the question weight (for the questions that DO have responses), multiply it by the total number of questions in the survey and then divide it by the number of questions with responses.  This will give you the new weighting for that question.

 

Provider A – Survey 1

Scores:  5, 5, 4, 4, 2

First, normalise the scores:    100%, 100%, 75%, 75%, 25%

Then, apply the weighting:     100 x 0.25 = 25

                                                100 x 0.25 = 25

                                                75 x 0.25 = 18.75

                                                75 x 0.15 = 11.25

                                                25 x 0.10 = 2.50

Score = 82.50

 

Provider A – Survey 2

Scores:  3, 5, N/A, N/A, 4

First, normalise the scores:    50%, 100%, 75%

Then, apply the weighting:     50 x (0.25 x (0.25/0.75)) = 20.83

                                                100 x (0.25 x (0.25/0.75)) = 41.67

                                                75 x (0.10 x (0.10/0.75)) = 12.50

 

Score = 75.00

Add the scores of multiple surveys and get an average:  78.75

TOTAL SCORE x GROUP WEIGHTING = 78.75 x 0.5 

= 39.375 points out of a possible 50 points for Survey Results

STAR RATING CALCULATION Finally, all Criteria Group scores are added to achieve the Star Rating Score.   Compliances = 9 points KPI Results = 33.644 points Survey = 39.375 points   Therefore, the total points = 82.015 points. This score is then translated to the Resulting Star Rating ranges as set on the Rating Template.   In this example, a score of 82 points would translate to a 4 star rating for the particular contract.