Publications & Resources

LAO Reports

Publications & Reources

Evaluation of Criminal Law Office

Third Year Executive Summary

Posted on: April 21 /2008

A. Reports to Date from the Evaluation

With funding assistance from the Federal government, Legal Aid Ontario (LAO) has funded three Criminal Law Offices (CLOs). These CLOs are staff offices with a mandate to supplement the judicare (certificate) service delivered by the private bar. They provide criminal representation to financially eligible accused in Barrie, Brampton and Ottawa.

LAO committed to ensuring that a major (three-year) independent evaluation of these CLOs was undertaken and made available to the general public. To-date, five reports have been produced by the team of senior private consultants and academics who were selected in the competition to conduct the evaluation:

  1. an Evaluation Framework which clarified both the objectives, specific questions to be addressed and the methodology of the evaluation;
  2. a Nine Month Progress Report (produced in March of 2005) which provided baseline statistical data on the environment of the CLOs and focused on the perceptions and expectations of LAO, the CLOs, and other stakeholders within the criminal justice environment in the three CLO sites;
  3. a "First Year" Report (December, 2005) that covered the operations of the CLOs from May 1, 2004 to June 30. 2005; and
  4. a "Second Year" Report that covered the operations of the CLOs from July 2005 through June 2006.
  5. the current, final, report, the "Third Year" Report which includes the operations of the CLOs through June 2007.

The processes leading to the First, Second and Third year reports are very similar.

All are based on intensive and extensive rounds of interviews in each of the sites, focusing on judges, Crown attorneys, duty counsel, non-governmental organizations, the private bar, CLO staff, and others who have had experience working with the CLOs and observing their impacts on clients and the criminal justice system. The evaluation team also spent considerable time working with LAO staff to extract a considerably wider range of data from Legal Files, the main automated data base utilized by the CLOs and LAO to record, store, and report data on CLO operations. In addition, the evaluation obtained and analyzed additional important data from the PeopleSoft system that records information on certificates that are issued to clients to cover legal aid services provided by the private bar. Data on certificates issued to the CLOs is also contained in Peoplesoft. As well the evaluation has obtained data from the LAO financial systems, from a special database maintained locally at the Brampton CLO and from a database of "Form 50s" for certificate cases from the Barrie CLO (the latter two for all but the third year report). The evaluators were also encouraged to design and implement a Criminal Peer Review of the files of the CLO staff lawyers and of comparable defence counsel certificate files - the first in Ontario.

The basic formats of all three reports are very similar. However, while the first- and second-year reports were presented in one part, the third-year report is presented in three parts:

  • Part 1: Highlights of Findings and Recommendations
    Chapter 1is an overview and a detailed summary of the results and findings of the other chapters in Part 2 of the report. References are often provided to relevant sections and Figures in Part 2. That summary also contains most of the specific recommendations that follow from the evaluation. Chapter 1 also ends with a list of what the evaluators believe are the most important lessons learned from the evaluation.

  • Part 2: Supporting Data and Analysis
    contains what in previous years would have been Chapters 2 through 6, each of which contains the more detailed data and analysis that underlies the major findings and recommendations. The reader interested in a specific finding highlighted in Part 1 is definitely encouraged to consult Part 2.

  • Part 3: Appendices
    contains additional information and data and supplementary materials.

B. Developmental Context of the Evaluation

Given the novel and experimental nature of the CLO initiative and the uncertainty with respect to what could be achieved in what were known to be complex and at times hostile environments, LAO decided to proceed in an exploratory mode. LAO did set guidelines for the types of outcomes to be achieved and the types of processes to use, but allowed the local CLO directors considerable latitude and local autonomy in experimenting with different levels of outcomes and processes. The experience of the CLOs would then be closely monitored to collect the more specific and detailed information needed by LAO to set-at a later date-the appropriate levels of standards that would be needed and used for longer term planning, modification and management of the CLOs. Since this exploratory mode has continued throughout the three years of the evaluation, it has been impossible for the evaluation to make normative assessments-in particular, to go beyond assessing whether the CLOs were "doing the right things" to consider either whether they were doing "enough of those things" or whether they were "doing the things efficiently enough".

On the other hand, LAO's choice of an exploratory approach placed a different type of pressure on the evaluation, namely, undertaking the monitoring and collection of the types of information on "what was done and what was achieved" by the CLOs-the types of information needed to allow LAO to now undertake the concentrated internal analysis, planning and decision-making exercise needed to set standards, to assess CLO performance in terms of those standards, and to determine whether modifications are required to the CLO design if it is decided to continue the model into the future. Given the importance and nature of this LAO work to be supported, and uncertainty as to which areas would be seen as requiring specific attention, it was appropriate to err on the side of over-inclusion rather than under-inclusion of data and analysis. This evaluation report thus covers a broader range of issues in more detail than might normally be expected.

C: Key Lessons Learned

Chapter 1 summarizes the major findings that are supported by the detailed analysis and data in Part 2 of the evaluation. That section also contains some of the more specific recommendations that flowed from that work.

The following takes a broader perspective and extracts and presents a short list of what the evaluators believe are the most important lessons learned from the evaluation.

1. Significant Value Added by the CLOs

  1. Finding: The CLOs have in the three years since their inception provided valuable services to over 2,300 clients in their three communities. By providing legal assistance to those who would otherwise likely have appeared in court unrepresented, the CLOs have ensured a more fair and effective judicial process.

  2. Recommendations: That LAO
    i. recognize the CLOs for their important achievements.

This and the following findings and recommendations should, however, be placed in a context that includes:

  • These offices have achieved this result in the face of a number of serious obstacles to their development, including:
    • Considerable resistance to their establishment and continued existence on the part of members of the local private bar
    • This resistance resulted in LAO placing a number of constraints on the CLOs that significantly handicapped their ability to operate effectively and efficiently (e.g., limiting advertising, prohibiting coordination between CLO and Duty Counsel services, having to devote considerable amounts of time to support a major independent evaluation effort)
    • In large part for the above reason, difficulty at the beginning in attracting staff in certain CLOs
    • Being implemented during a time when LAO was facing major challenges in the revamping of key automated information and communications support systems
    • Having to operate with (what was over time discovered to be) insufficiently clear central direction regarding key policies and practices.

The evaluation findings thus relate, not only to the CLOs per se, but also to the external and corporate environment in which the CLO model was implemented.

Within this context, the staff of the CLOs should be commended for the important results they have achieved.

It is within this context that the following key lessons learned are presented.

2. Unmet need represented by Non-certificate Cases and Persons with Special Needs

  1. Finding: The CLOs have clearly confirmed the existence of-and the benefit of addressing-a significant previously unmet need for legal services on the part of persons who qualify financially for legal aid but who do not receive such services through the legal aid certificate system. The evaluation has also shown that the CLOs have been effective in providing legal and legal-support services to these groups.

  2. Four types of needs can be identified:
    1. Non-certificate Cases: Needs of specific cases that are denied legal aid certificates because they are unlikely to result in a custodial sentences,
    2. Enhanced Services: Enhanced services needed by specific types of clients, especially those involving Aboriginal accused and accused with mental health issues.
    3. Unserviced Certificate Cases
    4. Law Reform and Outreach: Need to address specific systemic issues, including special problems facing persons incarcerated in mega-jails.

    The CLOs have clearly identified a general need for the first two types of services (non-certificate cases and enhanced services to specific cases). The evaluation report provides strong evidence that failure to address the needs of unrepresented persons in financial need who are refused certificates (because they are unlikely to receive a custodial sentence) potentially results in significant harm to those persons, and extra costs for the justice system. Similar legal aid support is required regarding persons with special needs.

    (The third and fourth types of need are addressed in Lesson 3 later.)

  3. Recommendations: That LAO
    i. Consider advocating (jointly with other Justice System stakeholders) for expansion of the criteria and funding for extending legal aid (via certificates) to financially needy persons and for certain special needs groups, especially mentally ill, indigenous persons and youth;
    ii. Explore alternative methods for improving services to financially needy persons (e.g., through CLOs, enhancing duty counsel, criminal clinics, merger of CLO with expanded salaried duty counsel in Barrie and elsewhere);
    iii. Clarify the extent to which the CLOs' should focus on meeting unmet needs for non-certificate cases; and
    iv. Enhance the specialized expertise that CLO staff already have in areas such as: serving indigent accused, the mentally ill, Indigenous persons and youth.

3. Needs related to Certificate cases and Outreach/ Law Reform

  1. Finding: The need for the CLO to address the third category (certificate cases) is not general, but does exist in specific areas of demonstrated needs unmet by the private bar. A particular need is also found for types of cases that the private bar finds difficult to handle within the financial constraints of the legal aid tariff (e.g., impaired driving cases). There is also a need for the CLOs to take on at least a small number of certificate cases for professional development reasons.

    The need to provide the fourth type of service (Law reform and Community Outreach and Education) also clearly exists within each community. However, because of a shift in individual CLO priorities and practices over time away from non-casework activities, less attention was given to this area. Further experience and clarification of priorities is needed before the CLOs' potential role(s) in Law Reform and Outreach is determined.

  2. Recommendations: That LAO
    i. Clarify its policy regarding the numbers and types of certificate cases the CLOs should accept;
    ii. Explore the most efficient and effective way the CLOs can participate-jointly with other justice, LAO and NGO stakeholders-in law reform and outreach activities.

4. Efficiency regarding Individual Cases

  1. Finding: The CLOs have demonstrated that they can handle and close individual cases in an efficient manner, a manner that compares favourably to the efficiency of the private bar in similar cases.

  2. Recommendation
    i. That the CLOs be recognized as handling and closing individual cases in an efficient manner.

5. Total Value of Services compared to Total Costs

  1. Finding: The CLOs have demonstrated that it would be extremely difficult under the existing tariff for a public or private office dedicated solely to providing the above four types of services to this clientele to be able to "break even" (i.e., "to earn" enough tariff fees to cover costs).

    However, the challenge seems to be more with the hourly rate under the tariff than with the number of hours set for different tasks and types of cases.

    First, the total imputed fees for the four primary services each of the CLOs offer fall considerably short of the total costs of operating each of those offices. In the absence of a clearer definition by LAO of the values it places on other services offered by the CLOs, it is unknown whether the values of these other services cover that shortfall. However, the CLOs have demonstrated the value of a number of "other" services, including:

    i. The CLOs have provided a environment that can be used to test out-on a limited scale-innovative approaches to the provision of services (for example, the use of Community Legal Workers as both community workers and for lawyer case management support, or different mixes of staff with different experiences)

    ii. Having available staff resources to pursue specific cases and/or law reform issues (e.g., the DNA litigation handled by the Ottawa CLO)

    iii. Being able to collect more complete data on certain types of operations to better understand costs and benefits of existing procedures (e.g., the onerous data collection borne by the offices for the evaluation to test out the time to complete certain tasks)

    iv. Providing a vehicle for testing and development of alternative monitoring, accountability and quality assurance strategies (e.g., peer review)

    v. To have available resources to test alternative partnerships and allocations of responsibility with other LAO organizations (e.g., exploring alternative relationships with duty counsel).

    Second, the evidence shows that the average numbers of staff hours the CLOs need to handle and close cases are similar to (sometimes lower than) the number of hours that are billed by the private bar for similar cases. As well, the shortfall between the total CLO overall costs and the total CLO imputed value of handling cases, outreach and law reform activities cannot be accounted for entirely by atypical overheads or inefficiencies. It follows that the tariff rate used to calculate imputed value for the CLOs is not high enough to cover their normal overheads. (This argument has been made for some time by private firms-although many private firms also have a tradition of providing services pro bono-at least by undercharging (cash) clients or by providing more hours of service than they bill.)

  2. Recommendations: That LAO
    i. Develop an explicit policy regarding whether, and if so to what extent, the costs of offices like the CLOs can exceed the measurable imputed values from the four core services they provide. Alternatively, LAO should develop a realistic notional value for the "other" services provided by the CLOs;
    ii. Review the adequacy of rates under the tariff.

6. High Proportion of Time Spent on administrative and other activities

  1. Finding: The CLOs devote a high proportion of time to overhead activity.

    Lawyers and/or CLWs in at least some CLOs spend a likely financially unsustainable proportion of their overall time on administrative and other non-billable activities not related to either specific cases or law reform or outreach)--sometimes in the order of 40% to 50%. Although some of this time relates to activities that would not normally be required in the private sector (e.g. supporting this evaluation and attending to matters related to broader LAO administrative issues), opportunities also exist for reducing this proportion.

  2. Recommendation: that LAO
    i. Undertake a review of ways to reduce the time spent on these types of activities of the CLOs. Evidence is presented in the evaluation report to suggest that such a review should consider areas such as work standards (i.e., standards related to the volume of cases to be handled and quality control), management activities, allocation of work, and more efficient and effective support systems.

7. Clarifying to CLO Staff the Mix of Services to be Offered

  1. Finding: Different interpretations exist regarding the mix of services that were to be offered by the CLOs. However, as things progressed over the three year period, the CLOs tended to place higher priority on providing more traditional legal services to individual cases-and less priority to services such as outreach, public education, and to law reform and systemic reform. This lack of clarity regarding the desired mix of services is an impediment to effective management of resources and the ensuring of accountability for achieving results. The uncertainty regarding expectations is also a detriment to the general quality of the staff work environment.

  2. Recommendations
    i. Clarification is required from LAO regarding the preferred mix of services to be provided
    ii. Once clarification has been provided, modification of caseloads and business practices on the part of the CLOs will be required to match the priorities
    iii. LAO senior management should determine and communicate its expectation as to what percent of their time the CLOs should devote to activities such as: public legal education, law reform activities, outreach and developing collaborative strategies with other concerned groups of stakeholders (such as: CMHA, duty counsel, clinics).

8. Considerable Variation Among CLOs

  1. Finding: There are considerable differences among the CLOs with respect to priorities and approaches. In the absence of clear statements of policies and procedures from corporate LAO, it is impossible to tell whether these differences are appropriate. However, some differences raise concerns on the part of the evaluators, and on the part of the staff of the different CLOs.

    The evaluation report notes a considerable number of areas in which policies, strategic approaches and operational procedures vary significantly from one CLO to another. Specific examples given include: the mix of certificate and non-certificate files, the manner in which outreach/law reform is carried out, trends in costs per case, and workloads. To some extent it may have been appropriate to give the offices "free rein" to experiment with and to develop alternative approaches. However, development of policies and practices has been a "bottom-up, operations-driven" process, rather than a "top-down, corporate policy-driven" process. There has for some time been a need for more explicit direction from corporate LAO to provide vision, direction and management guidance as to what practices are acceptable and optimal, and which are not.

  2. Recommendations: That corporate LAO:
    i. Provide clarification of policies and best practices
    ii. Enhance organizational structures for formulation, communication and management of policies
    iii. Provide (from corporate and from the region) stronger and clearer leadership with respect to direction, management and accountability, and quality control
    iv. Provide stronger substantive (criminal practice policy) leadership to and enhance management expertise of CLO directors
    v. Enhance accountability and quality assurance initiatives (including performance measurement and peer review).

9. Quality of Case-Specific Services

  1. Finding: There is a strong consensus in the three communities that the CLO lawyers provide services in individual cases that compare in quality to those offered by members of the private bar providing similar services. However, some concerns have been raised regarding a limited number of specific areas.

    As noted throughout the report, the analysis of data on performance from Legal Files, the Peer Review, and the interviews show that in many areas the CLO lawyers have achieved results for their clients that are as good as, and sometimes surpass, those typically achieved. However, those same three data sources have identified a limited number of concerns about specific operational issues. These issues are documented and discussed in more detail in later chapters of the report (issues ranging from documentation of cases, data recording practices, use of guilty pleas, etc.).

  2. Recommendations
    i. That LAO senior management work with the directors of the CLOs to better understand and to remedy specific quality issues that are identified in this evaluation.

10. Comparisons between CLOs and Private Bar require caution

  1. Finding: There are a number of strong reasons that comparisons between the CLO offices and private criminal law offices should be made with caution. In particular, it is unfair to the CLOs and the private bar to assume that a "level playing field" exists-or that it should exist in terms of client outreach and visibility of the CLOs.

    There are a number of areas in which comparisons with the private bar are appropriate (e.g., file documentation, hours per case, litigation strategies). However, (as noted earlier) the full report lists a number of areas that make comparisons between the two fraught with difficulties. For instance there are a number of ways in which either the CLOs or the private bar have advantages or handicaps vis ? vis the other (e.g., the private bar do not have fixed salaries; the CLOs cannot as readily hire or fire staff to optimize returns on investment; the private bar can take on any type of family, civil or criminal case; the CLOs must focus on certain types of cases that are difficult to handle within the tariff). In those situations, a simple comparison of the CLOs and private bar may be inappropriate or even misleading.

  2. Recommendations
    i. That future planning and communications regarding the CLOs recognize that the CLOs and the private bar in certain respects do not operate on a "level playing field".

11. Lessons learned regarding Management

  1. Finding: A number of lessons have been learned in the last three years as to how management structures and practices (at different levels within LAO and the CLOs) could be enhanced to improve the effectiveness and efficiency of the CLOs and CLO-like initiatives.

  2. Recommendations: That LAO develop
    i. Stronger and clearer direction from senior management and Area Directors to the CLOs
    ii. Enhanced management capabilities of CLO directors
    iii. Enhanced systems of accountability at all levels
    iv. Clarification of the role, and functioning, of the LACs
    v. Greater collaboration of other LAO providers with the CLO in their area
    vi. Greater collaboration with NGOs and other CJS stakeholders.

12. Lessons learned regarding operational practices and support systems

  1. Finding: A number of advisable improvements have been identified to operational practices and systems.

  2. Recommendations: That LAO undertake
    i. Major review of Legal Files
    ii. Reallocation of responsibilities regarding design and configuration and quality control regarding information systems
    iii. Re-assessment of roles of CLWs and administrative assistants and regular supervision and feedback to both lawyers, CLWs and administrative staff
    iv. Regular formal staff meetings, with periodic attendance at these meetings by the Area Director
    v. Review of and staff training regarding effective methods for targeted outreach and law reform (adopting more effective methods of reform and development-including involvement of local NGOs and representatives of the private bar in all stages of the reform process (including the development of priorities).

13. Attention Needed to Human Resource issues

  1. Finding: A number of specific and general human resource management issues have been identified.

    Many of these issues would be resolved if staff were given a clearer view of what was expected of them. Others have to do with the level of compensation paid, especially when compared to other jobs that are seen as comparable. Still others (e.g. those related to management and community development and organizing skills) would be relevant if one were asking the CLOs to take a stronger leadership role in their communities.

  2. Recommendations: That LAO human resource staff
    i. Conduct interviews with staff to ensure the existence of organizational mechanisms to ensure continuation of a healthy and productive workplace;
    ii. Conduct a review of the adequacy of CLO lawyer, CLW and support staff salaries.

D: Overall Conclusion: Improved Access to Justice?

Given all of above, the CLO model is a promising approach to improving access to justice for financially needy persons. However, there is considerable variation with respect to how different parts of the model have been designed and implemented in each site. Important lessons have been learned, but considerable design and development work remains to be done.


PDF Third Year Report