Results and solutions

The study was foreseen in three phases.

In the first phase, carry out the analysis of the implementation of PPPP in public administration authorities in Slovenia, and a comparative study in selected countries.

In the second phase, develop the PPPP implementation and efficiency evaluation model based on own catalogue of indicators and verify it with an empirical study.

In the third phase, test the updated and upgraded KRONOS catalogue of indicators in several selected areas of PPPP implementation.

The study was designed to lead us to new theoretical findings about the key PPPP implementation aspects and results and solutions which could significantly contribute to strengthening the PPPP implementation competences of the top hierarchical levels of ministries and directorates (and on a local level).

It was divided into three steps. Within the framework of Step 1 a thorough comparative analysis of this area in other selected EU countries was carried out. We defined the key phases of development programmes implementation life cycle, their main activities and carriers, competences and responsibilities. In studying the approaches and solutions of other countries, special attention was paid to the positioning and organization of specialized business functions involved in the analysis and implementation of PPPPs.

We established that Step 2 and 3 are, from the perspective of content, interconnected to such an extent that we also connected them in their implementation and provided a detailed analysis through empirical studies in the field. The aim was to analyze the approaches, organizational solutions, practices and competences in the field of PPPP implementation in ministries and public institutes in Slovenia, issues related to the positioning of analytical and development functions (policy making units) in the executive authority structure, i.e., the structures of ministries and directorates, their organization, if any, staffing, responsibilities, tasks, activities, etc. We determined the extent to which ministries systematically deal with the long-term development of the areas they cover, the extent to which strategic management has been developed, the status of strategic documents and their implementation. Special attention was given to the role of the normative regulation of this area (sharing of competences and responsibilities) and the importance of reliable provision of financial and human resources for programme and project success. We analyzed the extent to which structures established within institutions that implement development projects and programmes influence the implementation processes (isomorphism).

The empirical study was designed so that we could convincingly answer all the research questions raised in the project application. Based on the theoretical analysis and some probing, we concluded that the success of the PPPP implementation is most affected by six key factors: strategic, normative, institutional, economic/financial, methodological and organizational/HR/IT, which were placed in the core of the empirical study. In light of the nature of the characteristics of the investigated issues and the environments to which the study related, we choose a qualitative study where 22 interviews were carried out. First we created an appropriate questionnaire, conducted some pilot interviews and then made a list of approximately 35 qualified interviewees, mainly senior civil servants, ministers, state secretaries and directors of 15 (out of 24 officially certified areas of public policy implementation in Slovenia) PPPP implementation areas in Slovenia. Approximately 30 of the 35 invited experts positively responded to our interview request.

Due to broadly set study, the desire to cover the widest possible range of PPPP areas, provide a reliable picture and achieve a high attendance rate among the interviewees, the empirical study was carried out for half a year longer than initially planned and was therefore completed in September 2017. Together we were able to carry out 22 interviews, which is a very large number for such studies. After the completion of the interviews we began processing the obtained data, which lasted until mid-2018. We decided to combine quantitative and qualitative analysis, and used the ATLAS specialized software solution for the qualitative analysis of data obtained from the interviews.

Keywords: PPPP, indicator, pdca, public policy, public governance, public policy analysis, policy, agenda setting, public administration, public management, public finance, research and development indicator, R&D indicator, regulatory impact analysis, RIA, catalog of indicators, catalogue of indicators, benchmark, agenda definition, decision making definition, governance, monitoring and evaluation, plan do check act, public sector, social impact, social policy

In the second phase of the study we tried to evaluate the empirical study results in two ways. First we carried out a thorough quantitative/qualitative analysis of the collected data focusing primarily on the evaluation of the importance and role of the above-indicated key factors for the successful implementation of public policies in Slovenia. Quantitative analysis was carried out using the statistical evaluation of data and assessments obtained through the interviews. The study showed that the interviewees consider the ‘strategic approach’ and ‘normative regulation’ to be the most important factors for the successful implementation of public policies in Slovenia. In order to verify the obtained results, we organized a two-day international workshop in April 2018 to which we invited domestic experts, some interviewees and leading European experts in the field of public policy implementation and evaluation (approx. 30 experts). The international workshop was successful – above our expectations – and largely confirmed our baseline hypotheses, while upgrading and expanding them through highly in-depth discussions on all key aspects of our study and its starting points/findings.

All the findings of the theoretical and empirical tudy supplemented with the conclusions of the international workshop were combined in a scientific monograph titled Vidiki in dejavniki uspešnega izvajanja javnih politik v Sloveniji, which was published in September 2018.

Due to the reduced extent of financing, the entire course of the study had to be adjusted, whereby we tried not to change its conceptual and methodological design. In the first part of the study we were able to fully achieve this objective; moreover, the empirical study was carried out in a wider context than initially planned and covered as many as 15 areas of public policies, which greatly strengthens the validity of the obtained results and suggestions.

In developing a model for monitoring and evaluating public sector development programmes and projects we decided to focus on the indicators, since the empirical study revealed that indicators are one of the most critical points of the entire life cycle of implementing public policies in Slovenia.

Keywords: PPPP, indicator, pdca, public policy, public governance, public policy analysis, policy, agenda setting, public administration, public management, public finance, research and development indicator, R&D indicator, regulatory impact analysis, RIA, catalog of indicators, catalogue of indicators, benchmark, agenda definition, decision making definition, governance, monitoring and evaluation, plan do check act, public sector, social impact, social policy

In the third phase of the study, the research activities were focused on the model, design and IT solution of the catalogue of indicators. As envisaged in the research plan, the catalogue of indicators, which was developed as part of the previous research project – KRONOS, was used as the starting point. The baseline catalogue of indicators was developed for a specific area of ‘eGovernment’.

Although we have had a lot of experience in connection with models and approaches in developing indicators for evaluating public policies due to previous studies, this has once again proved to be an extremely complex area where the gap between theory and practice is the widest. Namely, numerous indicators, which in theory seem to be ideal, may prove inapplicable in practice, as there are no suitable methodological implications or data sources for them. Our in-depth analysis showed that it is possible and meaningful to maintain, upgrade and apply/generalize the model, structure, specifications of indicators attribute specifications and IT solution from the existing catalogue for a wider implementation in other areas of public policies; however, the situation is quite different with regard to the indicators themselves. An in-depth analysis of indicators in the KRONOS catalogue showed that only 50 out of approx. 500 indicators can be used/transferred to other areas of public policy implementation in Slovenia. The study therefore showed that we need one set of generic indicators for the evaluation of each policy area that can be implemented in other public policies, and a significantly larger set of indicators that are policy specific and can hardly be used in other areas. In order to determine what the relationship between these two sets of indicators is, we developed and verified a pilot set of indicators for the following selected areas of public policies: science and development, health, taxes, culture, environment and space.  A comparative analysis has clearly shown 1. the relationship between general and policy-specific and 2. the characteristics of indicators that are generally applicable in any area of public policy implementation. In the final study phase, we focused on the development of a catalogue of indicators that can be applied in planning, implementation and evaluation of different public policies in Slovenia. We have upgraded the , which is not only important for further research in this field, but also for professional public and practice. Now, there are 72 generic indicators included.

 

Catalogue of indicators

The basic structure of the Catalogue of indicators is derived from the exhaustive literature review and predominantly from our own empirical research. Catalogue of indicators contains 72 indicators, which can be applied through clustering method or individually. Each indicator within the catalogue has 14 attributes, including exact definition, policy cycle phase, evaluation level and evaluation aspect. Other attributes are related to indicator type, scale type, scale, methodology, data sources, measurement history and frequency, geographical scope, measured subject, measured property, and references. The proposed Catalogue of indicators provides an applicable platform for the construction of more specific and complex evaluation frameworks, which could be used for the assessment of concrete public policies, programmes, and projects. The Catalogue of indicators is available here.

Keywords: PPPP, indicator, pdca, public policy, public governance, public policy analysis, policy, agenda setting, public administration, public management, public finance, research and development indicator, R&D indicator, regulatory impact analysis, RIA, catalog of indicators, catalogue of indicators, benchmark, agenda definition, decision making definition, governance, monitoring and evaluation, plan do check act, public sector, social impact, social policy