On the Effectiveness of CVT Promotion Measures Within the Enterprises in Europe (1)

M’Hamed Dif(2)

BETA/Céreq Alsace
University Louis Pasteur of Strasbourg, France
Email: mdif@cournot.u-strasbg.fr

Paper presented at the European Conference on Educational Research, University College Dublin, 7-10 September 2005
(EERA/VETNET)

The overview and analysis of the collected meta-evaluation studies of measures fostering CVT by the enterprises will be channelled through two basic parts and concluding "outlook" section. The first part of the paper, titled "analysis of measures fostering companies’ participation and investment in CVT" is basically focussed on the analysis of funding-related incentives and schemes, which can be either supply or demand focussed measures. The traditional measures focussing on the supply-led funding are usually undertaken and regulated by public authorities at national or regional levels (in collaboration with social and sector partners), with aim of guaranteeing a minimum CVT provision and funding for the enterprises and individuals. Due to budget constraints and raising training costs in an environment where CVT and LLL need to be further encouraged, there is an increasing shift towards the use of (targeted and non-targeted) demand-led funding measures to stimulate the enterprises and the individuals’ investment in CVT. In this context, examples of "universal" or non-specifically targeted measures will be examined with a specific focus on "leave of self-directed CVT and learning" (LSD-CVT), addressed in principle to all individuals, such as it is the case of the French scheme "CIF" (Congé Individuel de Formation). As for the demand-led funding measures targeting specific categories of enterprises and groups of individuals will be exemplified by the following three cases of voucher-based funding schemes (from Austria, Belgium and the UK), namely:

• Training accounts scheme (Austria);

• Regional training voucher scheme ("chèque de formation" for SMEs in Wallonia, Belgium)

• ILA (individual learning accounts) scheme in the UK.

The second part of the paper will be a review of basically published (and peer-reviewed) overall evaluation studies of the "returns" to investment in CVT from the perspective of the enterprises, employees and the public authorities (in Europe and outside Europe).

In the light of the underlying conclusions and implications of the reviewed evaluation studies, the last ("outlook") section is an attempt to put together the basic elements for an "input-outcome" evaluation approach to CVT systems (to be further developed and applied in ulterior research stages of CVTS2 project). It is a two-level multi-indicator evaluation framework linking "input-measures" (and their objectives) to the "outcomes" through a two-level evaluation process in terms of a set of quantitative and qualitative performance indicators: "primary" and "outcome" performance evaluation indicators.

The paper is concluded by highlighting some of the overall trends in the results of the reviewed evaluation studies and their implications, including open questions for further research.

It is basically focussed on the analysis of funding-related policy measures. Green at al (2000) classified different CVT funding incentives and measures to foster investment in CVT into two basic categories:

These measures include incentives and policies undertaken and regulated by public authorities and/or social partners (at national and regional/local levels). They are targeted towards guaranteeing the provision and funding of CVT for both companies and employees. They usually take the form of direct subsidies or levy-funding schemes. The direct subsidies may be in terms of guaranteeing CVT provision for certain groups, or a certain type of training provision, funding information and guidance services, and through general or targeted subsidies for enterprises and/or employees to encourage participation and investment in CVT. The way these policies are delivered varies from one country to another. For instance, in the Netherlands, the government funds part-time CVT undertaken by adults within the Regional Training Programme centres (ROCs). It also provides CVT for individuals unable to sustain their employment without further training, within the framework of the General Training Scheme (KRS). Direct subsidies to enterprises are usually provided within certain specific sectors (e.g. the processing sector). In Finland, public funding of CVT for individuals is generally allocated to regional authorities. In addition to apprenticeship schemes, there are also training provision quotas for adults within the initial training structure. Specialised training institutions maintained by the enterprises receive direct subsidies. In Sweden, municipalities provide and finance adult education leading to formal qualifications. The educational system is organised in a way which allows individuals to study while they are employed. However for many countries it is not easy to ascertain the direct public funding. In Spain and France, the public funding of CVT is very closely tied to their levy schemes.

Moreover, most countries operate some kind of levy or fund schemes with social partners’ contributions. The levy/funds may be applicable (by law or through collective agreement) to all enterprises or to targeted categories of enterprises (of a certain size and/or specific sectors) and groups of individuals. They are usually collected from both employers and employees with the aim of carrying out specific objectives such as financing lower-qualified employees or unemployed individuals. They are usually absorbed by public expenditure on training and managed separately by asocial partner organisation (like OPCA in France). The demarcation line between levy-funds of social partners and public authorities (national/local) funding is usually not clear due to the fact that social partners’ funding in some countries like France and Denmark is initiated or formalised by the state. Most countries operate some kind of this type of schemes at least in some specific sectors. For instance, in Finland about 1% of the enterprises and employees’ contributions to unemployment insurance funds are allocated to training and severance fund, which is used (within the framework of CVT) to fund unemployed adults. In Greece, enterprises and employees contribute collectively 0.45% of the total wages to a fund (LAEK). Then they claim reimbursement of their training costs within the limits of their contributions. In Germany, there are levies specific to certain sectors such as agriculture and construction. There is in France a minimum statutory levy from the enterprises of 1.5% of the total wages, introduced with the aim of encouraging participation and investment in CVT and ensuring equal access to CVT for all workers. Despite the lack of formal social partnership arrangements in the UK, there are also some levy schemes adapted to certain sectors such construction.

These supply-led schemes (in terms of levy funding) have been criticised basically on the ground that the efficiency and the effectiveness of the enterprises investment decisions in CVT may be compromised by the compulsory levy schemes. Indeed, some enterprises feel that those compulsory levies are no more than extra form of taxation (CEDEFOP, 1998, 1999). This, for instance, was one of the reasons behind the abolition of the sector-based compulsory levy schemes which was operational during the 1960s and 1970s in the UK. But, this criticism does not undermine the principles of the schemes themselves but rather the way in which they are enforced. In the Netherlands, for instance, they are based on voluntary and adjustable agreements. Also in Spain the levy scheme is reviewed on 3-year basis. In France, the enterprises tend to spend on average more than the double of the statutory level.

According to Elson-Rogers and Westpalen’s study on funding CVT in the European Union (2000), many of the old EU member countries (Austria, Demark, Finland, France, Germany, Greece, Ireland, the Netherlands, Spain, Sweden and the UK) are moving towards measures focussed on demand-lead funding. As Green at al (2000) pointed out, the potential benefits of demand-led funding mechanisms include the reduction of the dead weight associated with public authorities funded provision, and can meet more adequately the training needs for both individuals and enterprises. They include policies and schemes to stimulate participation and investment in CVT by enterprises and employees. They can be grouped into Two basic categories: "universal" (non-specifically targeted) and targeted measures and schemes.

As universal policy measures and schemes, they concern all categories of individuals and enterprises. The most interesting example in this case is the provision of individual training leaves for employees such as CIF (Congé Individuel de Formation, i.e. individual leave for employees self initiated and directed training and learning) in France or the educational and training leave in Denmark.

This programme was created to promote free choice and equal access to learning and training by introducing three interdependent guarantees (Dif, 2000; Gahéry, 1996; P. Guilloux, 1996; Ph. Méhaut, 1996):

The beneficiary of this kind of leave is free to choose any type of training programme outside the firm's usual training scheme. The course chosen can be vocational with a variety of ultimate aims: advancement within the same company, a job conversion and mobility or just a simple adaptation to changes in technology and labour market structures. The training programme can also be non- vocational with the aim of allowing the trainee to acquire general qualifications and/or become more involved in social and cultural life.

It guarantees non-exclusion-based access which means that all workers with different types of work contracts can benefit from this regime. Originally, LSD-CVT (leave of Self-Directed CVT) was designed and implemented exclusively for workers recruited according to unlimited duration work contract. It took twenty years for this programme to adapt to the situation of precarious employment. In December 1991 it was officially extended to include employees on fixed duration contracts and temporary workers.

Since 1984 the LSD-CVT has been financed through a special fund held and run by an independent parity organism (called OPACIF). This fund is fed by an obligatory contribution from the employers representing of about 0.20% of the total wages paid to employees. The state takes over part of the training costs. However, the state contribution is variable in accordance with the time involved and still limited in general to financing special cases of vocational training leaves, such as long-term training leave and training leave within small companies (with less than 10 employees).

The performance of the LSD-CVT scheme was evaluated in terms of its contribution to a set of test criteria which stem directly from the basic principles of its founding missions and objectives. These test criteria are twofold:

Concerning "CIF" contribution to the reduction of unequal access to learning, and on the basis of a survey carried out by Céreq in 1985, J.-J. Paul conducted in 1989 an empirical study for this particular purpose. The results of the investigation revealed that while access to employer-directed CVT was observed to be biased in favour of "core" employees with higher initial qualifications (especially within large companies), access to "CIF" did not have this bias (Paul, 1989).

Concerning its contribution to the promotion of mobility, Paul's study (1989) showed that the contribution of the LSD-CVT scheme to socio-professional mobility (including access to higher socio-professional status and qualifications) is higher than the average performance of the whole CVT system. The observed evolution in this direction was also confirmed by the results of the National Statistics Office survey on vocational qualifications and training (INSEE, 1993). On the whole, LSD-CVT has been observed to contribute to the promotion of workers' professional mobility inside and outside the firm (Paul, 1989; INSEE's Surveys of 1985 and 1993).

More or less similar individual training leave schemes are also practiced in Denmark and Spain. Denmark educational leave programme (run by the National Labour Market Authority) concern all employed and unemployed individual. It is funded through a compulsory levy paid by all employed individuals (about 8% of gross salary). In Spain all employees who have worked more than a year in an enterprise can benefit from a paid training leave (including the social security coverage) by just submitting applications (three weeks in advance) to their employers.

They are policies and measures targeted at specific categories of enterprises and groups of individuals. They are exemplified here by the three cases of voucher-based funding schemes from Austria, Belgium and the UK:

Many surveys on the level of enterprises’ investment in CVT show that SMEs invest less than larger firms. According to FOREM (Wallonian vocational training and employment office), in 1996, SMEs in Wallonia region were investing below the Belgian average of eight hours of training per employee in a year. Given concerns about CVT provision in these firms, the Wallonian Minster for Employment and Vocational Training proposed the "training voucher (cheque de formation)" scheme which came into force in December 1998 (Decree of 23 July 1998).

The introduction of the voucher is based on the principles of:

Practically, the voucher scheme aims at achieving higher levels of training with 20 hours a year per worker as a benchmark (i.e. a minimum of 400 hours a year per a company).

The training voucher is targeting commercial companies with less than 50 employees, based in Wallonia. The eligible beneficiary can be any employee or employer (as manager or an independent self-employed person). The training courses are to be held during working hours and taught by an approved trainer from an accredited list training provision institutions. Training domains covered include data processing, languages, marketing and exports together with work-related specific programmes. Each voucher is valid for a year and has a face value of 29.7 €, determined on basis of an average wage rate of 19.8 € per hours and an estimated training cost per hour of 9.9 €. SMEs can purchase training vouchers up to a total of 400 for half of their face value (from Sodexaho: an independent issuing company), i.e. contributing 50% to the funding (where the remaining 50% is taken in charge by the Wallonian government). Then, these vouchers are given (by the enterprises) to individual trainees to use them as means of payment for their training courses. The whole operation of voucher scheme is run on behalf of the government of Wallon region by FOREM (the Walloon employment and training office), which plays the role of facilitator for SMEs (to meet their needs) and also as an interface between them and training providers.

Given that the scheme has been launched only by the end of 1998, it has not yet been evaluated yet thoroughly by means of surveying SMEs and individual beneficiaries. However the observations based on monitoring data (including interviews with persons in charge of the system) registered and made available by the "Cellule Chèque Formation" (after a year of its launch), indicate the following results:

• The overall reactions of the the SMEs to the scheme are positive. 1,383 SME employers and independent entrepreneurs have taken up training vouchers to train more than 3000 workers. For them, the voucher system addresses successfully a number of SMEs’ main obstacles and concerns. Among the advantages they frequently mention, the simple character of the procedure and the financial incentive to give access to suited training. The training providers are also satisfied, especially those who have been accredited as the accreditation procedure guarantees high quality standards of training provision.

• Responses from a sample of 211 SMEs show that only about 40 blue-collar beneficiaries out of a total of 521 beneficiaries. This indicates that blue-collar workers have benefited less from the scheme than the other employees, although the decision-maker considered them initially as a target group.

In response to the budget constraints and increasing training costs in an environment where CVT and LLL should be encouraged, demand-led financing mechanisms in terms of quasi-voucher "training accounts" schemes have been introduced in Austria at the regional level (by five out of eight provincial governments with the involvement of the social partner organisations) during the 1990s (1993-1999). Also the animated debates around the EC’s published White Paper (in 1994) on "Growth, Competitiveness and Employment", particularly in the context of LLL, provided a strong impetus for the development and adoption of this type of demand-led co-funding scheme (CEDEFOP, 2000).

In the Austrian case, this form of "training accounts" is different from pure-voucher mechanism due to the fact that there is no coupon involved in the transaction, and that the pre-paid training costs are reimbursed to the trainees rather than paid directly to the training providers. However, this type of "training accounts" does have some elements in common with the voucher insofar as there is a "financial entitlement" to participate in training. This is in addition to the fact that, while the trainee pays the training cost at the beginning of the training course, there is a pledge from the provincial government social partner organisations to pay certain costs before the start of the training course.

In essence, the "training accounts" involve basically co-funding of individual training by the provincial government and the individual participants (with a financial contribution from the regional social partner organisation in some cases). Five out of eight provinces are involved: Carinthia, Salzburg, Styria, upper Austria and Vienna.

The aims of the training accounts are basically to improve the qualifications and skills of working individuals or unemployed people in search for work in an overall context of LLL through financial incentives to participate in training.

The operational functioning of training accounts scheme (in terms of targeted groups, content of training, training provision and financial entitlement) varies between provinces.

In general all training accounts are designed for residents of the region concerned and fall into four basic categories of target groups:

• Employed individuals interested in undertaking CVT either with the aim of improving their skill levels or for obtaining certification;

• Unemployed or seeking employment individuals;

• Apprentices graduating from the dual system;

• Individuals on maternity or paternity leave.

Concerning the content, some of theses accounts are designed to encourage specific types of training which reflect skill shortage in the labour market while others are more general. Additionally, certain types of the training accounts are targeted toward certified training whereas others are not.

Common to all training accounts in different regions is that the trainees have to choose their training providers from a predetermined list of suppliers as means of ensuring a minimum quality standard in training provision (with preferential tendency to support local training providers). As for the financial entitlement available to trainees, is variable according to the type of the training account or the scheme (general, special, project oriented, professional improvement, post-apprenticeship, etc.) and the target group. In general, all types of training accounts allow for a refund of direct costs of the training up to 50% with specified minimum and maximum amounts. The maximum amounts tend to be higher for the beneficiaries of a training leading to certification or those who are unemployed. For the latter group, additional funds are made available to cover travel costs.

The process of evaluating the effectiveness of the Austrian training accounts scheme is complicated on the one hand, by the variations of their mode of delivery in various regions, and on the other by the limitedness of available evaluation data to two provinces basically (Upper Austria and Vienna).

However, in the light of the available data, it is important to underline the following overall trends and conclusions concerning the performance evaluation of the training account schemes (West and Sparkes, 2000):

• On the whole, various training accounts schemes have been successful in terms of stimulating demand for training. In upper Austria for instance, the number of applicants had increased during the period (1994-1998) by about 10% on average a year. Most of the participants used the voucher-based schemes to follow computer related CVT, courses in languages, technical or financial management training. Concerning the level of their satisfaction, most of them had a "good" or "mixed" feeling about training received and its relevance to their profession. Reinforcing unbiased information system might help potential participants to make well informed decisions.

• However, it is nearly universal finding that more highly educated and those in higher social groups are the main beneficiaries of CVT within these schemes. The fact that the participants have to pay in advance their training costs (which might be more expensive and exceed the available financial support limit), limits the access of lower paid potential trainees to theses schemes.

• On the supply-side (within the framework of these schemes), the majority of the participants undertook their training within the traditional (publicly-funded) training providers (82% in Upper Austria and 60% in Vienna). The share of private or small and innovative training is low. In order to encourage innovation and competitiveness in the training market, there may be a case for allowing new providers to enter the schemes.

The national Individual Learning Account (ILA) framework was formally introduced in the UK (covering England, Scotland and Northern Ireland) in 1999. It is a funding scheme whereby individuals are given a fixed amount of money which is earmarked for training, supplemented by a contribution from both employers and employees. Such accounts (ILAs) are expected to constitute the main financial source for adults to acquire qualifications and skills, to increase their earnings and employability. The enterprises are encouraged to contribute to the ILAs of their employees (especially the lowest paid workers), by means of a deduction from taxable profits. The overall aims of ILAs include:

The funding of an ILA is composed of a contribution from the regionally-based "Training and Enterprises Councils (TECs)" of £150 (€232.6), a variable tax-deducted contribution from the employer and a top-up of at least £25 (€38.8) from the ILA-holder. The latter (i.e. the employee’s contribution) is viewed as triggering-off of the mechanism and as means of ensuring the beneficiary’s commitments and willingness to take the responsibly of his for her own training. This is why, the employers are encouraged to contribute to the accounts, but they are not allowed to pay on behalf of their employees. The targeted groups within the operation of the ILAs scheme include those individuals whose employment falls into skill shortage area, the employees of mall-medium enterprises, especially those with low-skill or qualification levels or those individuals returning to work following a period of economic activity. During the year 2000, and following the take-up of the first one million accounts, the ILAs were reformed. The reformed programme dispenses with the incentive payment of £150 and shifts towards a discount-based financial incentive system. Accordingly, a 20%-discount is given off of the cost of a broad range of eligible training courses, whilst a limited list of IT and basic mathematics training courses attract a discount of 80%.

As part of the first year of the implementation of ILAs scheme, the government departments in each of the four UK home countries, commissioned an early evaluation of:

• The characteristics of ILAs redeemers (people who opened and used their individual accounts) and non-redeemers (individuals who opened, but not used their ILAs);

• The process and level of satisfaction.

This evaluation was undertaken by York Consulting Ltd and MORI Social Research Institute between 26^th February and 4the may 2001. It was based on primarily on a quantitative follow-up telephone survey with a sample of ILA redeemers and non-redeemers. Additionally, a more qualitative performance assessment was conducted through a series of workshops with 70 ILA providers and stakeholders. The basic findings are on the whole as follows (York Consulting Ltd & MORI, 2002):

• 85% of the redeemers (who opened and used their accounts) declare their satisfaction with the scheme, and that the training undertaken either met or exceeded their expectations. As for the non-redeemers (who opened but not used their accounts), over one-half of them cited personal reasons such as lack of time and family commitments (with almost one-quarter mentioned training course-related reasons).

• The profile of both redeemers and non-redeemers is more or less the same. The majority of them are:

– Employed or self employed (up from 73% to 75%);

– In possession of some form of qualification (down 84% to 83%);

– Female (57%).

– 52% within the age group 31-50 (23% within the age group 21-30).

• In terms of the socio-professional background characteristics of the redeemers, the class of supervisors (lower-middle management) and skilled manual workers (C1 and C2) formed the single most significant category (60%), followed by upper and middle management class (A and B) (21%). The least beneficiary category of ILAs scheme (16%) is made of those redeemers who belong to the semi-skilled and unskilled working class (D), including those at the lowest subsistence level (D).

• There is a wide range of reasons for the redeemers to undertake the ILA-supported training. The most frequent ones were in order of popularity:

– To develop new skills;

– To obtain a new or a better job;

– To succeed at work;

– To acquire qualifications;

– To increase self confidence;

– Personal development.

• Almost two-thirds (62%) of the redeemers had previously participated in work-related training which was either provided by their employer or funded through their own resources. However, there are noticeable differences between male and female redeemers. For the male redeemers for instance, 51% of them declare that they had participated in work-related training provided by their employers compared with only 39% for female redeemers. Conversely, a higher percentage of female redeemers (49%) had taken an evening training course than male redeemers (38%).

• Concerning the financial contributions to their training costs, over three-fifths (63%) of the redeemers declare that they have contributed less than £50 towards the cost of their ILA-supported training. This includes the 10% of them who declare that they have not made any financial contribution.

• Among the 11% of the redeemers who declared that someone else was contributing towards their training cost, 73% confirm that this contribution was paid by their employers.

In spite of the fact the ILAs scheme was far more popular than expected, the government decided to suspend the scheme in November 2001, following allegations that that a large number of account numbers had been extracted from the system for sale (by some providers). The new Department for Education and Skills (which took over from the Department for Education and Employment in June 2001) has agreed in principle, to work with Capita, as a new delivery partner, in developing arrangements for a successor scheme. For this new department the successor system needs stronger quality assurance mechanisms to prevent unscrupulous providers benefiting from it (National Audit Office Report, 2002).

Most of the evaluation studies of returns to companies’ investment in CVT looked at returns basically in terms of the "effects" of training on productivity and wage growth rather than the effectiveness of the training process itself. Given that in some countries enterprises receive a high level of state subsidies while in others there is an employer-employee co-funding, these studies do not address in detailed way the issue of whether these returns differ according to funding provider. They are mainly concerned with finding out the causal link between the training provided by the enterprises and its "returns", basically in terms of productivity growth (as return to investment in CVT at the enterprises level) and wage increases (as returns to employees’ participative investment in CVT). There are also other studies concerned with the evaluation of the return to investment in training from overall policy perspectives, mainly in terms of the growth rate of aggregate productivity, output, income and employment.

Most of the reviewed evaluation studies tend to concentrate on both productivity and the performance of participants as a result of the undertaken investment in CVT:

• In their study on the effectiveness of the training grants within the manufacturing companies, Holtzer et al (1993) looked at training benefits from the perspective of the employers. By surveying the enterprises which benefited from the training grants, a data set covering a number of years was generated with information training input and firms output (in terms of productivity). Then, the authors explored whether there is a link between the two, working on samples between 171 and 250 enterprises. They used the "scrapage rate (i.e. the proportion of manufactured units that must be discarded due to defaults)" as a measure of productivity output, and training hours per employee as a measure of training input. The findings were that there is a direct link between increased training and reduced scrapage rate (i.e. increased productivity).

• In his study on "the impact of previous training on productivity and wages", John Bishop (1994) used two surveys conducted in the USA. The first one was undertaken in 1982 and is based on 3,412 employer-interviews (only 480 were of use for the purpose of the analysis). The second survey is conducted through questionnaires mailed to 11,000 firms (by the National Federation of Business). 2,285 responses (out of 2,599) were of use for the purpose of the study. In both surveys, the employers were basically asked to give information on the training, wages and productivity of only two of their employees. The experiences of the two employees were compared across a larges number of firms. Then, in order to explore the link between the training and its outcomes in terms of productivity and wage, he constructed and applied a "training-time index" (obtained by multiplying the reported amount of training time by the unit cost of that time). In general, the findings were that training increases both productivity and wages. However, the contribution of training (received from current or previous employer) to increases in productivity was found to be higher than the increases in wages.

• In a more recent study from the USA on "do workers pay for on-the-job training?" Barron et al. (1999) showed that the impact of on-the-job training on productivity growth was much stronger than its impact on wage growth.

• In their study on "does training generally work?", Barrett and Oconnel (1998) estimated the link between training and productivity on the basis of data collected from two successive surveys within the Irish companies. The first survey (conducted in 1993) was for collecting detailed information about the investigated firms training activities. The second survey conducted in 1997 was a follow-up within the same companies to get further information on, among other things, output, capital stock and workplace practices during the period 1993-1995. On the basis of detailed information provided by about 200 firms, the authors proceeded to the test of the relationship between the companies training input and changes in their output and productivity. The findings were similar to those emerging form Black and Lynch’s study (1996) in the sense that training is not seen in itself to influence productivity, but rather the type of training undertaken that really matters. Since training was classified by the employers as "general training" (i.e. usable by other employers), the finding of Barrett and OConnel was that this type of training contributed effectively to increased productivity. As for the other type of training classified as specific by the employers (i.e. non-transferable or usable elsewhere), the results of the investigation indicated that it has no effect on the productivity of the firm.

• The study of Ottersten et al (1996) is one of the few European econometric studies which examined the cost and productivity effects of the firm financed training. It presents a model of the relationship between training and the cost reduction. Then, the authors tested their model by using annual data from eight Swedish companies in the machine tool industry for a period of 18 years (1975-1993). Their empirical finding is that there are large cost-reduction effects of training. As for training effects in terms of increased productivity, they are limited.

• In a study on "further vocational training and re-education in East Germany - experiences and perspectives", Hübler (1998) found that on-the-job training increases job security, whereas off-the-job training leads to higher wage earnings. But this is true only for privately financed training. Publicly financed training has only positive effects in the short-term. In another study in East-Germany, Lechner (1999d) examined enterprise-related continuous vocational training and found positive income effects too. Also Kraus et al (1998) observe, for a sub-period of their study, positive effects on (stable) employment for on- and off-the-job training (in East-Germany).

• In the UK, Dearden et al (2000) showed in their study "Who gains when workers train? Training and corporate productivity in a panel of British industries" that productivity levels of training-intensive sectors are higher than in other sectors, and that both the participating workers and firms gain from training.

A large number of evaluation studies tend to measure CVT-induced benefits accruing to the employees of the enterprises in terms of wage gains of participants and promotion opportunities as types of returns, namely:

A number of evaluation studies on measuring CVT-induced returns to the economy as whole has positively linked CVT to the improvements in knowledge and skills and to increases in output and income throughout the economy (Becker, 1975, OECD, 1996). A positive relationship has also been established between educational levels and aggregate productivity or economic growth (Barro, 1991; Mankiw, Romer and Weil, 1992).

Since training programmes are one of the most important measures of ALMP (active labour market policy) in Europe, many researchers in different European countries have compared (within the framework of their evaluation studies of national ALMPs) training programmes with other measures such as job-creation and/or subsidised employment programmes. On the whole, their findings were that the effect of vocational training programmes on employment and employability is positive in most of the cases, while the impact of other ALMP programmes was basically negative. Examples of these studies and their findings are as follows:

• In their study on "manpower training programmes and employment stability", Zweimüller and Winter-Ebmer (1996) analysed the effects of Austrian vocational training on employment stability. The results of the study indicate that training programmes contributes significantly to the reduction of unemployment risk for the participants in the training programme. Also in Austria, Winter-Ebmer (2001) "Evaluating an innovative redundancy-retraining project: the Austrian Steel Foundation" analysed effects of training programmes by the Österreichischen Stahlstiftung on employment. These programmes are thought to counteract staff reduction within the steel industry, so only steelworkers are eligible for the participation. The results of this study indicate that there are positive employment effects primarily for unemployed older than 27 years, whereas for younger people there are no effects.

• In a study for Belgium, Cockx et al. (1998) compared the effects of (in-house and external) training programmes and subsidised employment programmes on employment duration. They found a positive effect from in-house training programmes and no effect from external training (and subsidised employment) programmes. This is basically due to the fact that internal training programmes are an investment in firms’ human capital, and consequently they should reduce the risk of being laid off. In another subsequent study for Belgium titled "vocational training: does it speed up the transition rate out of unemployment", Cockx and Bardoulat (2000) analysed the effect of vocational training programmes on exit from unemployment by evaluating training programmes not organised by the companies. They found a negative locking-in effect during the programme and a positive effect after its completion. For them, the positive effects subsequent to the training programme realisation compensate for the locking-in effect and thus the overall effect becomes positive.

• In France, Bonnal, Fougère and Sérandon (1997) studied the impact of French vocational training programmes on job creation and exit of unemployed. In their analysis, they distinguished between unemployed people with vocational education and those without. Their finding was that there is a positive effect from training programmes on exit from unemployment for the unemployed without vocational education, but there is no positive effect on job creation. Brodaty, Crepon and Fougère (2001) obtained similar results in their study "using matching estimators to evaluate alternative youth employment programmes - evidence from France" where they used the same dataset but different evaluation methods.

• In Denmark, Jensen, Nielsen and Rosholm (1999) analysed the effect of vocational training programme for unemployed youth on entry into regular employment and into a regular vocational education. Their findings were that while there is no significant effect on entry into regular employment, the effects on entry into regular vocational education are positive.

• In their study "estimating the effect of vocational training on unemployment duration in West Germany: a discrete hazard rate model with instrumental variables", Hujer et al (1999a) found that participation in vocational training has significant effect in reducing unemployment duration. However, in a further study on "the effects of public sector sponsored training on unemployment duration in West Germany: a discrete hazard rate model based on a matched sample" (1999b), they discover positive effects only for short term training courses (less than 6 months), whereas long courses do not really have significant positive effects. Additionally, by comparing the effects of on-the-job training programmes with off-the-job training programmes on unemployment duration in another study (1999c), they (the same authors) found that the former has no significant effect on unemployment duration, whereas the latter reduces it in the short-term (with no significant effects in the long-run). This finding corresponds to Pannenberg (1995, 1996), who observed that participation in off-the-job training increases re-employment probability in the short-term.

• O’Connell and McGinnity (1997) analysed for Ireland the effects of off-the-job and on-the-job training programmes (compared with employment subsidy programmes) on employment. They found both types of training measures significantly increase employment rate, whereas the employment subsidy programmes have no significant effect.

• In Sweden, Johansson and Martinson (2000) compared traditional vocational programmes (organised by the labour administration) with a special training programme (carried out in the industry) to qualify the unemployed for jobs in the IT sector and to ensure a sufficient supply of qualified workers. By analysing the effects of both programmes, the authors found out that the IT training programme increases employment probability more than the traditional training programmes. They consider this finding as an evidence that a more focused link with enterprises can increase the effectiveness of training programmes. Moreover, in their empirical study, Richardson and van den Berg (2001) analysed the impact of the Swedish employment training programmes targeting unemployed individuals as well as employed persons who are at risk of becoming unemployed. The results of their study show highly positive effects, with a doubling of individual re-employment rates after completion of the training.

• In the UK, Firth et al. (1999) analysed the effects of both employment vocational training programme and job action programme on the inflows into regular employment in the United Kingdom. Their findings were that "Employment Training Programme" offered a significant positive effect whereas "Employment Action Programme" had no significant effects. These same results were also obtained by a prior study of Payne et al. (1996) who analysed the effects of both programmes on employment.

However, some researchers seem to think that the relationship between training and aggregate socio-economic performance indicators might be more complex than that. Walsh (1993), for instance, was not able to confirm whether there is a positive link between investment in human capital formation and achieving a better economic performance in Ireland. Moreover, there are some research findings which tend to indicate that certain circumstances are more conductive than others in capturing the training effects (or benefits). For instance, Psacharopoulos (1994) found out that returns to education and training appear to decline after a certain point. Using statistical data from the EU, a research undertaken by EIM Consultancy (1995) confirmed this finding. It showed that countries with comparatively low rates of participation in CVT (e.g. Italy, Belgium) perform sometime equally as well as those countries with higher participation rate such as Denmark and the Netherlands. The study concludes that there is an optimum level of return to investment in CVT, beyond which the working spent on training neutralises any expected productivity gain.

Having reviewed some of the leading overall European and international evaluation studies on "returns" to investment in CVT, it important to draw together the following related underlying conclusions and implications:

• In these evaluation studies, the "return to CVT investment" in its strict financial or economic sense of the term, like that calculated for investment in physical assets, has rarely been estimated. What has been done in most cases, is to relate the training inputs, in a manner dependent on the extent to which the available data allows such inputs to be measured (directly or par proxy), to the training induced output or effects. This is partly due to data limitation and lack of reliable information and on the speed according to which the acquired competences and skills depreciate over time, including the possibility of their transfer or use by other employers through the employees’ job-change and mobility. Perhaps, one exception to this lack of estimating the "rate-of-return to investment (ROI) in training, is Mincer’s attempt in his paper (1991) on "job-training: costs, returns and wage profiles". On the basis of the results collected from a range of studies, he estimated, under certain restrictive assumptions, a ROI ranging from 8.7% and 26% (based on assuming, for instance, a depreciation rate of 4%). Concerning the discussed issue of whether there is evidence of under-investment in training, he concluded that there is no evidence. It is a possibility that Mince did not rule out given the wide range of estimated ROIs.

• In spite of the fact that research in CVT performance evaluation may not have produced reliable ROIs in training, it has however shown that, training has, in general, a positive effect on wages (e.g. Lynch, 1992 and Booth, 1991; Groot at al, 1994) and productivity (e.g. Black and Lynch, 1996; Bartel, 1994; Holtzer et al, 1993; Barrett and Oconnel,1998). However, as Bishop (1994) found out, the productivity effect of training can be greater than wage effect. This makes investment in training more profitable for employers than for the employees.

• The observed positive effect of training on wages and productivity may be distorted by the "self-selection effect" in the sense that the training induced benefits are higher for those who choose to train compared to those who choose not to train, were they obliged to train. The study of Groot et al (1994) has shown, from the perspectives of employees, the importance of taking into consideration this issue through a corrective procedure in estimating the effect of training across a range of individuals (otherwise the estimated effects might be over-estimated for some beneficiaries). Also Ichniowski et al (1995) had drawn the attention to this type of effect from the perspective of the enterprises when they were discussing why some firms do not provide training, even though such training appears to have a strong productivity effect when viewed across the sample as whole. Without "self-selection effect" corrections by using the appropriate statistical correction technique, the obtained results concerning training effects cannot be generalised.

• There is evidence suggesting that training received from one employer can increase wages and productivity with another employer (e.g. Lynch, 1992, Bishop, 1994; Barrett and Oconnel, 1998). This finding has at least three basic implications: First, there is the possibility that current employers are inclined to prefer and reward previous off-the-job training on the basis that the content of such training is more general and/or transparent for them. Secondly, previously acquired off-the-job job training (financed by the previous employer) raises the productivity of the current job with less cost to the current employer, thus leading to the creation of "spill-over effect" from one employer to another. Thirdly, due to this "training spill-over effect", there is the possibility of reduced incentive for the employer to invest in training, especially in training of general nature (in the senses of its applicability in other workplaces). For this type of training, the employers might require the beneficiaries to finance it, since they are not in a position to secure the appropriation of any return on their investment in this case, if ever the trained employee were to change the job by moving to another employer.

• There is also evidence from certain evaluation studies (e.g. Black and lynch, 1996; Barrett and OConnell, 1998), suggesting that it is the types of training provided which really matters for training effect on productivity than the investment in training per se. For instance, in Barret and OConnell study (1998), it is the training of general nature which was found to be more effective than "specific training" in contributing to increased productivity. As for Black and lynch (1996), they found out that it is the provision of off-the-job training which matters more than on-the-job training in raising productivity in the manufacturing industry.

• Some of the evaluation studies brought attention to the fact that training, from the perspectives of employees and employers, might have effects other than increases in wages and productivity. For the employees, the other effects may be include reducing the risk of unemployment (Diedderen, 1994), making work more interesting or contributing to increased opportunities for job-mobility (Von Bardeleben et al, 1996). For the enterprises such other important training effect may be reduced staff turnover (Von Bardeleben et al, 1995). In this context, some of these studies indicated the importance of combining training with other human resource measures, with the possibility of making both of them more effective (Ichniowski et al 1995);

• From a macro-policy perspective, most of the evaluation studies conducted inside and outside Europe tend confirm the existence of a positive link between investment in training and the aggregate level of output, income, employment, knowledge development and productivity.

It is a multi-indicator evaluation approach linking "input measures" to foster participation and investment to their "outcomes" through a set of quantitative and qualitative performance indicators. The nature and the use of these (overall and specific) performance indicators (referred to in some of the available evaluation studies, surveys and reports) are determined by a set of influencing factors, namely:

Concerning the levels of evaluations, Kirkpatrick (1959, 1994) (one of the founding fathers of research on "returns" to investment in training) developed and tested what is now considered the classic "Kirkpatrick four-level approach" to assessing the effectiveness of the training measures and schemes. According to this guidance framework, the four-evaluation levels are:

These levels do not represent mutually exclusive outcomes, but rather inter-related, manifestations of the "returns" to investment training.

However, for the purpose of our approach and in connection with the samples of evaluation studies reviewed so far, these levels and related performance evaluation indicators can be reduced to two main evaluation levels with two related basic categories of (quantitative and qualitative) performance indicators:

• The first level is a combination of Kirkpatrick’s first three levels. It is the "primary" outcome evaluation level, which allow for obtaining some primary or indirect evaluation indicators. Most of the estimated indictors within the framework of CVTS1 and CVTS2 surveys fall within the category of "primary" or "indirect" performance indicators such the rate of participating enterprises in CVT and other rates related to the training access, training intensity, the training incidence and training costs. Indicators connected with this primary level of evaluation (such as the rate of participation and satisfaction) were also used in assessing the effectiveness of most of the CVT funding measures and schemes (presented in part I of this paper), especially in the early stages of their implementation and evaluation.

• Within the framework of a multi-criteria cost-benefit linking "input measures" to foster CVT investments to their "outcomes", the performance indicators linked to this level of evaluation will include for instance the contribution of the undertaken measures to the enhancement of:

– The rate of "participating enterprises" in the offer of CVT to their employees (where this participation rate is differentiated according to the impact of the enterprises’ size, the international dimension of the activity exercised, the economic sector, types and modes of CVT provision, the existence or not of CVT provision plan, etc);

– The rate of the employees’ access to CVT (including access to the accreditation of work-based learning) by taking into account the differentiating effect of age, gender and the level of initial qualifications including the socio-professional background.

• The outcome level remains basically the same as the "results" level of Kirkpatrick. It concerns the evaluation of the ultimate outcomes of the implemented training measures and schemes. It allows for obtaining "outcome" performance indicators concerning the attained level in achieving the measures’ objectives. Many of the reviewed overall evaluation studies on "returns" to CVT investment, in the last part of this paper fall within the range of this level of training evaluation, as they indicated that some training input measures led to increased wages and productivity as a training outcome. Other studies indicated increased job-satisfaction and/or mobility as an outcome of an investment input training. On the whole, within a "multi-criteria" evaluation framework, the performance evaluation indicators linked to this level of evaluation will include for instance, assessing the contribution of the training input measures to increasing the level of:

  – The productivity within the firm;

  – The employees’ wages, skills, flexibility and mobility, promotional opportunities and job-satisfaction;

  – The growth rate of aggregate output, income, employment and productivity.

As Bengtsson and Würzburg (1992) pointed out, training is changing in its content and in the way it is provided. This will certainly have an effect on the way the "returns" to such training can be identified and evaluated. There are many ways of categorising different forms of training: for instance, formal and informal learning, on-the-job and off-the-job training. With the exception of the evaluation studies of Grünewwald and Moraal (1996) and of Ichniowski et al (1995), most of the work in this area focused on the off-the-job training despite the increasing importance of learning by doing.

Moreover, training can be distinguished according to its content and objectives. For instance, the evaluation studies (e.g. Black and lynch 1997; Kirkpatrick, 1994) which distinguished training according to its content, were more able to generate interesting results on the relative impact of different types of training. As for the training objectives on the level of the enterprises, they are not always clear or real. Additionally, training may have multiple objectives which cannot be easily ranked.

Due to lack of clearly identifiable specific training objectives, on the level of the enterprises, returns to investment in training are usually evaluated in relation to more general objectives (such as training contribution to increased productivity). The performance indictors used are generally connected with the "tangible" and the "intangible" character of these returns. With regards to tangible returns, a number of studies tend to evaluate training benefits in terms of productivity through a variety of performance indicators such as: increased profitability, reduced costs, improved quality of products and services, reduction of scrapage or faults, increased customer satisfaction, etc. As for the evaluation of intangible training benefits, proxy performance indicators are usually used. This is the case of enterprises which try, for instance, to improve the communicative skills of their employees with customers, through the use of the turnover rate of participants and non-participants as an approximate performance indicator.

Concerning the evaluation of training returns from the perspective of employees, a number of the evaluation studies tend (as indicated in part two of this paper) to use some tangible performance indicators such as increases in wage gains and promotional opportunities. Other evaluation studies (from employees’ perspective) use no-wage related performance indicators such as job-satisfaction or increased chance of mobility.

The nature and use of training performance indicators are usually influenced by the objectives of who funds and eventually receives the training "returns"? In the review of overall evaluation studies presented in part 2, the "returns" to companies’ investment in CVT were evaluated from the perspectives’ of the enterprises, the employees, and the economy as whole (on the state level). Given that there are significantly different CVT funding arrangements in different countries, the performance indicators used were shown to be on the whole linked to a certain number of general objectives as indicated above. However, in spite of their ultimate complementarities, these objectives are not as general (or commonly shared) as such if they are looked at from the perspective of the three basic stakeholders involved in CVT funding. According Westphalen (1999), the CVT funding providers hold different reasons behind their effective participation in CVT financing.

For the enterprises, the main reasons for CVT financing include increased:

The employees’ financial investment in CVT is usually motivated by the perspective of obtaining:

As for the public authorities involvement in CVT funding is motivated by a variety of reasons such as securing:

One of the major issues in CVT evaluation studies is the diversity and interconnectedness of political initiatives and measures to foster participation and investment in CVT. The primary and outcome performance indicators usually used in these evaluation studies are still "summative" in the sense that:

• They do not link, through a "cause-effect" direct relationship, each input policies measure to its intermediary and/or ultimate output/result;

• As proxy/overall indicators, they do not do measure effectively the "incremental" impact of the policy measure evaluated (by isolating the deadweight effect);

For performance evaluation purposes within the framework of the two-level multi-indicator approach, the policy measure fostering CVT can be classified into two basic interrelated categories:

• Measures targeting CVT promotion: They are explicitly oriented towards raising companies and employers’ effective participation and investment in CVT, such as:

  – Direct subsidies and free services provision in terms information, guidance and advice;

  – Cost-reduction measures (e.g. tax relief/refund schemes);

  – Creation of levy/fund schemes;

  – Demand-led funding measures (e.g. individual training leaves, training vouchers, training accounts, etc.);

  – Legal measures regulating access and participation in CVT.

• Composite policy measures, such as multi-targeted active labour market policies and programmes (with more or less diffuse effects).

The evaluation methodology can be improved through the conceptualisation of specifically adapted cost-benefit evaluation criteria which allow for:

• Focussing on the "incremental" effect evaluation by isolating and estimating the deadweight of the measure whose apparent impact cannot be attributed to the undertaken measure itself. Examples are where a subsidy is paid for the enhancement of CVT within a certain type of enterprises which would have undertaken it in the absence of the subsidy, or where an unemployed individual entering employment after training would have found the same job without policy intervention;

• Estimating the degree of the sensitivity of the demand (for a certain type of training) to cost-reduction through CVT targeted policy measures;

• Taking into consideration the targeted actors (enterprises or individuals) actual change of behaviour, attributed effectively to the implementation of the measure.

Most of the reviewed evaluation studies of policy measures fostering CVT within the enterprises found positive training effects. However, very few of them linked these effects (or benefits) back to the investment costs in a way which allows for the determination of a rate of return on investment (ROI) in CVT. Most of the empirical research concerned with estimating this rate of return is rather sparse and underdeveloped compared to other methods of evaluation. Therefore, the question of how much can we get for investing an extra Euro in training remains open for more refined conceptualisation and further empirical investigations.

In terms of the multi-indicator framework sketched in the last section of this paper, the reviewed studies were found to be basically focussed on the evaluation of policies fostering CVT from the perspectives of three basic stakeholders: individuals, enterprises and public authorities. The main findings are as follows:

• Most of the evaluation studies conducted from the perspective of the individuals (mainly employees) tends to indicate that training has a positive effect on their wage level, their employment stability and/or mobility (functional and promotional mobility). Off-the-job training seems to have a higher and explicit effect than on-the job-training.

• From the employers’ perspective, most of the evaluation studies confirm that there is on the whole a positive link between investment in CVT and the achievement of productivity gains. However, many of these studies tend also to indicate that it is the type of training undertaken which really matters for productivity gains than just training investment per se. It is the "general training" in terms of off-the-job training, which seems to have a higher contribution to increased productivity, especially in the manufacturing industry. Additionally, some of these studies tend also underline the importance of linking investment in training to the process of human resource management and development with the aim of making them more effective.

• From a macroeconomic policy perspectives, a number of early evaluation studies (e.g. Becker, 1996) has positively linked training to improvements in knowledge and skills and to output and income increases throughout the economy. Barro (1991) and Mankiw et al (1992) have found a positive relationship between educational levels and aggregate productivity or economic growth. More recently, many researchers in Europe have compared, in their evaluation studies of active labour market policies (ALMP) in Europe, training programmes with other measures such as job-creation and/or subsidised employment programmes. Their findings were that the effect of vocational training programmes on employment and employability is positive in most of the cases, while the impact of other ALMP programmes was basically negative. Furthermore it was found that training programmes that are organised by firms seem to be more effective compared to publicly organised programmes (cf. Hujer, Caliendo and Radic, 2004).

However, there are certain issues (basically methodological ones) which still need to be taken into consideration in formulating further research work on the evaluation of the impact of policies fostering participation and investment in CVT within the enterprises. This includes:

• Given the diversity and interconnectedness of policy measures fostering CVT, the impact evaluation can be viewed as a two-level multi-indicator process. A primary level which allows (in an immediate fashion) for obtaining some primary or indirect (quantitative and qualitative) performance indicators on the participants’ reactions: This includes, for instance (as it was the case with the European CVTS1 and CVTS2 surveys), the rate of participation in CVT and other indicators related to the training access, training intensity, training incidence, etc. The second level concerns basically the end-outcomes of the implemented policies and measures. It allows for obtaining (quantitative and qualitative) "outcome" performance indicators concerning the attained level of the measures stated (and non-stated) objectives, including the effectiveness of the training process itself. Both primary and end-outcome indicators need to be specifically adapted to the nature of the "incremental" effect evaluation by isolating and estimating the deadweight of the policy measure whose apparent impact cannot be attributed to the measure itself.

• Training is changing in its content and in the way in which it is provided. This will certainly have an influence on the way the impact of policies fostering CVT can be identified and evaluated. Given that the boundary between work and learning is increasingly blurred, there are many ways of categorising different forms of training (formal/informal, on-the-job/off-the-job, general/specific). In this context, many of the studies reviewed (e.g. Black and Lynch, 1996; Barrett and OConnell, 1998) indicate that productivity is affected differently according to the type of training provided rather than the overall provision of training in itself. Additionally, the evaluation studies which distinguished training according to its content were able to generate more interesting results concerning the relative impact of different types of training (e.g. black and Lynch, 1997; Kirkpatrick, 1994).

• Moreover, the performance indicators are generally connected with both the "tangible" and "intangible" character of the outcome of CVT fostering policy measures. However, most of the evaluation studies reviewed tend to concentrate on the tangible impact of the policy measures. Therefore, there is a case for generating performance indicators which can measure intangible as well as tangible effects. For instance, some studies have revealed that increased job-satisfaction may be more important than the wage gains for some employees.

• Given that there are significantly different CVT funding arrangements in different countries, the performance indicators are generally influenced by the strategies and the objectives of who funds and eventually receives (directly or indirectly) the training "returns" (or benefits). This is why most of the reviewed evaluation studies were classified according to the perspectives of three basic stake holders: employers, employees and the public authorities. However, both "primary" and "end-outcome" performance indicators are still in need for further refinement in order to take into consideration explicitly the objectives and strategies of different stakeholders involved in CVT fostering and funding.

Notes

1. This paper was prepared as a work package assignment (“overview and analysis of meta evaluation of policies and measures fostering CVT in the enterprises”)  within the framework of the “Preparatory Phase” in Leonardo da Vinci project field investigation: CVTS 2 - Revisited (2004-2007): http://www.trainingineurope.com/.

2. Address:  Dr M’Hamed DIF, BETA/ Céreq Alsace, University Louis Pasteur of Strasbourg ; 61, avenue de la Forêt Noire , F-67085 Strasbourg  (France);   Tel:  +333 88 22 33 47 or + 333 90 24 21 67;  Fax: +333 90 24 20 70/71 ;    E-mail: mdif@cournot.u-strasbg.fr

Abrahamsson, K. (1999). Vocational education and training in Sweden. Luxembourg: Office for Official Publications of the European Communities,.(Cedefop Reference series)

Arret du Gouvernement wallon relatif aux chèques-formation,. 23 juillet 1998. In Moniteur belge 18.8.1998, p. 26249-26250.

Atkinson, D.; Elson-Rogers, S. (1999): The financing of vocational education and training in theUnited Kingdom: financing portrait. Luxembourg: Office for Official Publications of the European Communities, (Cedefop Panorama series)

Bagnall, R. (1994), "Performance indicators and outcomes as measures of educational quality: a cautionary critique", International Journal of Lifelong Education, Vol. 13, No. 1, pp. 19-32.

Barrett, Alan and Philip J. O’Connell (1998), "Does Training Generally Work: The Returns to In-Company Training", Centre for Economic Policy Research (CEPR) Paper No. 1879, London.

Barrett, A. Hövels, B, Boer, P. and Kraayvanger,G. (1998): Exploring the returns to continuing vocational training in Enterprises: A review of research within and outside of the European Union, Cedefop, Panorama, European Communities Publication, Luxemburg.

Barro, R. (1991): Economic growth in a cross-section of countries. In Quarterly Journal of Economics, Vol. 61, 2, pp. 407-433.

Barron, J. M., Berger, M. C. and Black, D. A. (1999), "Do Workers Pay for On-the-Job Training?", Journal of Human Resources, Vol. 34, No. 2, pp. 235-252.

Bartel, A. (1994), "Productivity Gains from the Implementation of Employee Training Programs", Industrial Relations, Vol. 33, No. 4, pp. 411-425.

Bassi L. et al. (2000), Profiting From Learning: Do Firms’ Investments in Education and Training Pay Off?, Saba and ASTD.

Becker, G. (1975), Human Capital, New York: Columbia University Press.

Beicht, U., and E.M. Krekel (1996), "Individuelle Kosten und individueller Nutzen der beruflichen Weiterbildung", in J. Münch (ed.), Ökonomie betrieblicher Bildungsarbeit: Qualität - Kosten - Evaluierung - Finanzierung, pp. 175-194, Berlin: Bundesinstitut für Berufsbildung.

Bengtsson, J., and G. Wurzburg. (1992), "Effectiveness research: merely interesting or actually relevant?", International Journal of Educational Research, Vol. 17, pp.527-535.

Bishop, John H. (1994a), The Incidence and Payoff to Employer Training, Working Paper 94-17, Ithaca: Center for Advanced Human Resource Studies.

Bishop, John H. (1994b), "The Impact of Previous Training on Productivity and Wages", in Lisa Lynch (ed.), Training and the Private Sector: International Comparisons, Chicago: University of Chicago Press.

Bishop, John H. (1995), Expertise and Excellence, Working Paper 95-13, Ithaca: Center for Advanced Human Resource Studies.

Black, Sandra and Lisa Lynch (1997), "How to Compete: The Impact of Workplace Practices and Information Technology on Productivity", National Bureau of Economic Research Working Paper No. 6120, Cambridge, Massachusetts.

Black, Sandra and Lisa Lynch (1996), "Human Capital Investments and Productivity", American Economic Review, Vol. 86, No. 2, pp. 263-267.

Booth, Alison L. (1991), "Job-Related Formal Training: Who Receives it and What is it Worth?", Oxford Bulletin of Economics and Statistics, Vol. 53, No. 3, pp. 281-294.

Bonnal, L.; Fougère, D.; Sérandon, A. (1997): Evaluating the impact of French employment policies on individual labour market histories. In: Review of Economic Studies, 1997, p. 683-713.

Booth, A. and M. Bryan (2002), "Who Pays for General Training? New evidence for British Men and Women", IZA Discussion Paper No. 486, April.

Brodaty, T.; Crepon, B.; Fougère, D. (2001): Using matching estimators to evaluate alternative youth employment programmes: evidence from France, 1986-1988. In: Lechner, M.; Pfeiffer, F. (eds) Econometric evaluation of labour market policies. Heidelberg: Physika Verlag, 2001 (ZEW Economic Studies, Vol. 13).

Cafarella, R.S., and J.M. O'Donnell (1991), "Judging the quality of work-related, self-directed learning", Adult Education Quarterly, Vol. 42, pp. 17-29.

CEDEFOP (1998a): The Financing of Vocational Training in Austria. Thessaloniki (DE and EN).

CEDEFOP (1998b) Le financement de la formation et de l’enseignement professionels en France. Thessaloniki (FR and EN).

CEDEFOP (1998c), Approaches and Obstacles to the Evaluation of Investment in Continuing Vocational Training: Discussion and Case Studies from Six Member States of the European Union, Thessaloniki: CEDEFOP.

CEDEFOP (1999a) Finanzierung der Berufsbildung in Deutschland. Thessaloniki (DE and EN).

CEDEFOP (1999b) The Financing of Vocational Education and Training in Denmark. Thessaloniki (DA and EN).

CEDEFOP (1999c) The Financing of Vocational Education and Training in Finland. Thessaloniki (FI and EN).

CEDEFOP (1999d) The Financing of Vocational Education and Training in United Kingdom. Thessaloniki (EN).

CEDEFOP (1999e): The Financing of Vocational Education and Training in the Netherlands. Thessaloniki: CEDEFOP.

CEDEFOP (2000a) The Financing of Vocational Education and Training in Spain. Thessaloniki (ES and EN).

CEDEFOP (2000b) The Financing of Vocational Education and Training in Sweden. Thessaloniki (SE and EN).

Cockx, B.; Bardoulat, I. (2000): Vocational training: does it speed up the transition rate out of unemployment? Amsterdam: Tinbergen Institute, 2000 (Discussion Paper No 9932).

Cockx, B.; Van Der Linden, B.; Karaa, A. (1998): Active labour market policy and job tenure. In: Oxford Economic Papers, 1998, Vol. 50, p. 685-708.

Coopers & Lybrand (1996), Economic dimensions of education and training in the member states of the European Union, European Commission (document).

Craig, Robert L. (1996), The ASTD Training and Development Handbook: a guide to human resource development, 4th ed., New York: McGraw-Hill.

Dearden, L., H. Reed and J. Van Reenen (2000), "Who gains when workers train? Training and corporate productivity in a panel of British industries", Centre for Economic Policy Research, Discussion Paper Series, No. 2486.

De Koning, J. (1991), Evaluatie en arbeidsvoorziening (The state of the art), Rijswijk: CBA.

De Koning, J., and A. Gelderblom (1992), "Company Training: Volume, Underinvestment and Return", a paper presented to the fourth annual conference of the European Association of Labour Economists, University of Warwick.

Diederen, J. (1994), Opleidingsniveau en beroepsloopbaan. Over de effecten van initiële opleiding en additionele scholing op de beroepsloopbaan van lager opgeleiden, Nijmegen: ITS.

DIF, M. (2000): "On the performance of self-directed learning within the French continuing vocational training system", in P. Alheit, J. Beck, R. Taylor and H.S. Oleson (Eds): "Lifelong Learning-Inside and Outside Schools", Second European Conference on Lifelong Learning, Bremen, 25-27 February 1999, Rosklide University (2000), Denmark, Volume 2, pp. 728-748.

Dougherty, Chr. (1992), "Evaluation of the economic and labour market effects of continuing education and training: practice and policy issues", International Journal of Educational Research, Vol. 17, pp. 549-564.

Drake, K. & Germe, F. (1994) Financing Continuing Training: what are the lessons from international comparison? Thessaloniki: CEDEFOP.

European Commission (1996): White Paper on teaching and learning – towards the learning society, Brussels: European Commission.

European Commission (1999) Continuing Training in Enterprises: facts and figures. Luxembourg: EC.

EIM Consultancy (1995): Continuing vocational training in SMEs. Zoetermeer: EIM.

Elson-Rogers, S. & Westphalen, S.-A. (2000): Funding Continuing Vocational Training in the European Union, in Journal of Vocational Education and Training, Volume 52, Number 4, 2000, CEDEFOP, Thessaloniki, Greece Journal of Vocational Education and Training, Volume 52, Number 4, 2000

Field, J., and R. Taylor, (1995), "The funding and organisation of adult continuing education research in Britain: trends and aspects", International Journal of Lifelong Education, Vol. 14, No. 3, pp. 247-260.

Firth, D.; Payne, C.; Payne, J. (1999): Efficacy of programmes for the unemployed: discrete time modelling of duration data from a matched comparison study. In: Journal of the Royal Statistical Society, 1999, Vol. 162, p. 111-120.

Forrester, K., J. James, and C. Thorne (1996), "Vocational education and training in the UK retail sector", in H.S. Olesen (ed.), Adult Education and the Labour Market II, Roskilde: Adult Education Research Group, University of Roskilde.

Gielen, E. (1995), Transfer of Training in a Corporate Setting, Enschede: University of Twente (diss.).

Gahéry, R. (1996): 1971 - 1996: 25 ans de formation professionnelle en France, In Problèmes économiques, no. 2460, 21 February, pp. 7-12.

Green, A., Hodgson, A., Sakamoto, A. & Spours, K. (2000) Financing Vocational Education and Training, 2nd Report on Vocational Education and Training Research. Thessaloniki: CEDEFOP.

Groot, Wim, Joop Hartog and Hessel Oosterbeek (1994a), "Returns to Within Company Schooling of Employees: The Case of the Netherlands", in Lisa Lynch (ed.), Training and the Private Sector: International Comparisons, Chicago: University of Chicago Press.

Groot, Wim, Joop Hartog and Hessel Oosterbeek (1994b), "Costs and Revenues of Investment in Enterprise-Related Schooling", Oxford Economic Papers, Vol. 46, No. 4, pp 658-675.

Groot, Wim (1995), "Type Specific Returns to Enterprise-Related Training", Economics of Education Review, Vol. 14, No. 4, pp 323-333.

Grubb, W.N. (1995), Evaluating Job Training Programs in the United States: Evidence and Explanations, Berkeley: National Center for Research in Vocational Education.

Guilloux, P. (1996): Le congé individuel de formation: Genèse et évolution d'un droit (1966-1996), Editions L'Harmattan, Paris

Hansson, B., Ulf Johanson, U. and Leitner, K.-H. (2004): "The impact of human capital and human capital investments on company performance: Evidence from literature and European survey results", in Descy, P. and Tessaring, M. (eds): Impact of education and training, Third report on vocational training research in Europe: background report. Luxembourg: Office for Official Publications of the European Communities, 2004, Cedefop Reference series, 54.

Holzer, H., R. Block, M. Cheatham, and J. Knott (1993), "Are Training Subsidies for Firms Effective? The Michigan Experience", Industrial and Labour Relations Review, Vol. 46, No. 4, pp. 625-636.

Hotchkiss, L. (1993), "Effect of training, occupation and training-occupation match on wages", Journal of Human Resources, Vol. 28, No. 3, pp. 482-496.

Houman-Sörensen, J. (1997), Contribution to the Leonardo-project Financing Models for CVT and FTU, Berlin, paper.

Hübler, O. (1998): Berufliche Weiterbildung und Umschulung in Ostdeutschland – Erfahrungen und Perspektiven. In: Pfeiffer, F.; Pohlmeier, W. (eds) Qualifikation, Weiterbildung und Arbeitsmarkterfolg. Baden-Baden: Nomos-Verlag, 1998 (ZEW-Wirtschaftsanalysen, Vol. 31).

Hübler, O. (1994): Weiterbildung, Arbeitsplatzsuche und individueller Beschäftigungsumfang – Eine ökonometrische Untersuchung für Ostdeutschland. In: Zeitschrift für Wirtschafts- und Sozialwissenschaften, 1994, Vol. 114, p. 419-447.

Hujer, R.; Maurer, K. O.; Wellner, M. (1999a ) Estimating the effect of vocational training on unemployment duration in West Germany: a discrete hazard rate model with instrumental variables. In: Jahrbücher für Nationalökonomie und Statistik, 1999a, Vol. 218/5+6, p. 619-646.

Hujer, R.; Maurer, K. O.; Wellner, M. (1999b): The effects of public sector sponsored training on unemployment duration in West Germany: a discrete hazard rate model based on a matched sample. In: ifo-Studien, 1999b, Vol. 3, p. 371-410.

Hujer, R.; Maurer, K. O.; Wellner, M. (1999c ): Analyzing the effects of on-the-job vs. off-the-job training on unemployment duration in West Germany. In: Bellmann, L.; Steiner, V. (eds) Panelanalysen zu Lohnstruktur, Qualifikation und Beschäftigungsdynamik. IAB – Beiträge zur Arbeitsmarkt- und Berufsforschung 229, 1999c, p. 203-237.

Huselid, Mark A. and Brian E. Becker (1996), "Methodological Issues in Cross- Sectional and Panel Estimates of the Human Resource-Firm Performance Link", Industrial Relations, Vol. 35, No. 3, pp. 400-422.

Ichniowski, Casey, Kathryn Shaw and Giovanna Prennushi (1995), "The Effects of Human Resource Management Practices on Productivity", National Bureau of Economic Research Working Paper No. 5333, Cambridge, Massachusetts.

Jensen, P.; Nielsen, M. S.; Rosholm, M. (1999): The effects of benefit, incentives, and sanctions on youth unemployment. Arhus: Center of Labour Market and Social Research – CLS, 1999 (Working Paper, No 99-05).

Johansson, P.; Martinson, S. The effect of increased employer contacts within a labour market training program. Uppsala: IFAU – Office of Labour Market Policy Evaluation, 2000 (Working Paper, No 10).

Kalleberg, A.L. (1992), "Job training in Norway: organizational and individual differences", International Journal of Educational Research, Vol.17, pp. 565-580.

Kirkpatrick, D.L. (1959), "Techniques for evaluating training programs", Journal of ASTD, Vol. 13, No. 11, pp. 3-9.

Kirkpatrick, D.L. (1994), Evaluating training programs. The Four Levels, San Francisco: Berrett-Koehler.

Kraus, F.; Puhani, P. A.; Steiner, V. (1998): Do public works programmes work? Some unpleasant results from the East German experience. Mannheim: ZEW, 1998 (Discussion Paper, No 98-07).

Lechner, M. (1999d): The effects of enterprise-related continuous vocational training in East Germany on individual employment and earnings. In: Annals d'Economie et de Statistique, 1999d, Vol. 55-56, p. 97-128

Leuven (2004), "A review of the wage returns to private sector training", paper presented at the EC-OECD Seminar on Human Capital and Labour Market Performance: Evidence and Policy Challenges", December 8, 2004, Brussels.

Loewenstein, M. A. and J.R. Spletzer (1999), General and Specific Training: Evidence and Implications, Journal of Human Resources, Vol. 34, No. 4, pp. 710-733.

Lynch, Lisa (1992), "Private-Sector Training and the Earnings of Young Workers", American Economic Review, Vol. 82, No. 1, pp. 299-312.

Lynch, Lisa and Sandra Black (1995), "Beyond the Incidence of Training: Evidence from a National Employers Survey", National Bureau of Economic Research Working Paper No. 5231, Cambridge, Massachusetts.

MacDonald, C., J. Bradley Cousins, A. J. Bailetti, and A.H. Rahman (1996), "Toward meaningful evaluation of manager training in technology-based companies", in M. Mulder (ed.), Training in business and industry. Selected Research Papers 1995, Enschede: University of Twente.

Mankiw, G.; Romer, D. and Weil, D. (1992): A contribution to the empirics of economic growth. In Quarterly journal of economics, 57, p. 407-473.

Méhaut, Ph. (1996): Se former tout au long de la vie?, In Céreq Bref, no. 120, May, pp. 1-4

Mincer, Jacob (1991), "Job Training: Costs, Returns and Wage Profiles", in Stern, David and Jozef M. M. Ritzen (eds.), Market Failure in Training? New Economic Analysis and Evidence on Training of Adult Employees, Berlin: Springer-Verlag.

National Audit Office (2002): Individual Learning Accounts, NAO Report, October.

O’Connell, P., McGinnity, F. (1997): What works, who works? The employment and earnings effects of active labour market programmes among young people in Ireland. In: Work, Employment and Society, 1997, Vol. 11, No 4, p. 639-661.

OECD (1996), Measuring What People Know: Human Capital Accounting for the Knowledge Economy, Paris: OECD.

OECD (1997), Labour Market Policies: New Challenges. Lifelong Learning to Maintain Employability, OECD/GD(97)162, Paris.

OECD (1998) Human Capital Investment: an international comparison. Paris: OECD.

OECD (2000) Alternative Schemes of Financing Lifelong Learning. Paris: OECD.

OECD (2004a), Employment Outlook 2004, Chapter 4, Paris.

OECD (2004b), Education at a Glance, 2004, Paris.

OECD (2005): promoting adult learning, pre-publication draft, Paris, March.

Ok, W. and P. Tergeist (2003), "Improving Workers’ Skills: Analytical Evidence and the Role of the Social Partners", OECD Labour Market and Social Policy Occasional Papers.

Onstenk, J., and E. Voncken (1996), The role of the company in generating skills: the learning effects of work organisation, The Netherlands, CEDEFOP document, Luxembourg: Office for official publications of the European Communities.

Ottersten, Eugenia Kazamaki, Thomas Lindh and Erik Mellander (1996), "Cost and Productivity Effects of Firm Financed Training", The Industrial Institute for Economic and Social Research Working Paper No. 455, Uppsala, Sweden.

Ottersten, Eugenia Kazamaki, Thomas Lindh and Erik Mellander (1997), "Evaluating Firm Training Effects on Performance and Labor Demand", mimeo.

Pannenberg, M. (1995): Weiterbildungsaktivitäten und Erwerbsbiographie – Eine empirische Analyse für Deutschland. Frankfurt: Campus-Verlag, 1995.

Pannenberg, M. (1996): Zur Evaluation staatlicher Qualifizierungsmaßnahmen in Ostdeutschland: das Instrument Fortbildung und Umschulung (FuU). Halle: Institute for Economic Research, 1996 (Discussion Paper, No 38).

Paul, J.-J. (1992): Le congé individuel de formation dans les trajectoires professionnelles, In Formation-Emploi, no. 39, July-September, pp. 55-75.

Payne, J., Lissenburg, S. (1996): ; Withe, M. Employment training and employment action: an evaluation by the matched comparison method. London: Policy Studies Institute, 1996.

Phillips, J.J. (ed.) (1994), In action: measuring return on investment, Alexandria: ASTD.

Philips, Jack J. (1997), Handbook of Training Evaluation and Measurement Methods, 3rd ed., Houston: Gulf.

Psacharopoulos, G., and E. Velez (1992), "Does training pay, independent of education? Some evidence from Columbia", International Journal of Educational Research, Vol. 17, pp. 581-591.

Psacharopolous, G. (1994): Returns to investment in education: a global update. In World development, September.

Richardson, K.; Van den Berg, G. (2001) "Swedish labour market training and the duration of Unemployment", 2001 Working Paper, IFAU-Uppsala.

Rist, R.C., and K. Joyce (1995) "Qualitative research and implementation evaluation: a path to organizational learning", International Journal of Educational Research, Vol. 23, No.2, pp. 127-136.

Shelton, S. and G. Alliger (1993), "Who’s afraid of level 4 evaluation? A practical approach", Training & Development, Vol. 47, No. 6, pp. 43-46.

Smith, A. and Billet, S. (2004): Mechanisms for increasing employer contributions to training: an international comparison, NATI, NCVER.

Stake, R. (1973), "The countenance of educational evaluation", in B. Worthen and J. Sanders (eds.), Educational Evaluation, Worthington: Jone.

Stevens, D. W., and J. Shi (1996), New Perspectives on Documenting Employment and Earnings Outcomes in Vocational Education, Berkeley: National Center for Research in Vocational Education.

Tuijnman, A.C. (1992a), "Effectiveness research into continuing education. Introduction", International Journal of Educational Research, Vol. 17, pp. 519-525.

Tuijnman, A.C. (1992b), "Continuing education, training, and lifelong chances: a linear structural relations analysis, International Journal of Educational Research, Vol. 17, pp. 593-608.

Van Leeuwen, S. and J. Dronkers (1992), "Effects of continuing education: a study on adult education, social inequality, and labour market position", International Journal of Educational Research, Vol. 17, pp. 609-624.

Von Bardeleben, R., U. Beicht, H. Herget, and E.M. Krekel (1996), Individuelle Kosten und Individueller Nutzen beruflicher Weiterbildung, Bielefeld: Bertelsmann.

Von Bardeleben, R., and U. Beicht. (1996), " ‘Investitionen in die Zukunft’ – eine Bildungsökonomische Betrachtung der Berufsausbildung aus betrieblicher Sicht", in R. von Bardeleben (ed.) et al., Kosten und Nutzen beruflicher Bildung (Zeitschrift für Berufs- und Wirtschaftspädagogik: Beiheft, 12), pp. 22-44, Stuttgart: Steiner.

Walsh, B. (1993): The contribution of human capital formation to post-war economic growth in Ireland. CEPR discussion paper No 819. London: Centre for Economic Policy Research,

West, A., Sparkes, J., Balabanov, T. and Elson-Rogers, S. (2000): Demand-side financing – a focus on vouchers in post-compulsory education and training: discussion and case studies, CEDEFOP, Panorama, Luxembourg: Office for Official Publications of the European Communities

Terrot, N. (1997): Histoire de l'éducation des adultes en France, L'Harmattan, Paris.

Westphalen, Sven-Åge (1999), "Continuing Vocational Training: from statistical necessity to analytical complexity", Journal of Vocational Education and Training, Issue 51.1.

Winter-Ebmer, R. (2001): Evaluating an innovative redundancy-retraining project: the Austrian Steel Foundation. London: CEPR – Centre for Economic policy research 2001 (Discussion Paper, DP2776).

York Consulting Ltd & MORI (2002): Evaluation of Learning Accounts: Final Report", June.

Zweimüller, J.; Winter-Ebmer, R. (1996) : Manpower training programmes and employment stability. In: Economica, 1996, Vol. 63, p. 113-130.

This document was added to the Education-Line database on 07 September 2005