Education-line Home Page

Performance and Learning Support With On-Line Help Systems:

An Effectiveness Study

Theo J. Bastiaens

Open University of the Netherlands

Jan N. Streumer

University of Twente, the Netherlands

Yvette Krul

Origin Human Performance Consulting, the Netherlands

Paper Presented at the European Conference on Educational Research, Lahti, Finland 22 - 25 September 1999

This study is on employees working in the field of production, planning & engineering and inventory, who depend heavily on a computer for carrying out their tasks. To cut training expenditure the company developed and implemented an on-line help system for all their questions and training needs. The researchers monitored the implementation phase for six months and looked at the usability and effectiveness of the help system in terms of the learning results and the performance improvement of employees.

When equipping the work environment with resources, there is an increasing use of new technologies. A very common way to integrate working and learning with new technology is providing employees on-the-job with programs to process data, for on-line reference, for electronic advice and -training. Some concepts as for example the concept of Electronic Performance Support Systems (EPSS) go further than the provision of stand alone programs and provide a complete integrated electronic environment (Gery, 1989). The goal of all the effort is a better working and learning process. It is based on the assumption that learning this way will create a more active learning process, more concrete forms of learning, an increase in the transfer of what was learned on the job and a reduction in learning time. Another assumption has to do with the expected performance enhancement of individual employees and the organization and refers to higher quality products and processes and productivity growth (Bastiaens, 1997).

Research Questions

In the previous section assumptions on the advantages of working and learning electronically have been mentioned but little has been empirically proven. On the contrary, next to these assumptions also possible disadvantages were found in the literature that could affect the effectiveness. The first disadvantage is related to the learning process. "Just-in time" training at the workplace, providing employees with small task-oriented training granules and employees taking control of their own learning process can create work behavior problems. Clark (1992) argues that employees may fail to build a unified picture of their job when they have to extract information from a help system. Several small information parts will create a fragmented knowledge base. Novices especially need a high level overview of the content to relate details of training. Clark also has her doubts about the learner control and she illustrates this with the research of Milheim and Martin (1991) which indeed proves that learner control is not as effective as instructional control. This means leaving learners to their own devices and let them make their own decisions is not as effective as let the computer environment taking over control and provide a predefined and structured route (for example with CBT).

An other possible disadvantage is related to the performance support and the work that has to be done. Is the support adequate? Will "just-in time" support the "de-skilling" of workers? Will it "demotivate" workers? Or will it automatize the low level tasks and as a result give workers the opportunity to concentrate on high level tasks (Carr, 1992)?

While these issues provide many questions, the study reported here focuses on the following main research question: 'What is the effect of on-line help systems on the performance and learning of the employees".

The Research Setting

This study was done in cooperation with a large Dutch manufacturing firm with 2700 employees. Together with new working procedures and new computer software, the firm also introduced an on-line help system for the support of 300 employees at the departments Production, Planning & Engineering and Inventory. From this population 4 jobs (planning engineer, planner, coordinator and warehouse employee) were selected to participate in this project. A planning engineer supports and prepares orders. He determines the sequence of steps and appropriate equipment. A planner coordinates technical activities and takes care of manufacturing orders. A coordinator takes care of the logistics and supervises the whole project. A warehouse employee is at services of the different departments and projects.

After a selection of these 4 jobs, on the whole 103 employees participated in this project, 50 planning engineers, 26 planners, 4 coordinators and 23 warehouse employees.

The main purpose of this study is to evaluate the existing computer environment and monitor the implementation of a new help system. The whole study consists of three parts: (1) an effect evaluation of the basic training as well as the on-line help system immediately after implementation, (2) an effect evaluation of a functional refresher training as well as the help system three months later, (3) measuring the reactions and opinions of end and key users concerning the whole training trajectory and the on-line help system.

This paper will only examine the second part and focus on the effect evaluation of the on-line help system. The competency of employees is evaluated and the following three hypotheses are tested: hypothesis A) Employees score higher on a case in their own field of expertise then on cases in related fields, B) Employees with an on-line help system score higher on a case in their own field of expertise then employees without an on-line help system, C) Employees score higher on the learner test then on the pretest, after working three months with the help system.

Methodology

The second part of the study, on which this paper reports, can be perceived as a summative evaluation three months after the implementation of the on-line helpsystem and after a specific refresher training course in the field of their own expertise. Employees (n=30) from the above mentioned areas in the division Operations are being asked to solve problems in the their own field of expertise (by means of a case study), but also in three other related fields. This is done because the company has the policy to broaden the present tasks of employees. From this group an additional 11 employees took part in an observation. In Figure 1 the quasi experimental design (a randomized block design) is diagrammed.

O1 X1 O2 O3

 

O1 X2 O2 O3

 

O1 X3 O2 O3

 

O1 X4 O2 O3

 

O1 Pretest, 8 questions (N=30)

X1...X4 Treatment consisting of four cases in every field of profession (n=30)

O2 Observation (n=11)

O3 Learning test (n=30)

Figure 1. Research design of the study.

 

In the pretest 30 employees are requested to answer 8 open questions (declarative and procedural). They have to respond to the questions without the support of the on-line help system. The pretest is used as a reference measurement for the learning test.

After that, the employees from the earlier mentioned areas in the division Operations have to solve problems in the their own field of expertise (one case), but also in three other related fields (three cases). The cases are developed in collaboration with key-users. These key-users are experts in a specific field. The cases are exemplary for their daily tasks. The employees have 20 minutes to solve each case. The key-users are reviewers and they will asses all the results on the cases.

It is expected that: 1. Employees after a specific refresher training course in the field of their own expertise will be competent to solve a complex case within 20 minutes; 2. Employees will score better on a case in the their own field of expertise; 3. Employees using the on-line help system will score higher than employees without a help system.

During their work on the cases, 11 employees will be observed by key-users. The key-users analyze in detail how the employees work with the on-line help system and when they need more information.

Finally, the employees have to sit for a learning test. The test exists of exactly the same sort of questions as the pretest (from a database, all the questions were randomly assigned to the pretest or learning test). The results on this learning test are compared with the results on the pretest. It is expected that employees in the learner test with the on-line help, will score higher than before, without the on-line help system.

Results

Before the hypothesis about the competence is tested (expectation 1, above), the key-users were asked the question what makes an employee competent ? According to these key-users, it will not be necessary for employees to solve every problem themselves. There is no need to know everything exhaustive. On the basis of this information a minimum mean score for the case was decided on: a coordinator is considered to be competent when he has a mean score of 18.6 points on the cases, a planning engineer needs 24 points, a warehouse employee needs a mean of 25.8 points and a planner has to have a mean score of 27 point. Additionally, time limits were set: the expectation is that employees after a specific refresher training course in the field of their own expertise will be competent to solve a complex case within 20 minutes. Table 2 shows the results.

Table 1

Mean score and standard deviation on cases

  n

M1

s.d.

M2

s.d.

M3

s.d.

M4

s.d.

Planning engineer

8

5.4*

6.5

6.1

8.3

1.6

1.6

6.3

5.8

Planner

11

6.5

7.3

4.9*

8.2

2.6

3.6

4.1

5.1

Coordinator

4

7.5

5.3

2.5

5.0

10.3*

1.5

6.4

4.1

Warehouse employee 7

3.0

6.0

7.9

4.5

2.9

3.8

11*

7.8

M1 = mean score on case, field planning engineer (maximum point 40)

M2 = mean score on case, field planner (maximum point 45)

M3 = mean score on case, field coordinator (maximum point 31)

M4 = mean score on case, field warehouse employee (maximum point 43)

* = mean score on case in the own field of expertise

 

As far as the first hypothesis "after a refresher training in the field of their own interest employees will be competent to solve complex problems within 20 minutes" it can be concluded that the scores on the cases are extremely low. Although the employees all had a specific refresher training course, their score three months after implementation of the help-system is far below expectation. No group meets the demands set by the key-users. No group could give a solution to a case within the 20 minutes time limit. It is noticeable that a planner and planning engineer has higher results on other fields then their own.

To test the second hypothesis ("employees have a higher score on their own field of expertise") all scores are standardized first into Z-scores. After a comparison of the Z-score on a case in their own field of expertise (M=.35, s.d.=1.21) with a Z-score on a different field (M=-.12, s.d.=.66) a significant difference appears ( t-test, t= -2.23, n=30, en p=.02). Contrary to the results of the planners and planning engineers, the hypothesis will not be rejected. Thus, employees score higher on a case in their own field of expertise.

To evaluate the surplus value of the on-line help system, the group of employees was randomly split into two conditions; 15 employees were allowed to use the help system when they were working on the case in their own field of expertise, the other employees were not allowed to use the help system while working on a case in their own field. The hypothesis was that "employees with a help system score significantly higher on a case in their own field of expertise". The employees with a help system had a mean Z score of .28 (s.d.1.15), the mean Z score of the group without a help system was .43 (s.d. 1.29). The t-test on these results showed that there is no significant difference between the two groups (t=.33, df=.28, p=.37). The value of the on-line help system was not evident.

To test the learner results, the employees were asked to answer 8 open questions in a pretest without the on-line help system. The 8 questions were derived at random from a data bank with 16 questions.

After the implementation of the on-line help system they had to answer these questions again. Now they had to answer all 16 questions. This time they were allowed to use the help system. The different score between the two test is interpret as learner results. Table 3 shows the results on the pretest and on the learner test.

 

Table 2

Mean score and standard deviation on the pretest (M1) and learner test (M2)

 

n

M1

s.d.

n*

M2

s.d.

Planning engineer

8

4.5

3.1

5

16.6

5.2

Planner

11

6.2

3.5

9

22.3

10.8

Coordinator

4

11.1

7.5

3

22.0

10.7

Warehouse employee

7

10.6

5.3

7

25.5

14.4

M1 = mean score on pretest (maximum point 40)

M2 = mean score on learner test (maximum point 80)

* = the learner test was not completed by 6 employees

To compare the pretest with the learner test first all scores were standardized into Z scores. The hypothesis was that the employees score higher on the learner test then on the pretest. The results on a T-test (t=4.45, df=23, p=.001) show a significant difference. The employees score higher on the learner test after implementation of the help system.

Next to this quantitative measurement 11 employees were observed while using the help system. In spite of a functional refresher training with a practical component (using the help system) it appears that not all the employees can be qualified as an expert. Many of them have problems while working. It appeared even that e.g. three employees still encounter serious problem while starting the on-line help system. Although the content is complete some links from the list of reference to a certain topic fail. It appeared that the most important and mostly used feature is the "search engine" (44%). "Activity scheme per field" (28%) and "work instructions per job" (25%) are also used often. In general, the employees told to be content with the on-line help system. Employees especially like the on-line support. Whenever they have a problem, they can look for information. This is at odds with the sporadic use of the help system. Reasons for the non use vary from "to many mistakes in the help system" to "a preference for books". The information that is looked up, is basically procedural knowledge. Employees do not use the help system to learn more or look for background knowledge. On the question what information is lacking in the help system no univocal answer is given. Most employees do not know what information they really want to be present in the help system. They told us that they need up-to-date information in a complete system with different possibilities for search and retrieval.

All employees are in general positive about the impact of the help system. Employees believe that the on-line help system will stimulate a more effective performance. Although the third part of this study (a questionnaire) will not be elaborated on it is interesting to see the same results in a larger measurement (N=67). Table 4 shows the answers of the employees on the item "traditional handbooks and materials are more effective then the on-line help system ".

Table 3

Results of the respondents on the item "Traditional handbooks and materials are more effective then the on-line help system".

 

Traditional handbooks and materials are more effective then the on-line help system

N %

Strongly disagree

16

23.9

Disagree

13

19.4

Neutral

15

22.4

Agree

15

22.4

Strongly agree

8

11.9

 

76

100.0

Apparently there is a difference between the perceptual and the measured effects (see results on cases).

When the respondents were asked about the critical success factors for the use of on-line help systems it appeared to be the completeness of the information that was seen as most important. Figure 2 shows the frequencies on the seven most important factors.

Figure 2. Frequencies in the critical success factors.

The second critical success factor appears to be the possibility to search on key words. Respondents also think that it is important that the on-line help system has to have a clear structure, not to much layers etc. Least important is the graphic design of the interface. They think that a glossy appearance is not critical for the use of the help system.

Regarding a question about a successful implementation and becoming familiar with the on-line help system, figure 3 shows the results of the respondents. The most important factor for successful implementation is a demonstration of the features of the on-line help system. An information retrieval training making use of the on-line help system makes the help system become more familiar. An other important factor is "the time to practice". Employees need to get familiar with the help system. They want to try and test the help system off the job. Giving them time to learn the help system themselves is also important.

The third in row is the integrated use of the help system during functional training. Whenever employees get a functional training, let them use the help system as a tool. The least important is the exhaustive topic list.

Figure 3. Success factors for the implementation and becoming familiar with the on-line help system.

Conclusions

In general this study shows again the relative impact of electronic sources for the learning process (see also Bastiaens, Nijhof, Streumer & Abma, 1995,1996,1997). Main reasons for these results depend often on technical defects as well as on content problems. Often a system is not ready for implementation. Technical problems make the system work with deficiencies, the content is often not structured enough (only scanned text) or incomplete. Companies, even nowadays, expect employees to use electronic sources automatically. No attention is paid at the implementation process of electronic sources, at the training of information retrieval skills, or at the human interface. Employees do not use electronic resources inherently. Particularly with help systems it is important to learn more about the features and advantages of help systems. The expectations on electronic sources are often pitched to high. Precisely as in this project the company expected the help system to answer every need for training. The regular training program was cut down and the help system was developed separately from the general training. As the results show three months after implementation the performance of the employees on the cases is still below the required competence level. Even if it is recommendable to develop a on-line help system then a integrated development and use of the help system in the general training is important. Employees need to learn how to use an on-line help system.

References

Bastiaens, Th. J. (1997). Leren en werken met Electronic Performance Support Systems. (Learning and working with Electronic Performance Support systems). Dissertation University of Twente, Enschede.

Bastiaens Th. J., Nijhof, W.J., Streumer, J. N. & Abma, H.J. (1997). Working and Learning with Electronic Performance Support Systems: An Effectiveness Study. Enschede. Training for Quality, 5(1), 10-18.

Carr, C. (1992). Performance Support Systems- The Next Step? Performance & Instruction, Feb., 23-26.

Clark, R.C. (1992). PSS- Look before you leap: Some Cautions About Applications of Electronic Performance Support Systems. Performance & Instruction, May/June, 22-25.

Gery, G. J. (1989). Training vs. Performance Support: Inadequate Training is Now Insufficient. Performance Improvement Quarterly, 2(3), 51-71.

Milheim, W. D. & Martin, B.L. (1991). Theoretical Basis for the Use of Learner Control. Three different perspectives. Journal of Computer-based Instruction, 18(3), 99-105.

 

This document was added to the Education-line database 21 September 1999