|
|
 |
By Wolfgang Ilg
Group travel is among the most attractive activities of youth work.
In Germany, it is very common for groups of ten
to thirty teenagers to participate in two-week
youth travel camps led by volunteers or professional
youth workers during summer vacation. Youth exchange
programs are additional attractions of youth travel; for example, the
meeting of groups of young people from different countries at a youth
hostel. But, these types of youth programs require careful evaluation.
Educational challenges, responsibilities, possibilities, risks, and
the amount of money exceed those in other fields of youth work by far.
The ability to carry out an appropriate evaluation offers many advantages.
Internally, it supports the development of a clearer and stronger profile.
Externally, it facilitates a self-confident depiction of pedagogic chances
within the working field. Additionally, the establishment of an "evaluation-culture" can
contribute to the quality of discussion within
a whole organization, which offers supervisors
the chance of a goaloriented development of their
work.
But, appropriate evaluation instruments were
not available for a long time. Many teen travel
practitioners worked with "self-made" questionnaires,
which often had to be evaluated through tallies.
On the other hand, current academic research only
shows intricate studies of a few selected programs,
which are not always transferable to a wider context.
Consequently, a cooperation of academics and practitioners initiated
the development of standardized and widely applicable questionnaires
for the evaluation of youth camps. The studies resulted in a standardized
evaluation-procedure that can now be used by anyone who leads youth
travel groups. The CD-ROM, Jugend und Europa (Youth and Europe), (sponsored
by the Federal Agency for Civic Education in Germany) contains all questionnaires,
tables for the comparison of the results, and an easy-to-use computer
program called "GrafStat" for
statistical analyses. To avoid fears of external
control, the procedure can be used by the organizations
independently. Afterward, they can voluntarily e-mail their data to
the project team, where all data is anonymously collated for a continuous
evaluation.
Methodology of the Standardized Evaluation
The CD contains
a series of questionnaires, which can be applied
to different program concepts. The language within
the questionnaires is easy to understand for young people ages thirteen
and up. The questionnaires consist of four pages. They are designed
as protected Microsoft Word documents that can be complemented with
further items. These can either be chosen from a series of scientifically
tested items ("item
pool")
or supplemented with individual questions.
A questionnaire
for members of staff (referring to the team's goals) has to be
filled in by the whole team either before or during
the first days of the program. The questionnaire
for participants should be completed by the day
before the last day of the camp program. The completion of the form
takes about twenty minutes. Alternatively, "GrafStat" also
offers the opportunity for a "screen interview," where participants
answer their questions directly on a computer.
The answers from all questionnaires, both staff
and participants, are entered into a template,
which is provided by the software, and the rest
is automatically done by "GrafStat." The
user receives data, graphics, and tables with statistics
such as mean values for all aspects of the youth
camp. The evaluation provides information such
as socio-demographic data (e.g., age and type of
school); degree of satisfaction (e.g., journey, food, rules); and answers
to central statements such as, "We (the
campers) had enough opportunities for an active
participation within the youth camp."
Advantages
of the Evaluation Procedure
Because of the high
demand for the CD, a second edition had to be published
briefly after the CD first came out. What are the
reasons that led to this increased interest among
operators of youth travel and international exchange
programs?
- Reliable
results can only be achieved through a scientifically
developed questionnaire. The wording of questions
as well as the design of the questionnaires (a
predominantly sevenfold ranking scale) had been
refined in a long process. Test runs with thousands of participants
were carried out and finally confirmed an efficient procedure. The
questionnaires contain only questions that have proven to make sense
and which have shown stable results over a longer period of time (assured
through a follow-up questionnaire in the baseline survey). Thus, a
reliable evaluation instrument could be provided.
- The
benefit of a questionnaire is strongly linked
with important prerequisites: a) participants have to understand the
questions and b) they have to be prepared to complete the form. Therefore
an additional item within the above mentioned baseline survey was
dedicated to the question, of how participants assessed the completion
of the questionnaire. Answer: Only 15 percent said that the completion
of the questionnaire irritated them. And, 62 percent thought the questionnaires
were "okay"; 23 percent said that filling in the questionnaire
was fun (N=1415). Therefore, young peoples' disposition
to feedback should not be underestimated.
- The interpretation
of data is particularly interesting, if further
comparable data is available. Data that originates
from the baseline survey in 2005 can be used
for such comparison. Every organization can compare its own data with
the 2005 results. A comparative table is added to the CD Jugend und
Europa. The table allows the comparison between one's own results
and the results of other camps. This supports independent
benchmarking, while the control over the interpretation
stays with the camp organizer. The continuous
data flow from camp organizers to the project team provides a data-basis
for updating the comparison tables.
- An
evaluation of data clarifies and helps us appreciate
the individuality of different types of programs
rather than just assuming that all are equal.
According to the principle that a good youth camp/ exchange program
is a program that achieves its own goals, the goals of the team will
be compared to the statements of the participants in the end. It is
particularly interesting to compare different camps with each other
as this will show clearly if, and to which degree, the goals of the
teams can be found within the participants' feedbacks. As an example,
the graph on page 20 depicts this kind of coherence for the subject
"political stimulation": each dot represents a different camp within
the baseline survey. On the x-axis it shows the intensity of the average
employees' goal, "The participants should receive impulses for social/
political themes." An average participants' feedback with regards
to the question, "During this camp I dealt with a number of social/
political topics" can
be found on the y-axis. The ellipse shaped cloud
of dots depicts a high coherence between employees'
goals and actual achievements (correlation between goals and achievements
r = 0.76): low goals among the members of staff correlate with low
results among participants; high goals correlate with high results.
Every team decides individually if political discussions are desired.
The message is clear; those who want to stimulate political discussions
normally succeed in doing so. A similar scientifically recorded influence
of predefined pedagogic goals on the effects described by participants
is very rare. The evaluation of youth camps/international
exchange programs can open supervisors' eyes: it can contribute
to a general recognition of the meaning, which
one's own pedagogic
work can have with regards to young peoples'
experiences during their stay.
Perspectives
The described evaluation
system is a free service that is designed to support
teams leading youth travel groups. It has become
the standard procedure for evaluating youth group travel in Germany,
France, and Poland.
The approved procedure is currently being adapted
and further developed for several projects in different
contexts. The development of new versions in different languages is
one of these processes (there is currently only a first test version
of the questionnaires available in English). Another step is the development
of shorter and simpler questionnaires for children between eight and
twelve years of age. The further development of the evaluation tool
might also become a starting point for cross-Atlantic exchange on academic
research concerning youth travel — there
is quite a lot to discover on both continents!
References
Ilg, W. (2008).
Evaluation von Freizeiten und Jugendreisen. Einführung und Ergebnisse
zum bundesweiten Standard-Verfahren. Hannover (Germany): aej. This book
is only available in German.
Dubiski, J.; Ilg, W. (eds.) (2008). Evaluation
Internationaler Jugendbegegnungen. Ein Verfahren zur Auswertung von
Begegnungen. Berlin (Germany); Paris (France); Warsaw (Poland). This
book is available in German, French, or Polish.
Note: The CD, Jugend
und Europa, is enclosed in the books above. It
contains all material in German, French, and Polish, as well as a first
pilot version of the questionnaires in English.
For further information,
visit www.freizeitenevaluation.de.
Originally published in the 2009 July/August
issue of Camping Magazine.
|
