Technologically and linguistically adventurous EFL teacher, trainer, writer and manager

Posts tagged ‘evaluation’

NILE MAPDLE MAT: Materials Development module (week one)

This is my second NILE MA module, Materials Development for Language Education, abbreviated to MAT. I have previously complete the Trainer Development module. You can see my related blog posts here.

Here are various bits and pieces from week one of the course, things which I wanted to remember, notes I’ve made while reading, and on-going tasks we’ve been asked to provude. The notes are there for me, but you may find something useful in there, or something you’d like to investigate further. Please note: this is not intended as a subsistute for doing this reading yourself – it’s very subjective and based on my interests! I’ll post one of these in each of the three weeks of the online course.

Unit 1: Introductions

My metaphor for coursebooks is that they can be a guidebook:

  • It shows you where you can go, but you can pick and choose.
  • There are lots to choose from – different styles suit different people.
  • Some people don’t bother with them and prefer to explore by themselves.
  • People use it in different ways: some read cover to cover, some dip in at random, some know exactly what they’re looking for.
  • You can pick up all kinds of interesting or unusual ideas from it.
  • They can inspire you to want to try new things, or tell you more about places (methods) you were already familiar with.
  • It can date quite quickly!

Initial beliefs about Teaching, Learning and Materials

These are some of my own beliefs about teaching and learning materials, compiled at the start of the course. The ‘But…?’ part is possible arguments others might have against these beliefs, not necessarily reflecting what I think. I haven’t included the counter-arguments as that would make it very long!

  • Teaching and learning materials should be engaging for both learners and teachers.
    Why? If teachers or learners aren’t engaged by the materials, they won’t want to interact with them, and they are less likely to be open to learning/teaching with them.
    What does it entail? This involves having a clear and transparent aim for the use of the materials, which both learners and teachers can see will develop the English level of the learners. It also involves choosing engaging topics, with clear reasons for learners to care about the topics and the aims of the materials. Those reasons are most likely to come from helping learners to personalise the topic in some way and/or connect it to their own experience. Good design is also an important component of engagement – we have to want to pick up the materials / open the website.
    But…? Who decides what is engaging? What role does the teacher play in bringing materials to life? What about self-study materials which need to be self-mediated? What about learners/teachers who feel uncomfortable sharing personal information?
  • Materials should enhance and support the learning experience for all learners.
    Why? If they don’t do this, then they’re making our jobs harder in some way! Materials which don’t support the learning experience add unnecessary barriers for learners and teachers, and can demotivate them.
    What does it entail? A smooth User/Learner Experience (UX/LX) is important – finding your way around the materials easily and with the minimum of stress. This should be true for every learner, not just those who are neurotypical. We need to make sure as many learners as possible are catered for with our materials. This can be done through design aspects, such as our choice of fonts or spacing, as well as through the types of tasks and the options we provide within materials.
    But…? How do we know that materials which work for one learner will necessarily work for another? Is there enough space in the materials to provide the necessary support? Or enough time to create materials with this level of scaffolding? Is it the materials job to do this, or should it be the teacher’s?
  • Materials should provide opportunities for interaction.
    Why? We learn better when we are actively involved, rather than passively receiving information. We retain new knowledge for longer.
    What does it entail? This interaction could be with other people, for example sharing or explaining ideas. It could be interacting with the materials themselves, through creating our own notes (as I’m doing now!), diagrams, or summaries of the information. Each of these methods force us to process the content of the materials in some way.
    But…? What if learners don’t want to interact with others or with the materials? What if they prefer to just be ‘fed’ information? What happens if you’re working with large groups? How can you manage noise levels during social interaction, or monitor effectively online, or check that they have processed information effectively when they interact with the materials by themselves?
  • Materials should not just be about language; they should also include learner training, and, where necessary, teacher training.
    Why? We often make assumptions that learners know the best way to learn, but this is rarely true unless they are very experienced language learners, and even then they might pick up something new. Teachers also benefit from support within materials – this is a very valuable avenue of professional development.
    What does this entail? Materials should be accompanied by teacher’s notes, explaining the rationale behind methods used, and feeding in variations and extra ideas to support teachers, as well as cultural or other supporting information as appropriate. Learner training can be highlighted by feeding in ideas directly in learner materials, or via teacher’s notes, showing tips and tricks to help them become more effective language learners, and encouraging them to reflect on the learning process and what does and doesn’t work for them. This is particularly true of areas like revision and memorisation, where our instincts might run counter to what science shows are effective learning strategies.
    But…? Is it the job of materials to teach teachers? How do you decide what assumptions you should have of learners’ language learning skills or teachers’ methodology knowledge in terms of what you decide to highlight/omit?
    Note: I believe this is to some extent what Allwright (1981: 9) calls ‘guidance’ [see quotes below for full reference].

What do we want teaching materials for?

I found this quote from Allwright thought-provoking, partly because of my interest in classroom dynamics, but also because of how many people I know who think they ‘can’t’ learn languages because, I suspect, of attitudes that were ‘available to be learned’ in the classrooms they studied in:

Attitudes

It is well accepted that one of the goals of school language instruction is to improve the attitudes of speakers of different languages to one another. However seldom this may be achieved, the development of positive intercultural attitudes remains important, but it is not often discussed as part of the content of instruction. Even where attitudes are not being explicitly ‘taught’, however, they are almost certainly ‘available to be learned’ in any language classroom, from the teacher and from everyone present. They include attitudes to learning, of course, and not just language or intercultural attitudes. To summarize, anyone involved in the management of language learning has necessarily to deal with attitudes as part of what learners may learn.

Allwright, R. L. (October 1981) ‘What do we want teaching materials for?’ ELT Journal Volume 36/1, p8

Another quote from the same article:

‘What activities, or what learning tasks, will best activate the chosen processes, for what elements of content?’ A less deterministic version of this question might be ‘What activities of learning tasks will offer a wide choice of learning processes to the learner, in relation to a wide variety of content options?’ This amendment suggests, I think correctly, that we can neither predict nor determine learning processes, and therefore perhaps should not try as hard to do so as we usually do in our teaching materials.

Allwright, R. L. (October 1981) ‘What do we want teaching materials for?’ ELT Journal Volume 36/1, p9

It’s interesting that this quote is nearly 40 years old, and yet the concept of learner choice with regards to processes or content is still not really all that common within materials.

Allwright also mentions the implications for teacher training of his views of materials. Here are a couple of excerpts:

Teachers, it appears, seem to do ‘all the work’ and exhaust themselves in the process. [Allwright goes on to describe the results of this, such as failing to present the language to be learned as clearly as intended]

If, however, we entertain the possibility that teachers are not just doing ‘too much’ work, but doing work that the learners could more profitably be doing for themselves, the immediate implication for teacher-training must be that teachers need to be trained not to do so much work, and trained instead to get the learners to do more. Hence the concept of ‘learner-training’, since it is unlikely that learners will be able to share the burden without some preparation.

Teacher ‘overload’ often entails learner ‘underinvolvement’ since teachers are doing work learners could more profitably do for themselves.

‘Involvement’ means something akin to Curran’s ‘investment’ (Curran, 1972 and 1976), which suggests a deep sort of involvement, relating to the whole-person. [including decision-making and management of language learning]

Allwright, R. L. (October 1981) ‘What do we want teaching materials for?’ ELT Journal Volume 36/1, p10

I think this is a balance many teachers, particularly those new to the profession, struggle with – they feel like they need to be seen to be teaching demonstrably to meet learners’ expectations. It’s a real challenge for them to let go. This reinforces my belief above about the importance of teacher’s notes and guidance in terms of how to use materials and how to learn effectively.

He goes on to suggest how teachers can share their expertise with learners, without imposing it on them, in order to make learners more independent:

I suggest that teachers, in addition to their role as ‘activities managers’ in the classroom, need to accept the roles of:

1. ‘ideas’ people, ready with practical advice about language learning strategies and techniques, both for classroom and for outside use;

and 2. ‘rationale’ people, ready to discuss language learning and justify their opinions and advice.

Allwright, R. L. (October 1981) ‘What do we want teaching materials for?’ ELT Journal Volume 36/1, p14

For me, this demonstrates the importance of teachers being (language) learners themselves, as they can then share ideas and rationale that have worked for them. While it’s obviously possible to be an excellent language teacher without ever having learnt another language, I do think it can make a huge and very valuable difference (said as an avid language learner myself!)

This is the final sentence from the article:

The most important point for me is that materials should be related to the conception of the whole of language teaching and learning as the cooperative management of language learning.

Allwright, R. L. (October 1981) ‘What do we want teaching materials for?’ ELT Journal Volume 36/1, p16

I feel like this is far from true in most materials and most contexts – the teacher uses the book they have chosen/had chosen for them, and they manage the language learning, with learners the somewhat passive recipients of this learning management, regardless of how active they may be in a given lesson. This teacher-/materials-mediated learning may fit into a broader plan of what learners are doing to improve their language, for example through self-study, but there is rarely a connection that could be described as ‘the cooperative management of language learning’.

Why use textbooks?

Robert O’Neill wrote a (kind of) response to Allwright’s article. This is my favourite paragraph from it, particularly the third sentence and the final one.

O’Neill, Robert (January 1982) ‘Why use textbooks?’ ELT Journal Volume 36/2, p107

Even though technology has moved on a lot, and textbooks are more often than not ‘glossy, glittering products in full colour’, I think they are still good value for money and easy to use.

Further down the same page, we find:

In my opinion it is important that textbooks should be so designed and organized that a great deal of improvisation and adaptation by both teacher and class is possible.

O’Neill, Robert (January 1982) ‘Why use textbooks?’ ELT Journal Volume 36/2, p107

I’m not convinced how possible this is in the market-driven production of coursebooks which we have today, in terms of how materials writers might put products together: everything needs to have its own USP, and be seen as a complete package. Having said that, this view has implications for teacher training and learner training: both need to know how to improvise and adapt materials as appropriate to meet language learning goals. O’Neill goes on to share his own implications for teacher training:

There can be no model of an ideal teacher, or lesson, or learner (or textbook). […]

A teacher-training programme must seek not to mould all teachers according to a pre-conceived notion of what teachers should be, but must try to build on the individual and differing strengths of each teacher so as to make the maximum effective use of that teacher’s qualities.

O’Neill, Robert (January 1982) ‘Why use textbooks?’ ELT Journal Volume 36/2, p108

I think all I can say to that is: Amen!

O’Neill gives an example of a textbook unit with three different objectives designed to cater for learner choice. This is an idea I’d like to explore further, based on his statement that:

There are many ways of designing textbooks so that they can be used by a variety of learners with a variety of ultimate goals, and so they can be taught by a variety of teachers with a variety of teaching styles.

O’Neill, Robert (January 1982) ‘Why use textbooks?’ ELT Journal Volume 36/2, p108

I found myself nodding along to the final paragraph of the article. It’s over a page long, but I feel like these excerpts summarise O’Neill’s ideas:

Textbooks can at best provide only a base or a core of materials. They are a jumping-off point for teacher and class. They should not aim to be more than that. A great deal of the most important work in a class may start with the textbook but end outside it, in improvisation and adaptation, in spontaneous interaction in the lcass, and development from that interaction. Textbooks, if they are to provide anything at all, can only provide the prop or framework within which much of this activity occurs. Textbooks, like any other medium, have inherent limitations. The authors of textbooks must make it clear what those limitations are.

O’Neill, Robert (January 1982) ‘Why use textbooks?’ ELT Journal Volume 36/2, p110-111

The roles of English teachers and materials

This section is a copy of a(n over)long post I put in the forum – I doubt it makes sense without the article itself! The post was based on McGrath, I. (2013) Teaching Materials and the Role of EFL/ESL Teachers, Bloomsbury pp. 2-24

  • Which of the models (Figs. 1.1 – 1.4) best represents your relationship (and that of your learners) with materials?

I think 1.3 (pdf p17) or 1.4 (pdf p18) most closely represent my relationship with materials, depending on who I’m using them with. With adult groups, I think I’d lean more towards 1.3, with learners taking more responsibility for creating content and with less of hierarchical nature in the relationship. With teens and young learners, I suspect it’s more 1.4, with the materials taking precedence in many ways, but trying to feed in bits and pieces of the children’s lives, largely because I feel less confident teaching them – the materials may serve a little as a crutch here too. My role is to try to reduce the distance between the learners and the materials, especially with teen books (the bane of my life!)

  • How important for you are the advantages listed in Section 2.3?

As a new teacher, coursebooks were particularly useful for me, and I still find the ‘visible coherent programme of work’ (point 2) to be helpful, though I’m better able to make one of these myself now.
The time-saving element (point 1) is also very important, as it often takes much longer to create the programme of work and find the materials for it yourself than it does to riff off a coursebook.
I learnt a lot from Teacher’s Books, especially English File and Straightforward, when I was a new teacher, due to the clear methodological input in them (point 3).
I also agree with 4, 5 and 7, though I’m not so sure about point 6 in the age of the internet and (for many people, though not all) instant access to up-to-date cultural information.
I’ve found integrated resources to be useful as well, particularly photocopiable extras and suggestions for varying activities – seeing these has provided a lot of the input I’ve had in terms of my own ideas for materials design and different ideas for engaging activities for learners. As a DoS, having editable tests available has also been useful, although they often require a fair amount of work before I’m happy putting them in front of our students! Coursebook software is also very popular at our school, with the Oxford Discover Futures being the best example I’ve used so far.

  • Or are you one of McGrath’s ‘doubting voices’?

Not catering for the whole person, etc: I think this has improved over recent years, though there’s still a real need for a well-trained teacher to mediate the materials for learners and bring them to life. The arguments about catering for different needs regarding who learners are likely to interact with are also being addressed in some modern materials, though there is still a way to go. A wider range of voices can be heard, not only British, American and the occasional Australian or Irish speakers, though they are very much still in the majority: “native speaker norms continue to dominate”. One example of a new course trying to change this norm to some extent is National Geographic’s Voices, which aims to take a more global perspective. The idea of a hidden curriculum or what Jill Hadfield calls a ‘covert syllabus’ is a very interesting avenue to explore too.

Not reflective of research, etc: While there is still work to do in this area, I believe most global coursebooks are now based on corpus research, though many are still heavily influenced by the grammar syllabus. Outcomes by Walkley and Dellar is an attempt to create a more lexically-led syllabus, while still having the overt grammar syllabus many stakeholders might look for in a coursebook. The most recent studies quoted were in 2010 – I’d be interested to see how this has changed in the intervening 11 years. 
In coursebooks I’m aware of, task design has also improved, though this is again not universal. Some books still isolate grammar and present it out of context, but the vast majority of global coursebooks I’ve come across now use a reading or listening text to introduce new language points in context – they don’t always capitalise on this later in the sequence though, with the context being abandoned once rules or practice activities come into play.
The issue of misrepresentation and underrepresentation in coursebooks has also received ever more attention, though changes in global coursebooks are still somewhat glacial in pace! James Taylor and Ila Coimbra have worked on an independently produced series called Raise Up! which aims to be more representative of the real world than a typical global coursebook. I’ve also recently seen examples of other minor changes, such as a family featuring two female parents in a global teen coursebook (a Cambridge one, I think?) Gone, too, (I hope!) are the mother doing all of the housework or the female secretary supporting the male boss in images, though more diversity would still be good to see here.

Marginalise teachers, etc.: 
(pdf, p12) “If teachers hand over responsibility for decision-making to textbooks, the argument goes, this reduces their role to that of mere technicians.” – if they are passive in this, then yes, but it is up to schools, trainers, and managers to make sure that this is not the case, and that teachers are supported in finding their way around the materials, and trained in how to exploit them effectively to meet learner needs. Teachers also need to tell learners why they are making changes: as Bolitho says (pdf, p20), “learners are entitled to know why they are asked to behave in certain ways…and how they can learn most effectively.”
(pdf, p12) “There is now a real danger that it is the coursebook which determines course aims, language content and what will be assessed.” – this was certainly true at our school to a large extent, but I disagree with the wording ‘a real danger’ (note: the section on Control, pdf p22, counters this statement in a way I agree with). For our brand new teachers, this was a boon – 80% of our teachers are in the first 3 years of their teaching career, and this enables us to provide some level of standardisation across the school and maintain a high level of quality in our general English and exam classes (potentially dealing with the deficiences/limitations of new teachers). We train teachers in how to exploit the coursebook and learn more about their group students to adapt it to their needs, as well as learning to critique materials and decide what is good and bad about them (moving towards a difference perspective, reflecting the Harmer quote on p14 of the pdf of reducing “unthinking coursebook use”, as well as the final paragraph of the whole excerpt about implications for teacher education). With 121 and ESP groups we may or may not use a coursebook. Our books are chosen in a (somewhat!) principled way by an experienced senior team who know the school well, and our typical students, somewhat because we are still working on developing these principles (the section on Choice from p20-22 of the pdf is interesting regarding this) 

Unit 2: Learners and Context

The implications of context on materials

Here are three different ways in which context might vary, and my ideas about what implications this might have for a materials writer.

Socio-economic profile

  • Being ‘seen’ in the materials – not only portraying affluent people, but having a range of images shown or experiences described.
  • Realistic target uses of language, for example writing focussing on a range of different genres, not only essays (these may only be relevant for those going on to further study), or doing what Bruno Leys described in The Grammarless Syllabus and focussing on functional language exponents rather than grammar study for learners most likely to use English in vocational contexts, such as working as a mechanic.
  • Acknowledgement of challenges and affordances of people from different socio-economic backgrounds, e.g. time available for learning, money available to invest in learning/opportunities/extra materials/resources, space available for study – for example, materials which require learners to pay separately for access to audio which they then need a quite place and a strong internet connection to access may not be achievable for some learners. On the other hand, learners with a lot of time and money available may require materials which provide lots of in-built opportunities for extending their learning.

Noise tolerance

  • Having quiet/loud variants of the same activity.
  • Balancing the amount of individual and pair/group work.
  • Providing information in teacher’s notes about which activities are likely to be noisier so that teachers can warn colleagues in advance.

Teacher’s training and experience

  • The amount of guidance needed in teacher’s notes: balancing spoon-feeding with support.
  • Providing opportunities for extending/adapting/reducing materials so teachers can use them flexibly.
  • Being aware that materials are not always going to be used ‘as is’ – this may mean including information in teacher’s notes about which activities are reliant on other activities, and which can be used in a more stand-alone way or in a different order.

Considerations I need to remember when writing possible materials for students at IH Bydgoszcz

This is a selection of possible areas based on what we’ve looked at in this unit. I’d be interested to hear what you would add.

  • Age
    Will the materials be for very young learners? Young learners? Teens? Adult groups? Properly adult (i.e. 22/23+) or including older teens/university-age students too?
  • Level
    We teach everything from beginner to proficiency! Also, have students worked through our school to get to this level or have they joined the school at this level? That has implications for the ‘coverage’ of the level and how spiky their profile might be.
  • Resources
    Assuming we’re teaching face-to-face, we have projectors and access to the internet. Teachers can also write on the whiteboard to highlight things on projected materials. Learners have coursebooks, so am I writing a coursebook unit? Or supplementary materials? Or stand-alone units?
  • Time
    Courses are generally 90-minutes x 62 lessons per year, running twice a week. Materials need to comfortably fit that time, with some flexibility for teachers to choose what to use. Time for assessment and building good group dynamics also need to be built in.
  • Socio-economic profile
    As learners can afford private language school classes, they are probably in at least a middle-income bracket. Many of our learners come from families with occupations such as medicine, teaching, law or engineering featuring strongly, or families own their own businesses. Manufacturing and agriculture are also strongly represented. As far as I know, students can all afford holidays, many of them abroad and often in quite far-flung places, despite the Polish zloty being relatively weak compared to the Euro/Pound/Dollar.
    Catholicism is an important cultural influence, and caution should be exercised when dealing with potential ‘hot-button’ issues. Particularly controversial areas in Polish politics in the past few years have been abortion and LGBT rights.
  • Number in class
    Although some students have 121 classes, most students study in groups of 6-12. Materials should include opportunities to exploit the small group nature of the courses.
  • Classroom layout
    Student chairs have small desks attached which can be folded down out of the way. These can be arranged in many different ways. There is a teacher’s desk with connections for a projector, speakers and the internet – this can be move a little, but not much. There are two display boards in every classroom for student work and other important information. Materials can make use of the opportunity to reorganise the furniture, and to display information in different places in the classroom.
  • Noise tolerance
    Teachers generally expect other classes to be noisy at points and quiet at others, though occasionally parents complain if they think there is too much noise when they are listening from outside. Most activities that would be classed as noisy are possible within the school, provided they are balanced with quiet activities too.
  • Collectivism vs individualism
    Learners expect to have individual attention from the teacher, but are also happy to work in groups. Family is very important, and from my observation I believe it is the defining social unit in society. Learners who come from a family background which is considered non-traditional within Polish society may be reluctant to share this information as it can be potentially stigmatising, so this is an area to be treated with potential caution when writing materials. There is generally respect for people in positions of power, including teachers, though there may also be cynicism depending on the people involved. [Please note, these are my personal impressions and should be taken as such. These insights are very interesting and (possibly) more scientific, and seem to reflect at least some of my impressions.]
  • Learner expectations
    For YLs and teens who have come through our school, they expect engaging lessons with lots of speaking, a bit of writing, and enough of a language focus for a clear sense of progress. For adults, or teens joining our school after learning elsewhere, they tend to expect a strong grammar focus with plenty of speaking. Learners expect their teachers not to speak/know Polish, and for lessons to be completely in English, with materials fully in English to reflect that. Adult learners may expect ‘serious’ lessons, especially older learners who have been out of education for a long time. They may be reluctant to do activities which they feel are too childish or game-like.
    Most learners are quite motivated, and if they aren’t, adults tend to quit the course. Teens may be forced to continue by their parents, though thankfully they are very much in the minority. Many students come to us for 6 or more years, working towards Cambridge First or Advanced exams over a period of time. They expect to be trained to succeed in these exams, so materials need to help them achieve this goal, while also catering for the smaller number of students who don’t want to take exams.
    Learners (and parents) also expect high quality classes and to have a clear sense of progress over their time at the school. Materials need to factor in opportunities for assessment to help learners to notice this.
  • Teacher’s training and experience
    The majority of teachers at the school are within the first three years of their career, with an initial CELTA or CertTESOL certificate. Some come to the school with a little prior experience, but most may have only done a few weeks teaching, if any, before they join the school. Materials therefore need to provide guidance and support, be clear and flexible, and be accessible to early career teachers, without assuming too much prior knowledge about how they can be exploited. There is support at the school to help with this, but we also aim to make teachers as independent as possible, so materials which help with this would be a boon.

Cultural appropriacy

‘PARSNIP’ topics are often considered taboo. We were asked to consider whether these topics are appropriate or taboo in the culture we work in. These are my answers for Poland.

  • Politics
    You’d really have to know your group, as politics can be very divisive and controversial in Poland, especially since 2015 or so. As mentioned above, issues such as the politics of abortion and LGBT rights are particularly divisive.
  • Alcohol
    This should be fine, though portrayals of drunk characters may not be.
  • Religion
    Poland is very strongly Catholic, and many issues are tied into religion. Questioning faith or the church would be very controversial. I would generally avoid this topic, unless it was a group I knew well and they specifically asked to be able to talk about it.
  • Sex
    Because of religion as well as the politics of abortion, I think this would be a topic to avoid.
  • Narcotics
    I don’t think I’ve ever come across any particular issues with this, but I’d avoid it as it may trigger religious or political topics.
  • Isms (such as communism or atheism)
    Both communism and atheism are probably topics to avoid, not least because of Poland’s difficult history. However, with a group you knew well who had asked to talk about them, they could be discussed civilly and safely.
  • Pork
    This is Poland’s national meat 😉 so it wouldn’t cause any issues.

Beliefs regarding vocabulary in materials

These are some of my own beliefs about vocabulary in materials at this point in the course. The ‘But…?’ part is possible arguments others might have against these beliefs, not necessarily reflecting what I think. I haven’t included the counter-arguments as that would make it very long!

  • Learners need to be exposed to the same vocabulary in context and in a range of different ways.
    Why? Context aids both understanding and retention. A range of contexts helps learners to see the spread of where the vocabulary can and can’t be used. It also provides exposure to a range of typical co-text – other vocabulary and grammar which typically co-occur with the target items. Fluent language use requires collocational awareness, which cannot be developed if words only appear in isolation.
    What does it entail? Providing repeated encounters with the vocabulary within the materials, for example in a reading/listening text, in vocabulary focussed activities (such as matching definitions), and in models for speaking/writing activities. Highlighting co-text and context and drawing attention to collocations.
    But…? There’s a limited amount of space in materials and a large vocabulary load: how do you decide what takes precedence? How do you ensure that all vocabulary is encountered sufficiently without contrivance?
  • Vocabulary work should provide opportunities for learners to use the vocabulary actively.
    Why? Learners need to experiment with the language and get feedback on how successfully they’ve used it, for example whether they have chosen the correct vocabulary item for a given situation. Saying vocabulary enables them to practise the pronunciation, and writing it, to practise the form.
    What does it entail? Including activities such as opportunities for personalisation, categorisation, and speaking and/or writing using the new vocabulary.
    But…? We need to be clear whether vocabulary is introduced for receptive or productive use. There’s a large vocabulary load, and it’s difficult to provide opportunities to use all vocabulary actively. Learners may be reluctant to experiment with new vocabulary, particularly if they don’t feel confident about it, and may stick with vocabulary they already feel comfortable using.
  • A key component of learning vocabulary is memorisation.
    Why? If we don’t remember the word/chunk, we can’t use it! A larger ‘in-built’ vocabulary store allows more fluent use of English across all areas: reading, listening, writing and speaking.
    What does it entail? Including memorisation stages in the activity sequence, and showing learners why these are useful and how they can work with the same techniques themselves. Including spaced repetition, and requiring learners to attempt to retrieve vocabulary from their memory rather than the teacher/materials always supplying the vocabulary for activities.
    But…? We have translation software and dictionaries, so we don’t necessarily need to memorise vocabulary – we can look it up when we need it. Some people find it difficult to memorise language particularly if they have problems associated with their working memory, and others find it boring or demotivating.

How are materials evaluated at IH Bydgoszcz

We mainly evaluate materials when we look at the spread of coursebooks we use each year to help us decide what was (un)successful, what we want to keep and what we want to replace for the following year – this is ‘pre-use’. We also evaluate potential new materials to use at the school – ‘post-use’. We only do informal evaluation ‘in-use’, listening to teacher and learner comments about what they (don’t) like about books and considering how our use of them may need to change based on teacher/student needs as we go through the year – this is particularly possible if the senior team are teaching from the books themselves. These are some of the ways we evaluate materials:

  • Flick test
    First impressions of the book, including whether teachers/students are likely to want to pick it up, density of information on the page/throughout the book, general impressions of the design (for example, does it look old-fashioned?)
    + Provides a quick way to remember which book is which!
    – Very superficial.
    – Publishers expect this and might put the ‘shiny things’ in the top right corner to appeal to those flicking through.
  • Teacher questionnaire
    For books we’ve used previously, we have a short questionnaire for teachers based on various aspects of the book, including usability, general suitability for their groups, topics, engagement, level of challenge, grammar and vocabulary covered, skills work, whether they would want to use it again.
    + Gives teachers a say in the materials evaluation process.
    + They have first-hand experience of using the materials with students, so their opinions are valuable.
    + It tells us what teachers are looking for in coursebooks in general, informing our decisions about which ones to adopt.
    + Getting a range of opinions about the same books can tell us how they suit different teaching styles / groups.
    – It’s not obligatory, so we only get a few responses.
    – It can take teachers a while to complete.
    – It’s very subjective.
    – Teachers haven’t been trained to complete such questionnaires, and may only have limited awareness of what makes good or bad materials, especially if they haven’t been teaching for long and have little to compare their current coursebooks to.
    – The questions were created by me based on previous experience, without necessarily having a grounding in theory.
  • Trialling materials in class.
    Some teachers might volunteer to test out a lesson or two from a coursebook we’re considering using.
    + We can see how it might work in practice, including possible student responses.
    + It’s practical, using the materials rather than just discussing them.
    – It’s only a snapshot – sometimes one lesson has been fine, but the book as a whole has not worked for our school/ teachers/ students.
  • Student feedback
    Either based only on the book students have been using, or showing them a range of possible books for their level.
    + They’re the end users of the book, so they should have a say in what materials are chosen.
    + Students who have learnt English for a while have quite a good idea of what might be a good/bad English coursebook would be for them.
    + When they can compare books, students can be very responsible and offer considered and useful insights into the materials which teachers/ managers may not have seen.
    – Some students don’t take it seriously.
    – Students don’t necessarily have anything to compare the materials to, and they don’t have training in recognising good/bad materials.
    – It can be very subjective.
    – It can be quite superficial: for example, the design or the topics can influence them, without regard for the quality of the language work.
  • Comparative evaluation
    This is largely connected to the language and skills syllabus, looking at how the coursebook fits into our overall selection of coursebooks, what the progression is from one level or age group to the next is, and whether there is the coverage of language we’d like.
    + This helps us to provide some level of standardisation across the school, and maintains our sense of progression.
    + As it’s partly based on a list of grammar items compiled a few years ago, there is consistency from year to year.
    + We have practice at doing this now, so compare a wide range of different factors, for example: language clarification, topics, skills coverage, flow of units, length of units, and many others.
    – There’s a risk of trying to find a book which is the same as ones we’ve previously used – we may be less likely to take a risk.
    – We may end up focussing too much on the grammar syllabus, without considering other areas as much.

Materials and culture

I’ve put this paragraph here because I need to think about it – definitely requires some more processing before I can fully take it in I think!

Pulverness, A. and Tomlinson, B. (2013) ‘Materials for Cultural Awareness’, page 446, in Tomlinson, B. (ed) (2013) Developing Materials for Language Teaching, pp. 443-459

Another interesting quote from the same chapter:

As readers, we should always be ‘suspicious’ of texts and prepared to challenge or interrogate them. However, in the foreign language classroom, texts are customarily treated as unproblematic, as if their authority need never be questioned. Learners, who may be quite critical readers in their mother tongues, are textually infantilized by the vast majority of course materials and classroom approaches.

Pulverness, A. and Tomlinson, B. (2013) ‘Materials for Cultural Awareness’, page 451, in Tomlinson, B. (ed) (2013) Developing Materials for Language Teaching, pp. 443-459 (my emphasis)

This sentence is part of a section on ‘Critical Language Awareness (CLA)’, the idea that “language is always value-laden and that texts are never neutral” (ibid.) This is not something I’ve ever considered before, though I’m not really sure how you would go about remedying this in mainstream materials production, or even in the small amount of materials I’ll be creating for my MAT assignment. I wonder whether the increased inclusion of critical thinking tasks is enough, though ones I remember seeing don’t necessarily ‘challenge or interrogate’ texts in the materials. This is what they go on to suggest as a possible solution:

Pulverness, A. and Tomlinson, B. (2013) ‘Materials for Cultural Awareness’, page 451, in Tomlinson, B. (ed) (2013) Developing Materials for Language Teaching, pp. 443-459 (my emphasis)

I think some of the questions they mention are reflective of some of the critical thinking tasks now included – I wonder how they would rewrite the chapter if they published it today?

Evaluation of materials

These notes are based on chapter 3 of McGrath (2002) Materials Evaluation and Design for Language Teaching, looking at close evaluation when choosing a coursebook. They are my summary of the main points of the chapter to refer to when writing my MAT assignments, so there’s not much commentary.

The first section concerns using a checklist. The examples of published checklists include the following variants in design:

  • Rating systems
    Value x Merit = Product (from Tucker (1975: 360-1)): Value rated 0-5, Merit 0-4
    Weight / Rating: Ratings 4-0
    Rating and comments: Ratings = Poor, Fair, Good, Excellent (1-4)
    Yes / No / Comment
    Tick boxes (next to some quite long questions, not always with yes/no answers!)
    Yes / Partly / No: scored as 2, 1, 0 respectively
  • Categories
    Pronunciation Criteria (3) / Grammar Criteria (4) / Content Criteria (3) / General Criteria (8)
    General (4) / Speech (4) / Grammar (3)
    Factual Details (16) / Factors (17)
    No categories – only 3 long questions
    Language content (5) / Skills (6) / Topic (7) / Methodology (7) – though somewhat misleading as the questions are long and often cover multiple areas
    Does the book suit your students? (10) / Does the book suit the teacher? (10) / Dose the book suit the syllabus and the examination? (10)
  • Criteria expressed as:
    Noun phrases of 3-7 words
    Statements of around 10 words
    Nouns, occasionally with adjectives (max. 4 words)
    Yes/No questions, all at least 10 words long
    Yes/No questions, sometimes followed by information questions, all at least 5 words, but averaging at least 10
    Yes/No questions, varying in length from 4 to about 20 words

Potential problems with textbook evaluations based on checklists (based on McGrath, 2002):

  • If you decide to have a specific number of items in each category (like the final one which has 10 questions in each), you may exclude important information or include trival questions to make up the number. (p42)
  • Weighting is complicated – it’s important to ensure that different items are weighted appropriately. (p42) This is especially important as weighting can help you to differentiate between materials which may seem to have a similar number of strengths and weaknesses. (p52)
  • Having the same kind of response to every question might not be appropriate – some may lend themselves to a score, others to a comment for example. (p42)
  • It’s important to only have one focus per question. (p42)
  • You need to consider the difference between answers of ‘No’ and ‘Not applicable’, especially if connected to weighting. Do you ignore statements which are ‘Not applicable’? What does this do to your total scores if you have them? (p42-43)
  • Transparency of criteria (p44) – “certain concepts […] may be unfamiliar to or only partially understood by potential teacher-users. (= you very much need to be aware of the target user your checklist)
  • Criteria date – they need to “reflect new insights into language description, theories of learning and teaching and changes in society.” (p47)
  • Evaluation is values laden. (p48)
  • The conflicts “between breadth and depth, between informativity and economy, between the needs of the evaluator and the needs of the checklist designer – if these are different people, and between the forces of conservatism and innovation.” (p48)
  • Making a final decision can still be difficult, as you might struggle to “reconcile strengths and weaknesses in the same textbook” (p53)
  • You have to ensure validity and reliability, perhaps through arriving at a consensus for criteria (inc. involving end users) for validity, and carefully briefing evaluators for reliability. (p53)
  • They can “encourage rather superficial judgements.” (p54)

McGrath (2002: 43) comments that while published checklists “vary considerably in their scope, form, detailed criteria and the terms used to describe criteria”, most make reference to:

McGrath (2002) Materials Evaluation and Design for Language Teaching, p43

Specific areas which criteria might I might want to include when compiling my own list, in no particular order and taken from throughout the chapter:

  • Representation: gender, disability, ethnicity etc.
  • Learner training
  • Purposeful communication (key word!)
  • Rehearsing for real-world target language use
  • “Opportunities to express their own meanings in their own words” (p46)
  • Balance between meaning/use and form
  • Inclusion of pronunciation work
  • Varieties of English represented
  • Authenticity of language
  • Opportunities for assessment

This sums up some of what I’ve written about elsewhere in this post:

The reality is that evaluation is value laden, and this will be less of a problem if evaluators (1) look critically at the criteria formulated by others; (2) are aware of their own values; and (3) in specifying criteria for use by others, investigate and take the values of the ultimate users into account.

McGrath (2002) Materials Evaluation and Design for Language Teaching, p48

McGrath describes some of the potential conflicts inherent in creating evaluation checklists:

McGrath (2002) Materials Evaluation and Design for Language Teaching, p48

The challenge is to “minimise the chance of decisions being taken on the basis of individual subjective judgement.” (p48)

When deciding how to format a checklist, McGrath mentions the following (p48-):

  • Including a summary of basics about the book at the top. (e.g. title, publisher etc.)
  • Decide between (a combination of?) open-ended questions and questions/statements/prompts requiring a tick/score: the latter allows easier comparison and can be completed faster, the former adds information
  • Consider the order of categories / criteria, including whether any overlap
  • Rating, weighting and scoring:
    Rating is often 3-5 points – picking 4 means the evaluator has to make a decision.
    Weighting could be scored, or a system like A / B / N – absolutely essential, beneficial / preferred, not applicable (Skierso, 1991), rated as 4 / 2 / 0 if a numerical score is needed
    Score = R(ating) x W(eighting) (p50)

Improving your evaluation:

  • McGrath advises piloting a checklist if at all possible (p51), preferably against both a familiar and an unfamiliar book.
  • Daoud and Celce-Murvia (1979) suggest group evaluation, by three experienced teachers. (p52 of McGrath), thus creating discussion, a more thorough examination, and shared responsibility.
  • Teachers may need time to understand the checklist, especially important if different teachers have the responsibility for evaluating different materials. Some kind of practice (standardisation?) would be useful by working through a familar book and “checking that all would make similar judgements about its key features”. (p52)
  • In addition to using a checklist, do an in-depth analysis of one or two units, along with analysing some specific features, for example the treament of a particular grammatical feature (Cunningsworth 1995 in McGrath 2002: 54). This “affords an insight into the view of language learning on which the materials are based” (McGrath 2002: 54). However, this can create a lot of demands on the evaluator, requiring a lot of effort and analytical expertise. (p55)
McGrath (2002) Materials Evaluation and Design for Language Teaching, p56

That’s it for week one. Next week: Units 3, 4 and 5.

The consequences of INSET (Martin Lamb)

For homework last night we read The consequences of INSET, an ELT Journal article from 1995 by Martin Lamb (Volume 49 Issue 1, pp72-79). I’m really sorry to keep sharing articles which are hidden behind paywalls 😦 but hopefully my very short summary will give you the general idea. This article was a real eye-opener for me, and I hope you get to read the original at some point!

Abstract

Teachers attending short INSET courses are usually exposed to a great amount of new information and ideas. While this can be exciting at the time, the after-effects may be less salutary. This article describes one particular INSET course and the reactions of the participating teachers one year later. It suggests that very few of this ideas presented on the course were taken up in the way anticipated by the tutors, mainly due to the mediating effects of the participants’ own beliefs about teaching and learning. Any INSET course which is seriously concerned with long-term change in teachers’ practice will have to take these beliefs into account.

Before reading this article, I knew that training that I do is not always taken wholesale into the classroom and incorporated into teachers’ practice – if anyone could manage that, it would be a miracle! But I suspected there were three states for any given activity/theory/idea I might present:

  • No uptake
  • Confusion
  • Complete uptake

How wrong I was! In fact, according to a study done by Lamb there are lots of different ways that ideas from courses can be taken up. Interviewing and observing teachers one year after a 2-week, 25-hour course, Lamb found “seven different ways in which participants had reacted, consciously or unconsciously, to ideas presented on the course” (p75):

  • No update
  • Confusion
  • Labelling (applying a term to an activity they were already doing)
  • Appropriation (justifying changes in teaching not anticipated by the tutors)
  • Assimilation (transferring techniques without necessarily understanding the rationale)
  • Adaptation and rejection
  • Engagement

In short, very few of the ideas from the training were actually incorporated into the practice of the participants, although they had responded positively to the course.

As a result, Lamb highlights the importance of making participants aware of their routine practice and the values [beliefs] behind it. He also reminds us that participants should decide which areas to develop and “formulate their own agenda for change” (p79).

For me, it’s another example of the importance of including an examination of teacher beliefs in training courses, something which I rarely remember being included in any of the training I have done or delivered (!) but will definitely be adding to my training!

Evaluating Lessons

As part of my CAM I had to teach a lesson and then evaluate it. It was my first attempt at Task-Based Learning (more on that in a later post). The evaluation was done in two ways:

  • a self-created questionnaire which I filled in after the lesson.
  • a questionnaire for the students, again which I created.

This is my completed self-evaluation. I based the questions on a handout we were given during the input session, some ideas from ‘Learning Teaching’ by Jeremy Harmer, and my own knowledge of my strengths and weaknesses (hence, for example, the question about instructions).

  • Did you fulfil the lesson aims? What evidence do you have to support this?
    • They will have been introduced to and practised the use of participle clauses to replace both active and passive subject + verb constructions in relative clauses and following conjunctions. (Devastated by the fact…, Because he is devastated; carefully keeping, who is carefully keeping)
      We practised the use in class. Two of the SS will need more practise of this to feel comfortable with it, but the other four had no trouble with the clauses. All three groups managed to include at least one participle clause in their final film review.
    • They will become more accurate and confident in using adjectives they have previously learnt for describing books / films (depressing, entertaining, fast-moving, gripping, haunting, heavy-going, implausible, intriguing, moving, thought-provoking)
      All groups correctly incorporated at least one of these adjectives in their reviews.
    • They will become more accurate and confident in the use of adverbs of degree to modify (the above) adjectives, including the difference in register implied by the choice of different adverbs. (a bit, slightly, a little, really, very, absolutely, rather, pretty, quite, incredibly, extremely)
      Although this was not something which the students included in their reviews, they became aware of register differences which they did not know about before the lesson.
    • They will become more aware of the typical contents and layout of a review. (Introduction including author / director’s name; plot outline; strong points of the book film; weak points; whether the reviewer recommends the book; who the book is suitable for)
      The review improved between the first and second versions as students were more aware of what they needed to include in the review.

  • How involved were the learners in the lesson? Were they responsive to the materials, tasks, each other?
    The learners were really involved in the tasks. They spoke English throughout the whole lesson, and were enthusiastic about writing, which is unusual! The tasks were motivating for them.
  • How closely did you follow your lesson plan? Was it necessary to deviate for the lesson to be successful?
    I didn’t need to deviate at all. The lesson worked without a problem.
  • How realistic was your timing for these learners?
    It was realistic – one task took 5 minutes less, and one 5 minutes more, but overall it was as I planned.
  • Was there sufficient variety in the interaction patterns during the lesson?
    Yes. There was pairwork, groupwork and individual work.
  • Were your instructions clear at all times? Did learners need more / less explanation than you gave them originally?
    The instructions I gave were clear. SS understood what they had to do.
  • What did learners need more / less of in your lesson which you had not included in the plan?
    Nothing – everything they needed was covered effectively in the plan.
  • Was sufficient error correction given? Was the correction clear? Did SS activate the corrected language at all?
    This was only an issue during the controlled practice. SS activated the corrected language by changing answers in their book and then saying the sentences out loud. They then used the newly acquired language in the writing they produced.
  • What were you pleased with at the end of the lesson? Why?
    I was pleased with the way that the students responded to a TBL lesson. They were always engaged and enjoyed the writing – this is unusual as writing is often seen as a ‘boring’ topic.
  • What features of the lesson would you change in the future? Why?
    The only thing I would change is to make it clear to students that they should spend the first five minutes brainstorming ideas and planning when writing their first draft. I wrote this stage in my plan, but failed to do it in the lesson. Apart from that I would teach the same lesson again, depending on the students.

To collect feedback from the students I decided it would be easiest to create a form using Google Docs. Here is a link to a pdf version of the document (SS completed it online).

Lesson Evaluation for SS to complete

Here are a couple of the students’ comments:

  • “I liked the idea of correcting the text we’ve written right in the lesson (for example making the sentences shorter by using participles). I also found it interesting to read the other’s work and trying to compare it or to find some mistakes.”
  • “The most useful thing about today’s lesson was that we could immediately apply new things we’d learnt in the writing and therefore remember them better.”

Overall it was a very useful exercise, and something I will probably repeat when doing further action research. It was certainly good for my ego too! 🙂

Tag Cloud