Show Notes

• Free CELPIP Practice Tests
• Handouts
• YouTube CELPIP Channel
• Your feedback is important to us. Please take a few minutes to fill out our survey.
  https://www.surveymonkey.com/r/celpippodcast

Transcript

CJ [00:00:00] Hello, everyone, and welcome to the Official CELPIP Podcast, where we aim to help you, our test takers, get the best possible score you can and support newcomers building a life in Canada. My name is CJ, and along with my co-host, Chris, we talk to a variety of guests from test takers, language teachers, and test raters to employment counselors and immigration consultants, just to name a few. We also bring in our in-house staff on the show to get their perspective, and they’re the people who are in the company that work behind the scenes to make the CELPIP Test available to you. Hello, Chris, how’s it going?

Chris [00:00:39] Hello.

CJ [00:00:40] How are you, Chris? I know that you’ve been pretty excited about doing this episode for a while now. Maybe you can tell our listeners why.

Chris [00:00:48] Well, I would say this is the most behind the scenes episode that we’ve ever done, because we’re going to speak to a former Writing rater and discuss the rating process a bit as well. I’m excited because I know this is something that test takers are going to appreciate—you know, a bit of insight into how their responses are assessed.

CJ [00:01:08] Super interesting, and we’re going to be talking to a special former rater. Is that right?

Chris [00:01:14] Yes. Meaghan is probably well-known to many of our listeners. Along with Brandi, she presents many of the webinars that we offer each month. Meaghan’s background as a Writing rater gives her a unique perspective on the preparation process, and not only was she a rater, she was a benchmark rater.

CJ [00:01:34] Hmm, sounds important. Can you tell our listeners what the difference is between a benchmark rater and a regular rater?

Chris [00:01:41] Well, essentially a benchmark rater checks the work of other raters when there is disagreement among them. But actually, Meaghan could probably explain this much better than I can.

CJ [00:01:51] OK, well then let’s introduce Meaghan and find out. Meaghan, welcome. How are you doing today?

Meaghan [00:01:59] Thank you. Doing great. I’m glad to be joining the podcast.

CJ [00:02:03] So could you tell our listeners a little bit more about your background and your background as a rater?

Meaghan [00:02:08] Oh, sure, I became a CELPIP rater in the fall of 2013, and about six months later I was asked to become a benchmark rater. And that is somebody who’s picked by the scoring team, partly based on their education or their work background, and then partly based on their performance, to basically handle rating responses that get very different ratings in the initial round of rating, so they need to be looked at again, they need to be reassessed. So I did both roles. I was both a regular rater and a benchmark rater until the spring of 2019, and that’s when I got hired to the Instructional Products team. So that was about five and a half years altogether, and rating test takers’ responses was the main part of the job, but there were also weekly communications and exercises from the scoring team, and as a benchmark rater, I took part in conference calls with the scoring manager and the other benchmark raters. And in those sessions, we would discuss responses that we ourselves had found challenging, and we talked about how we would rate them and why. And that was really fun and also very valuable just for sort of establishing a shared perspective among all of us.

Chris [00:03:26] And could you also explain the writing process for the Writing Test? You know, like how many raters and all that?

Meaghan [00:03:32] Yeah. So each test taker completes two Writing tasks, and each of those two is randomly assigned to two raters. Those two raters, well four all together, work independently, and the response that they get, they rate it in four categories—and I won’t go into detail about the categories, but you can learn more about those on our website and also at our Writing Pro webinars. So all of the four raters submit their assessments and then the test taker’s CELPIP Level is calculated electronically according to all of those pieces of information.

Chris [00:04:10] And what about your role as benchmark rater?

Meaghan [00:04:14] So as, as you said, benchmark rating is what happens when the initial ratings in that first round don’t match closely enough, they’re too far apart. And what’s interesting—in my five years or more on the job, I rated over thirty thousand responses, and every single one of them was different. And sometimes there’s something a little tricky or unusual about a certain response that maybe makes it a little more challenging to rate. And it could be anything, or it could be a combination of things, things like language structure or word choice or interpretation of the question or their tone, any number of things. So if there’s too much difference in the first-round ratings that are submitted for a test taker’s response, the system will not accept that, and then depending on where the differences lie, one or both of their responses is automatically assigned to a third rater, which is a benchmark rater. So the benchmark rater has no information about the ratings that were given by the original two raters, and they just rate the response as they always would, and then their assessment overrides the initial ratings. But it is important for test takers to know that this is a very small proportion of responses that have to be benchmarked. In the vast majority of cases, benchmarking is not required. Most of the time, rating is a very straightforward task, and the raters are very well trained before they start rating and they rate every single response according to the exact same scale.

CJ [00:05:46] Wow. OK, thank you. That was incredible and super insightful. So after all those years as a rater, what would you say are some of the most common mistakes that test takers make? You probably have incredible insight on that.

Meaghan [00:06:00] One would be not developing the points in the question with specific details. For all Writing responses, it’s really important to have some identifiable main ideas, and those should be supported with details. So for your response to be thorough and precise, you need to build on the information in the question with your own ideas. And there are people at all levels who sort of didn’t do that as much as they could. Another would be being rude or being really sarcastic in their responses, and I would say that this mainly applies to higher-level speakers or maybe native speakers. In some cases, maybe they’re upset that they have to take this test at all. Maybe they feel kind of insulted by the language level of the questions, but it’s just a fact that the IRCC requires every person who wants PR or citizenship to take a language test. So if your first language is English, that may be frustrating, but the best thing to do with your frustration is to go in knowing the criteria and just write something that’s fluent and detailed and precise and on topic so you can get credit for your abilities and score well. Just take the test seriously. Rudeness or sarcasm is really not going to help your score. It can only hurt you. So it’s not, it’s not recommended. Another would be not following the instructions: things like addressing all of the tasks, or for Task 2, choosing an option. Some people would write way over the word count limit, things like that. And again, for people at all language levels, the best thing they can do is read the question carefully—the whole question, including the background information and the instructions—and they should do exactly what the instructions tell them to do. You should not write as much as you possibly can in the time that you have. All responses that are too far under or over the word count limit will get a penalty, even if they’re well-written. You need to keep your tone polite. Stay on topic. These are not language skills that can only be accomplished if you’re a high-level test taker. These are things that everyone can do, and should do to help themselves get their best score. One more would be using templated content: things like memorized introductions or memorized conclusions. On the CELPIP Test, you have to make sure that everything you write is very specific to the questions that you are answering. Raters rate thousands of responses. They are very, very familiar with the kind of templates that sometimes language learners use, where like you memorize a full sentence of content that is so general that it could be used for any response at all. They’re most common in introductions and conclusions: things like “Both options have advantages and disadvantages,” or things like “For all of these reasons, I prefer Option B.” You don’t want to do that. Don’t bring a memorized template into the test. It’s very obvious, and you can’t get credit for language that you use that isn’t your own work. So there’s, there’s no advantage. Always use your own words. My last point would be formatting issues and kind of like mechanics issues, things like strange formatting, it looks weird on the screen: not using paragraphs, using the wrong punctuation marks, actually pressing the wrong keys on the keyboard because maybe they are not familiar with the standard keyboard. Something you can do to help you before the test is to just get familiar with the keyboard. Learn where the period is, where the comma is, how you’re going to do capital letters. Get comfortable with that. You can do that using one of our practice tests. You can certainly look up standard keyboards online, so just go in knowing kind of what your keyboard will look like so that that doesn’t happen to you.

Chris [00:09:49] Thanks, Meaghan. That was a lot of great advice.

CJ [00:09:53] Oh my gosh. It certainly was. And for any of our listeners out there who might have been frantically trying to write all this down or thinking they’re going to go back and re-listen to it again, you can do that. But we’re also going to include a summary of these points in the show notes, so that no one feels like they’ve missed any of it because it was really great. And I did just want to pick up on that first point, Meaghan, about developing ideas with specific details. I think test takers sometimes just can’t think of anything to write. We’ve, we’ve seen that before, and we’ve heard about it. So where can test takers get ideas and, you know, might they be able to include details that are not provided in the questions?

Meaghan [00:10:36] Absolutely. So, Writing questions are going to put you in kind of a role play situation and you should go into the test knowing that what the question says is going on may not reflect what’s actually currently true about your life. For example, a Writing question might start with something like “You work in a small office.” Maybe you in fact work in a large office, or you work in a restaurant, or you stay home with your kids. But that’s OK because you can still come up with ideas about working in an office because that’s a really common situation. Everybody knows what happens in an office. The questions are going to describe very general, open-ended situations that you can imagine and come up with ideas about. The ideas can come from your own life experience, if you have worked in an office in the past, even if you don’t right now. Your knowledge: maybe you know people who work in offices, you’ve seen TV shows about offices. They can come from your imagination. You can make up the name of the company. You can invent the kind of thing that they do. You can use any or all of these: your experience, your knowledge, your imagination. You absolutely can, and you should, develop the question. The question will be presented in a very general kind of way, and what you want to do is build on that with specific details. Use the name of things, like the city or the neighborhood or the restaurant or the person or the office that you’re talking about. Again, these can be real or imaginary. One thing that I would recommend to everybody is brainstorm before you start to write. Give yourself a couple of minutes to just think about what you’re going to say. What specific details are you going to include? Give yourself that time and get your details organized before you start to write, because that is just breaking up the task. And then when it comes time to write, you can really focus on your, your words and your language structure.

Chris [00:12:35] Oh, interesting. I can see that could really help with cases of writer’s block on the test. I want to switch gears here a bit and ask you about native speakers who take the test, you know, because sometimes we get native speakers who don’t do as well as they think they should on the test. What advice would you have for them to maximize their score?

Meaghan [00:12:57] One piece of advice would be—like I said earlier, if you’re frustrated about taking an English test, that is understandable, but stay away from expressing that in your responses. Recognize that these questions are written in fairly simple terms, and that is for accessibility reasons. It’s not to be condescending to you. We don’t want any test taker to lose the opportunity to complete a Writing task because they don’t understand the question. If you, as a native speaker, find the details quite simple and general, then you absolutely can and should develop those ideas with realistic and precise and more complex details. That’s something you can and should do. Again, avoiding comedy and sarcasm, things like where you’re just trying to be clever: being clever and being funny aren’t aspects of the rating scale, so they aren’t things that raters have the ability to give you any credit for. Your sense of humor may not be shared. It may not even be understood by the raters who are reading your response. Your tone could be seen as dismissive or your content could be seen as off topic. So that’s just, it’s, it’s a risk that you probably don’t want to take on a test that is a high stakes situation. Follow the instructions carefully. Take time to read them and just complete all the tasks thoroughly. Stay within the word count range. Don’t lose part of your score because you didn’t follow the instructions. It’s just unnecessary and very sad when that happens. And the last point would be: know going in that every response is rated according to the same rating scale. Nobody automatically gets a 12 in any part of the test. Some people—I’ve seen, I, sometimes you don’t know if it’s a native speaker or not, but I’ve certainly seen responses that seem to be kind of mediocre or maybe half-baked, maybe not the person’s best work. And I think that the idea there is, “Well, I’m a native speaker. Whatever I write will be good enough.” And then they’re surprised when they don’t get the score that they want. So you need to demonstrate the full range of your abilities. You have two pieces of writing, and that’s all the raters are going to see of you. They don’t know your background and your—they don’t know anything you don’t give them, right, so give them everything you have: complex grammar, precise vocabulary, good, clear, well-developed ideas. I’ve also seen responses where just out of absolute nowhere, the writer will mention they’re a native speaker or they’ll, like, list their academic credentials or things like that. Don’t do that. It’s, it’s not relevant. Just answer the questions and let your abilities speak for themselves.

CJ [00:15:53] Okay. Right. That actually that makes a lot of sense. And I think that leads to the question of like, what are the factors that go into getting a Level 12 or Level 9 or whatever the test taker needs, right? Everyone has their own needs. So where can test takers find out more about those factors? What’s the best way to understand that better?

Meaghan [00:16:17] Yeah. So I mentioned the, the four dimensions earlier, and each of those dimensions has a few individual factors in it. And without getting into those kind of details here, I would say, first of all, check our Writing Performance Standards, which are available in the Scoring section of our website. We also have a tool on the website called the Score Comparison Chart, and that is a series of responses, two different Writing responses, at every level from M all the way up to 12, and they come with detailed analysis of how the writer demonstrated their abilities in their responses. That is a great place to go to just see, progress through those responses and watch the complexity, watch the vocabulary, watch the language construction improve from level to level. That’s a really good thing to do. And lastly, I will plug our Writing Pro webinars—I give those, so those run every month, Writing Pro Target 9+; we have one also called Writing Pro: Target 5 that runs a little bit less often, but there are recordings of each one available on YouTube if you’re unable to attend, and we get into great detail about the Performance Standards and we look at sample responses and we talk about how the person has demonstrated their skills or could improve, in cases where they could.

Chris [00:17:47] And we’ll provide links to all of those resources in the show notes. I’d like to go back to the beginning of our conversation when you were talking about the rating process. What would you say are some of the misconceptions that test takers have about the rating process?

Meaghan [00:18:02] There are a lot, and I see them on message boards and things like that, and I wish I could just, like, talk to these people. One that I would really object to first is people really think that raters have access to the test takers’ scores, like their overall scores on other parts of the test, or we can see or listen to their other Speaking responses or their other Writing response. Raters have access to none of that. Another would be believing that raters are kind of, their job is to count mistakes; the idea that, OK, you can make three grammar errors, or you can make two word choice errors, or like that there’s a number of errors that you’re allowed to make before your score gets pushed down or something. At the same time, I sometimes have test takers in webinars commenting, “Oh, that person used—whatever, a very long word, so their score should be higher,” and that also doesn’t happen. They’re not raising your score because you use one uncommon word or something like that. It’s, it’s a much more holistic thing. Raters are looking at the response as a whole. They are considering how well it communicates the message. It’s not about counting errors—or impressing us, on the other end. And the last one I’ll mention is that people think that raters are encouraged to give low scores. That is absolutely not true. That would be very unethical, first of all. We have to submit all kinds of documentation to IRCC about our scoring process, how our scores work, all kinds of things, and they just wouldn’t work with us if we were artificially lowering scores. We’d all lose our jobs. So there is no benefit to us; at a purely selfish level, it just wouldn’t be smart. Another thing is “low score” has a different meaning to different people, and raters have no idea what any test taker is hoping to get on their test, and that’s exactly how it should be. That’s a key element of staying fair and staying impartial. Also, multiple raters, as I explained, contribute to scores and then they’re electronically calculated. So it is not at all the case that, like, one single rater is giving somebody any score. And in my years as a rater, I worked with two different scoring managers. Both were very, very smart, ethical people, and certainly they never encouraged us to rate harshly. They also never encouraged us to rate leniently. All that they ever encouraged was consistent and accurate rating.

Chris [00:20:51] OK, thank you. That I think is going to help to clear up a lot of those misconceptions. Wow. Just to pick up on your first point, could you clarify exactly to what raters do have access to during the writing process?

Meaghan [00:21:06] Yeah, they basically, they see the question that you answered, both sides of the screen, so the background information and also any instructions or tasks, and they see whatever you typed into the text box for that particular question. They have no access to any other part of your test. That includes your scores, your personal information, any of your answers to any questions on the test. They don’t even see your response for the other Writing task, the one that they didn’t receive.

Chris [00:21:37] OK, thanks for clearing that up.

CJ [00:21:39] So I actually wonder if you could clear something else up as well, because we sometimes hear test takers complain that their scores aren’t even across the four skills. And you know, we hear you. We, we see those comments. And they may they say that they may get higher Reading and Listening scores and lower Speaking and Writing scores. And I’m wondering what you would say to those who expect to get similar scores for all four parts of the test?

Meaghan [00:22:08] Yeah, I would say this is less a rating question and more just a general language learning question. Most people are not equally good at all four skills; whether they’re a learner or a native speaker, that really doesn’t matter. Everybody has strengths and weaknesses, regardless of their language level. And the productive skills— and that’s where you have to actually produce content, so that would be writing and speaking—those tend to be more challenging, especially for language learners. For most people, it’s more challenging to start with a blank page or an empty recording and write 200 words or speak for a minute than to, you know, answer multiple choice questions about a reading or listening passage. And I will say writing in particular poses challenges for a lot of people, and that includes native speakers. There are plenty of native speakers who have degrees, high-level positions, very high-functioning people whose writing skills are fairly weak, or maybe not as developed as some of their other skills. That’s very common.

CJ [00:23:15] OK. Thank you so much. That makes sense. As you were speaking, I was like, I can, I feel I’m hearing some of that I can definitely relate to. So thank you for sharing that.

Chris [00:23:25] Yeah. Thanks so much, Meaghan. I think our listeners gained a lot of insight into the scoring process of the CELPIP Writing Test today.

Meaghan [00:23:32] Yeah, it was great to be able to focus on the topic of rating because we do get a lot of questions about it in our webinars and there’s, there’s not always time to get into all the details. And I think it’s just really important that test takers have the fullest possible picture. Otherwise they’re going to get a very distorted picture of what we actually do to get a fair assessment of their English level.

CJ [00:23:56] Yeah, absolutely. Thanks so much, Meaghan. This has been truly illuminating. I think you’ve brought a lot of much-needed clarity to the rating process, so thank you so much again. So, Chris, what have we got planned for the next episode?

Chris [00:24:13] OK, next time we’re completely switching topics. We’re going to take a look at some fun ways to build English language proficiency by talking about some of our staff’s favorite books. If you’re someone who enjoys reading and would like to get some recommendations of some great books to help you develop your English in an enjoyable way, this episode is for you.

CJ [00:24:34] OK, well, that sounds like it’s going to be super useful, especially for all the bookworms out there. Until then, we hope your preparation is going well and that you’ll join us next time.

Chris [00:24:47] Bye.

CJ [00:24:48] Bye, everyone.