Crafting AI-Complementary Skills and Bulletproof Assessments

Universities face an unprecedented challenge. The knowledge and skills we’ve traditionally taught can now be replicated by AI within seconds. Surprisingly, there's little public discourse on how to adapt or innovate. While colleagues may be experimenting privately, we really need to share and learn publicly. In this spirit, allow me to share my approach, in strengthening AI-complementary skills and building robust assessments.

The Perils of Business as Usual

Before we dive into solutions, let’s consider the consequences of inaction. If universities merely ask students to promise no plagiarism, and otherwise continue with business as usual, we create four significant risks:

1. Falling behind: Some students may disregard AI altogether, leaving them ill-equipped for a labour market that increasingly rewards technological mastery. A new survey of 100,00 workers in Denmark finds that, even within occupations, women are less likely to use ChatGPT. This puts them at a disadvantage.

   

   Anders Humlum and Emilie Vestergaard 2024

2. Skills development: Many students will use AI tools (since they provide decent answers, yet cannot be checked or punished), but not learn how to use these technologies critically. Moreover, if our assessments stay the same, we are not encouraging students to creatively harness the technological frontier.

3. Repressive culture: My globally comparative research, as well as a broader empirical literature, indicates the crucial importance of a culture where everyone is encouraged to openly question, inspire, critique, and explore. As Timur Kuran articulates in his seminal work “Private Truths and Public Lies”, a culture of repression comes with serious, often hidden costs. By prohibiting the use of AI, universities risk recreating climates of fear and secrecy. Students may not share new strategies or insights gained from AI tools, for fear of getting in trouble. This hurts learning, especially for marginalised groups.

   

   

   After a century of totalitarian dictatorships, all Uzbek men wear all black, reflecting intense fear of speaking out. In Namangan, it was sub-zero, many homes lacked gas, yet people were often too scared to say they were cold.

4. Credibility: Firms heavily rely on university credentials as signals of talent. Universities must maintain public trust, by demonstrating our capacity to identify and certify talent. That’s the point of a degree. We need to show that our assessments cannot be outsmarted by ChatGPT.

These risks underscore the urgent need for adaptation and leadership. But how can we rise to this challenge? I suggest we embrace technological progress while innovating to build AI-complementary skills and ensuring robust assessments.

Harnessing AI: A Path Forward

Technology is a wonderful thing - for both global and individual prosperity.

As a lecturer in International Development, the overarching lesson from our field is that technological innovation is absolutely fundamental to:

• Agricultural productivity (high-yielding seed varieties)

• Economic upgrading (hello Samsung!)

• Sustained economic growth

• Environmental sustainability (Electric Vehicles and Solar Panels)

• Global health (e.g., the latest malaria vaccines).

Universities should be at the technological frontier: encouraging the pursuit of knowledge, freely sharing new insights, and driving progress.

Electric buses in China. (Credit: Getty Images)

So, instead of asking, “How can we prevent students from using AI?”, we could be thinking more constructively:

“How can we enable our students to harness advances in technology to become more creative thinkers and deep learners?”

This shift in perspective aligns perfectly with the realities of the modern job market. Firms are massively investing in AI. Now, they want talented workers who understand the strengths and limitations of cutting-edge technologies. As educators, our role is to help students build AI-complementary skills. By reframing our approach to education, we can prepare students for the technological frontier.

My strategy is thus two-fold: teaching technological literacy, so that students use AI carefully and productively, while simultaneously changing my assessments so that they go beyond AI’s current capabilities, and instead assess students’ ingenuity.

My Two-Fold Strategy

1. Cultivating AI Literacy

Design classroom activities which integrate AI tools like Claude, e.g.:

• In pairs, ask Claude whether land reform was important for East Asia's industrialisation. Identify the limitations of this response and explain how it could be improved.

• Copy and paste Oliver Kim and Jen Kuan Wang’s Table 5 into Claude. Ask Claude to explain the regression table. Ask more questions until you understand.

• How did South Korea strengthen its domestic innovation? Draft an answer, then give it to Claude and ask for critique. Then improve on your answer.

Kim and Wang 2024

Exploring Ideas & Getting Feedback with AI

In another class, I presented Sam Van Noort’s research industrialisation and democracy. He finds that the share of manufacturing employment is positively associated with democratisation.

Then I asked my students to form pairs and draft an essay plan:

“How might Communism affect the relationship between industrialisation and democratisation?”

Together, we discussed their answers.

Then I showed how they can ask these same questions to Claude, request feedback and continually brainstorm.

By engaging with Claude mid-lecture, evaluating its answers, and adding prompts, I demonstrated how to harness AI effectively.

3. Design Assessments That Go Beyond Current AI Capabilities

• Assume AI use: I take it for granted that some students will use AI in assessments, which is practically impossible to prevent, check, or punish consistently.

• Preventing plagiarism through innovative assessments: My questions require deep understanding of course material, careful analysis, and creative thinking.

• AI-proof questions: I’ve tested all my assessment questions against Claude, and it cannot produce answers worthy of a first-class grade.

• 3 hour or 48 hour online assessments recreate the conditions of the modern workplace, giving students to concrete task within a set deadline.

Sample Assessment Questions

These questions enable students to demonstrate their deep learning and creative thinking.

1. India’s Hindus have increasingly united to champion Hindu Nationalism. Indonesia’s Muslims, meanwhile, have pushed for jurisprudential Islam. How do these case studies challenge “The Narrow Corridor”?

2. Acemoglu and colleagues (2021) find that democracy increases growth, on average. Does it follow that poor countries should democratise? Explain why democracy may impede growth. Provide academic references.

3. Sub-Saharan African countries typically have = low agricultural productivity. Design a competitive bid for £50 billion – such as to research new technologies, provide infrastructure, tackle binding constraints in state capacity or rule of law, or trial new government policies. What would you do? And how would you assess its efficacy? Justify your answer and provide economic references.

4. What is the primary reason why Latin American countries did not emulate the East Asian model. Justify your answer and provide academic references.

5. How does evidence from historians like Cook and J.R. Neil challenge the conventional wisdom from political science about the roots of authoritarianism in the Middle East and North Africa?

6. Since studying this course, what beliefs about international development have you revised? Explain what evidence and justification led you to change your priors.

7. Identify the primary driver of low fertility in South Korea. Design an intervention that would tackle binding constraints. Provide justification and academic references.

A picture of Claude crumbling at my questions!

Why These Questions Outsmart AI

1. AI-complementary skills: My assessments are designed to evaluate and encourage the development of skills that complement AI capabilities, such as critical thinking and creative problem-solving. Even if students copy and paste texts into Claude, it still struggles to think laterally. It will not necessarily recognise that,

   1. In democracies with low GDP per capita, a labour aristocracy may mobilise for protections that hurt international competitiveness and wider job growth.

   2. Environmental histories of MENA’s weak agriculture suggest states could not have been very strong (thereby challenging claims of religious authoritarianism, made by Faisal Ahmed, Ahmet Kuru and Jared Rubin).

   3. Hindu Nationalism is a good example of what Acemoglu and Robinson call a ‘strong society’. Hindus are increasingly united and cohesive, yet contrary to A&R’s expectations this may actually restrict the liberty of Muslims. In Indonesia, many Muslim organisations have increasingly organised for religious laws and institutions that actually restrict individual freedoms, especially for women (e.g. hijab mandates).

2. Course knowledge. Claude is good at listing all the possible factors, but cannot necessarily assess their relative importance, or binding constraints, especially as it does not necessarily have the latest evidence. My assessments will reward students who understand and critically evaluate the latest empirical literature, e.g. on South Korea’s ultra low fertility, or Sub-Saharan Africa’s agriculture and institutions.

3. Creative problem-solving: let students showcase their unique brilliance!

4. Justification and referencing: The requirement for academic references and justification ensures that students engage deeply with the literature and form well-reasoned arguments. Claude often makes up references, so this is a good signal of plagiarism.

Conclusion

I want to teach my students how to become brilliant - to develop AI-complementary skills and harness the latest innovations so they can be at the technological frontier and drive progress. That’s the human edge.

Personally, I also need to change. I need to get smarter and outwit AI, so that my assessments enable students to demonstrate their creativity and ingenuity. This way, I hope to embody the spirit of my field, driving progress through constant innovation!

P.s. this upcoming week, I have two awesome guests on my podcast: Oliver Kim (who has a brilliant new paper on Taiwan’s land reform) and my great friend Daron Acemoglu (back for the fourth time).

Algo Al Art Generated

NightCafe

Financial Times 2023

Further Resources

• Stanford’s advice on how to address and integrate AI (a good example of a university taking AI seriously)

• “Teaching effectively with ChatGPT” (if you are looking for an introduction to the basics)