Student AI chat logs: what their questions reveal about their thinking
A student's questions to AI show two things about their thinking: whether it started with them, and who made the final call. Stating a position and asking AI to test it builds understanding. Pasting the assignment and asking for the paper hands the thinking over, and in experiments that kind of use went with less understanding.¹
Cognity, an AI visibility platform for classrooms, records how students work with AI inside an assignment. This post follows Are AI detectors accurate? What universities decided and Take-home finals when students use AI: five formats compared: once a score cannot tell you who used AI, the conversation is where you read what they understood.
What do students' questions to AI show about their thinking?
They show where the thinking started and who made the decisions. How much a student used AI tells you less than what they asked it to do.
Researchers in Germany taught university students a programming topic in the lab, some with an AI model and some without. Access made no difference to overall learning. What mattered was use: students who had the model generate solutions covered more topics but understood each one less, and students who asked it for explanations understood more.¹
Writing shows a similar split. In a study of 70 US college students writing an argumentative essay, researchers recorded every query to ChatGPT, "queries that are typically hidden from instructors", and sorted them into planning, drafting, reviewing, and asking for the whole essay. Students who had ChatGPT write the entire essay felt less ownership of it than those who worked through the writing themselves.²
Both are early studies: the first is a preprint on programming, the second an online study with a single essay task. They do not tell you a student's grade from their prompts. They do suggest the questions are worth reading: in both, how students used AI went with what they understood or how much of the work felt like theirs.
What does a good question to AI look like?
A good question starts from the student's own attempt, idea or confusion, and asks AI to explain, test or check it. The student keeps the decision. The prompts below are illustrative, written for this post across several course types; real ones are usually less tidy.
Signals of a good question
| Signal | Illustrative prompt | What it shows |
|---|---|---|
| 1. States their own attempt first | (Intro economics) "I think a price ceiling of $4 creates a shortage of 200 units, because demand is 900 and supply is 700 at that price. Is 'shortage' the right term here, and is my reasoning right before I check the numbers?" | The thinking starts with the student. AI is asked to review it, not produce it. |
| 2. Asks why, not just what | (Statistics) "I keep reading that p = 0.03 is not the probability the null hypothesis is true. Can you explain why with a coin-flip example, not my lab data?" | The student knows what they do not understand, and asks for an explanation they can then apply themselves. |
| 3. Checks a claim AI made | (History) "You said the 1918 flu killed more US soldiers than combat did. Which sources say that, and do they count deaths in training camps?" | The student does not take AI's evidence on trust. This is the habit a research paper depends on. |
| 4. Asks for the strongest objection | (Public policy) "My thesis is that congestion pricing cut traffic without hurting small businesses. What is the best evidence against the second half, and what would I need to answer it?" | The student is testing the argument, and plans to answer the objection in their own paper. |
| 5. Rejects or adapts an answer, with a reason | (Psychology) "Sample size isn't our limitation, we had 240 participants. The real issue is that they were all psych undergraduates. Help me think through what that does to generalisability. I'll write the paragraph." | The final call is the student's, and the reason is on the record. |
A conversation does not need all five. Two or three in a row, around the same part of the paper, usually mean the student was thinking with the tool rather than through it.
In Cognity, these signals are what the AI fluency card, COGNITY PAIR, reads in each student's conversation. It rates four dimensions: Partition (which parts of the work, and which data, the student handed to the AI), Articulate (how the requests were specified and steered), Inspect (what the student accepted from the AI, and on what basis) and Responsibility (what the student told others about the AI's part). The five signals fall under the first three: stating your own attempt first is Partition, asking why is Articulate, and checking a claim or asking for the strongest objection is Inspect.
What does handing the work over look like?
The assignment goes in, finished text comes out, and no decision of the student's appears in between. The same course types, again illustrative:
Signals of handing the work over
| Signal | Illustrative prompt | What it shows |
|---|---|---|
| 1. Pastes the assignment and asks for the product | "Write a 1,500-word argumentative essay answering the prompt below. APA citations. [assignment instructions pasted]" | The student's own position never enters the conversation. This is the "All" category in the essay study.² |
| 2. Asks for a finished section | "Write my discussion section in an academic tone based on these results. [results table pasted]" | The student asks for the writing, not for help deciding what the results mean. |
| 3. Asks AI to supply the evidence | "Give me three peer-reviewed studies proving congestion pricing helps small businesses, with citations." | The conclusion came first and the sources are wanted to fit it, unchecked. |
| 4. Hands over the decision | "Which of these two research questions is easier to get a good grade on?" | The choice the assignment exists to make has been made by AI. |
| 5. Polishing with no question | "Make it flow better." "Shorter." "Again, more natural." | Nothing is being asked. The student's own revision does not appear. |
Do not grade a single line. In Anthropic's analysis of 574,740 student conversations with its own model, about 47% sought answers or content "with minimal engagement", yet the report notes that many of these "serve legitimate learning purposes".³ A student may ask for a sample introduction and then write their own; read what happened next, not one prompt in isolation.
Where should you start reading in a large course?
Do not read each conversation from the top. Three places show most of what a conversation can tell you:
- The first message. Did the student open with their own idea, attempt or confusion, or with the assignment text?
- The turn after AI pushes back or gets something wrong. The student's next message shows understanding most clearly. Narrowing a claim, checking a source or disagreeing means they followed; repeating the request or writing "just fix it" marks where they got stuck.
- Where AI's wording meets the paper. Which suggestions reached the final text, which were dropped, and whether the student said why.
In Cognity, for example, the Conversation tab shows each student's exchange with Jello, the AI tutor, under the stage of the assignment it happened in: My idea, Write it, Make it stronger and Hand it in. That makes the first message and the turns around a revision quick to find.
How do you turn this into one rubric criterion?
Add one criterion about how the student worked with AI, next to your existing ones, and publish it with the assignment. Pitt's teaching centre makes the case for grading the log itself: "evaluating the quality of their prompts assesses conceptual depth directly".⁴
Rubric criterion: Working with AI — questions and judgement
| Level | Description |
|---|---|
| 4 — Strong | Starts from their own position, draft or attempt. Asks AI to explain, test or check it. Accepts, changes or rejects suggestions, and the reasons show in the conversation or the paper. The key sentences are in the student's own words. |
| 3 — Competent | Asks about concepts, evidence or objections at least some of the time. The main decisions are the student's. Checks AI's claims in one or two places. |
| 2 — Developing | Requests are mostly for summaries, rewording or polish. Some of AI's claims reach the paper unchecked. |
| 1 — Beginning | Hands the task over and takes back finished text. The student's own judgement does not appear in the conversation or the paper. |
Grade it with the paper open, since level 4 depends on where the questions led. Weight it alongside argument, evidence and writing according to what the course is for. Leave it out of any assignment where AI is not allowed.
For the assignment instructions
AI is allowed on this assignment, and your conversation with it is part of what is graded. Questions that start from your own idea and test it score higher than requests for finished text. Asking AI for an answer is fine; what counts is what you do with it next.
In Cognity you do not have to write this criterion from scratch. Paste the instruction above into the assignment description, then choose Suggest with AI in the rubric builder. It drafts the criteria from what the description says you value. In our test it wrote the conversation into the content criterion at every level; the top level reads "The Jello conversation starts from the student's own ideas and uses probing questions to test the position, check evidence, and explore objections." Edit the wording, then grade with the conversation open next to the paper.
For grading a process record more broadly, see process portfolios: assessing the work, not the text.
What do you do with what you read?
Turn it into one line of feedback per student, a note of what the whole class asked about, and a change to the next assignment.
- One line per student. "Your question about generalisability is why your limitations paragraph works." Or: "You asked about p-values twice, but the paper still treats 0.03 as a 3% chance the null is true. Look at the week 6 reading again." A comment tied to a question tells the student what to ask next time.
- What the class asked. If a third of the conversations circle the same concept, the gap is in the course, not in the students. Open the next class with it.
- The next assignment. If most conversations hand the work over, the task may have asked for something AI can produce whole, such as a summary or a definition. Anchor the next one in material only your course has, as the take-home finals comparison sets out.
All of this depends on having the whole conversation. A chat log the student exports and attaches is the one they chose to share, and a conversation in another tab does not have to appear in it. In a large course, collecting and opening those files is work of its own before the reading starts.
How can you read the whole conversation, not only the part a student exports?
Keep the AI conversation inside the assignment, so it is recorded as the student works. In Cognity, students do the assignment with Jello, which follows the AI rules you set for that assignment. When you open a submission, the review screen has three tabs: Conversation, Activity and Rubric.
The Activity tab's authorship card, "Where the text came from", splits the final paper into Written by Student, AI-assisted and Copy-pasted. It is measured in words, and 8 or more words in a row that match Jello count as AI-derived. Read next to the conversation, it shows which questions led to which sentences, and where text arrived with no conversation behind it.
Cognity does not give an AI score and does not detect AI. It records the conversation and the writing in one place, and the judgement on both stays with you.
If students use their own AI tools outside the assignment, Cognity sees none of it; the signals above still work on any log they hand in. To see how it runs across a course, including AI settings per assignment and summaries first for large classes, see Cognity for universities.
Cognity is free to start. Try it on your next assignment.
FAQ
Should I require students to submit their AI chat logs?
If you will read them, yes, and say so in the syllabus and the assignment. Pitt's teaching centre suggests asking for the chat history along with a short reflection on how the student's "own thinking shaped or challenged the AI's output".⁴ Remember that an exported log is the part the student chose to share.
Is asking AI for the answer always a bad sign?
No. Asking what a term means or how a method works is part of studying, and Anthropic's report notes that many answer-seeking conversations "serve legitimate learning purposes".³ What matters is whether the student's own judgement follows the answer.
Does a prompt that hands the work over prove misconduct?
No. It is a reason to look at the paper and the rest of the conversation, and if AI was allowed, it is something to grade and comment on. If AI was not allowed, follow your institution's procedure; a single prompt is not proof, any more than a detector score is, as Are AI detectors accurate? sets out.
Can I use this rubric criterion when AI is not allowed?
No. It grades how a student worked with AI, so it belongs only on assignments where AI use is allowed and expected. For a final where some parts must be done without AI, see Take-home finals when students use AI.
References
- Lehmann, M., Cornelius, P. B. & Sting, F. J., "AI Meets the Classroom: When Do Large Language Models Harm Learning?", arXiv 2409.09047, revised 8 March 2025. Preprint, not peer reviewed; laboratory experiments with university students in Germany and a field study in two graduate programming courses in the Netherlands.
- Jelson, A., Manesh, D., Jang, A., Dunlap, D., Kim, Y.-H. & Lee, S. W., "An Empirical Study to Understand How Students Use ChatGPT for Writing Essays", Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, April 2026. Online study, 70 US college students, one argumentative essay.
- Anthropic, "Anthropic Education Report: How university students use Claude", 8 April 2025. Conversations on Claude.ai Free and Pro accounts linked to higher-education email addresses.
- University of Pittsburgh Teaching Center, "AI's effect on finals and end-of-term assessments", updated 15 September 2026, checked 19 October 2026.
