Research Methodology

The OnTilt Self-Check is built on peer-reviewed research spanning behavioral addiction, operant conditioning, HCI design patterns, and workplace wellbeing. This page explains how we designed the tool, what it measures, and where we are in the validation process.

Research Foundation

Our starting observation: the interaction loop of agentic coding tools (Claude Code, Cursor, Copilot) may exhibit structural similarities with slot machines — low cost per try, variable quality outcomes, instant retry, and near-miss effects that keep you going. We call this the "slot machine hypothesis."

This is not a metaphor pulled from thin air. It rests on a research base of 32 peer-reviewed sources from gambling studies, social media addiction, digital product design, and clinical psychology. This is a research hypothesis, not an established clinical finding. The mechanisms are drawn from peer-reviewed addiction research in other domains (gambling, social media) and applied by analogy to AI coding tools. Direct empirical validation for this specific population is pending. From these sources, we proposed six mechanisms as a framework for understanding patterns when developers work intensively with AI tools:

1. Variable ratio reinforcement — AI delivers valuable outputs at unpredictable intervals, the same schedule that makes slot machines compelling [3]
2. Near-miss effect — code that "almost works" triggers continuation more powerfully than outright failure [4]
3. Sunk cost escalation — invested time and attention make it harder to disengage [5]
4. Unintended immersion ("dark flow") — AI coding creates a rapid challenge-feedback loop that may produce a pattern resembling what gambling researchers call "dark flow" [6]
5. Checking habit / intermittent reward — short, repeatable interactions reinforced by fast informational rewards [7]
6. Loss aversion — absence of the tool feels like a loss, not a return to baseline [5]

Our dimensional structure draws on Griffiths' component model of behavioral addiction [8], the ICD-11 gaming disorder criteria [9], and DSM-5 gambling disorder criteria [10]. We also incorporated recent scales for problematic ChatGPT use [11] and conversational AI dependence [12].

The Six Dimensions

Each dimension maps to one or more mechanisms from addiction research that may parallel patterns in AI tool use. Together they provide a multi-faceted profile rather than a single score.

Quiz Design

Design Principles

• Self-reflection, not diagnosis. The quiz is a screening tool. It does not diagnose any condition. There is no recognized clinical category for "AI tool addiction" in DSM-5 or ICD-11.
• Behavioral anchoring. Each item describes a concrete, observable behavior (not an emotion or trait), with a built-in dysregulation marker such as "despite planning to stop" or "despite knowing it causes harm."
• 30-day time window. Responses are anchored to the last 30 days to reduce recall bias and capture current patterns.
• Red flag system. Two items — continuing despite known harm (csq-3) and repeated failure to keep self-imposed limits (ctrl-3) — trigger specific feedback regardless of total score. These correspond to core addiction criteria in ICD-11 and DSM-5.

Question Distribution

Loss of Control and Negative Consequences each have 3 questions (21.4% weight each). The remaining four dimensions have 2 questions each (14.3%). This weighting is intentional: impaired control is the central construct of behavioral addiction, and consequences provide the hardest evidence (observable, external).

Validation Status

We believe in transparency about where this instrument stands.

Current: v2 (research preview)

• Item wording informed by peer-reviewed instruments (Problematic ChatGPT Use Scale, CAI dependence scales) and clinical frameworks (ICD-11, DSM-5)
• Improved psychometric properties over v1: clearer item separation, behavioral anchoring, disambiguated flow vs. dysregulation
• Preliminary scoring thresholds (25/50/75%) — these are rational cut-offs, not empirical norms

Planned validation roadmap

1. Cognitive interviews (5–10 participants) — do respondents interpret items as intended?
2. Exploratory & Confirmatory Factor Analysis (EFA/CFA) — does the 6-factor structure hold empirically?
3. Item Response Theory calibration (IRT / GRM) — which items discriminate best across the severity continuum?
4. Measurement invariance — does the instrument work equivalently across PL/EN and different usage modes (IDE copilot vs. chat vs. agent)?
5. Empirical norms — replace arbitrary percentage thresholds with percentile-based cut-offs from real cohort data

Known Limitations

• Self-report bias — respondents most affected may minimize their answers
• No clinical validation yet — test-retest reliability, construct validity, and factor analysis are pending
• Context sensitivity — scores may fluctuate with project deadlines, tool availability, and mood
• Engagement vs. dysregulation — some captured behaviors (long focus sessions) can be healthy in the right context
• Population specificity — the quiz targets AI power users (developers, 10+ hours/week). Casual users or non-coders may find items less relevant

Our Framing: Hygiene, Not Abstinence

OnTilt is not an anti-AI project. We use AI tools extensively ourselves. The framing is deliberate:

• Hygiene, not abstinence. Like sleep hygiene or dental hygiene — it's about sustainable practices, not prohibition.
• Awareness, not pathologizing. We explicitly avoid labeling intensive use as "addiction." The outcome is a behavioral profile and red flags, never a clinical label [14].
• Dual nature acknowledged. AI coding tools are genuinely powerful. The same mechanisms that make them compelling also make them worth studying. Both things are true.
• Data-driven. We collect anonymous research data to eventually replace intuition with evidence.

Selected References

1. [3] Lindström et al. (2021). A computational reward learning account of social media engagement. PMC7910435
2. [4] Clark et al. (2009). Gambling near-misses enhance motivation to gamble and recruit win-related brain circuitry. PMC2658737
3. [5] Arkes & Blumer (1985). The psychology of sunk cost. Organizational Behavior and Human Decision Processes
4. [6] Abuhamdeh (2020). Investigating the "flow" experience: Key conceptual and operational issues. PMC7033418
5. [7] Oulasvirta et al. (2012). Habits make smartphone use more pervasive. Personal and Ubiquitous Computing
6. [8] Griffiths, M.D. (2005). A 'components' model of addiction within a biopsychosocial framework. Journal of Substance Use, 10(4), 191-197
7. [9] WHO. Gaming Disorder FAQ. WHO
8. [10] APA. What Is Gambling Disorder? (patient-facing resource). APA. For research, see: DSM-5-TR (2022).
9. [11] Zhao, Y. et al. (2024). Development and validation of the Problematic ChatGPT Use Scale. Current Psychology. Turkish CFA/IRT validation: IJMHA 2025
10. [12] CAI dependence scale — uncontrollability, withdrawal, mood modification, negative impacts. Frontiers in Psychology
11. [13] Compulsive ChatGPT use and associations with anxiety, burnout, and sleep. Acta Psychologica
12. [14] "People are not becoming 'AIholic'" (2025). Caution against premature "addiction" labels for AI use. Addictive Behaviors

Full bibliography (32 sources) is available on the Research page.