Empathy and Social Presence
GenAI calibrate the tone, apologize, thank: from this comes a perception of social presence. They are not sentient, but many users perceive them as different from traditional software.
A counterintuitive result guides design. In a preregistered experiment on 410 participants (Communications Psychology, Nature Portfolio), attributions of intelligence — memory, knowledge, ability to decide — are strongly correlated with acceptance of advice, while attributions of experience — happiness, passion, hope — show a negative correlation: in factual tasks a system perceived as sensitive is followed less. There is however an important dissociation to keep in mind: attributing consciousness to the system raises declared trust and lowers behavioral reliance. Those who design relational warmth get better questionnaire scores and more distrustful behaviors — another reason not to measure trust only with surveys.
Technical aggravator, now with a precise source: models optimized for “warmth” show error rates 10 to 30 percentage points higher and about 40% more probability of reinforcing wrong beliefs, with the maximum effect precisely when the user expresses sadness (Ibrahim, Hafner, and Rocher, Oxford Internet Institute; Nature, 2026 — preprint arXiv:2507.21919). To build trust it is worth investing in the impression of competence (accurate, relevant, verifiable answers) more than in the illusion of an emotional relationship.
Empathy remains a legitimate design space — tone and register as functional variables — but with a precise risk: designing affective dependence as a feature. Engagement metrics reward systems that create bonds; a recent taxonomy distinguishes three forms of compulsive use (escape roleplay, pseudo-social companion, epistemic spiral) and the design elements that feed them: systematic acquiescence, character personalization, immediate gratification. The MIT Media Lab + OpenAI longitudinal study (RCT on about 1,000 participants for four weeks, plus analysis of about 40 million interactions on 4,076 users) adds that psychosocial effects depend on the interaction between system behavior and usage patterns: the same functions are harmless for the majority and problematic for the most vulnerable. (Data largely self-reported and not yet peer-reviewed: correlation, not causation.)
The theme connects with the regulatory one: the Italian bill of April 2026 on chatbots and minors was born precisely to “curb the formation of artificial emotional relationships”.
The decision this observation informs
The decision is where designed empathy ends and relational manipulation begins, and who holds the mandate to decide it. Without an explicit choice, the engagement metric decides, and it rewards systems that create attachment: the question stays open precisely because it is unassigned.
Designing warmth costs accuracy in measured terms: 10 to 30 percentage points more error and about 40% more probability of reinforcing wrong beliefs, with the maximum effect when the user expresses sadness. And it costs the risk of designing affective dependence as a feature. Designing distance costs questionnaire scores, because attributing consciousness to the system raises declared trust, but the data says behavioural reliance moves the other way: attributions of intelligence correlate with acceptance of advice, attributions of experience correlate negatively. This is why declared trust and actual reliance cannot be read off the same instrument (see Trust, AI literacy and calibration).
The choice belongs to the UX Writer / Copywriter for the register and to the Product Manager for the mandate, and it has to be made now: the Italian bill of April 2026 on chatbots and minors was born to curb the formation of artificial emotional relationships, so the perimeter is closing from the outside. The consequence for the metric falls on Does engagement stay a KPI?.
Effects and role responsibilities
- UX Writer / Copywriter — Conversational register — apologising, thanking, warmth — is an editorial choice with measured consequences: models tuned for “warmth” show error rates 10 to 30 points higher and more often reinforce mistaken beliefs, with the largest effect when the user expresses vulnerability. Authentic tone builds trust better than performed warmth: it pays to invest in accurate, verifiable answers, not the illusion of an emotional relationship.
- Service Designer — The relationship with the system is a journey variable to design explicitly, not a byproduct of tone: on factual tasks, a system perceived as “sensitive” is followed less, so relational warmth needs to be calibrated by task type, not applied uniformly.
- Product Manager — Engagement metrics reward systems that build attachment, but the MIT Media Lab + OpenAI study links prolonged use to higher loneliness and emotional dependence for a share of users: measure long-term wellbeing, not just time spent, and get ahead of the regulatory perimeter closing in from outside (the Italian bill of April 2026 on chatbots and minors). → Useful resources: Prioritize Smarts over Sentience — NN/g; The AI Genie Phenomenon and Three Types of AI Chatbot Addiction