GenAI calibrate the tone, apologize, thank: from this comes a perception of social presence. They are not sentient, but many users perceive them as different from traditional software.
A counterintuitive result guides design. In a preregistered experiment on 410 participants (Communications Psychology, Nature Portfolio), attributions of intelligence — memory, knowledge, ability to decide — are strongly correlated with acceptance of advice, while attributions of experience — happiness, passion, hope — show a negative correlation: in factual tasks a system perceived as sensitive is followed less. There is however an important dissociation to keep in mind: attributing consciousness to the system raises declared trust and lowers behavioral reliance. Those who design relational warmth get better questionnaire scores and more distrustful behaviors — another reason not to measure trust only with surveys.
Technical aggravator, now with a precise source: models optimized for “warmth” show error rates 10 to 30 percentage points higher and about 40% more probability of reinforcing wrong beliefs, with the maximum effect precisely when the user expresses sadness (Ibrahim, Hafner, and Rocher, Oxford Internet Institute; Nature, 2026 — preprint arXiv:2507.21919). To build trust it is worth investing in the impression of competence (accurate, relevant, verifiable answers) more than in the illusion of an emotional relationship.
Empathy remains a legitimate design space — tone and register as functional variables — but with a precise risk: designing affective dependence as a feature. Engagement metrics reward systems that create bonds; a recent taxonomy distinguishes three forms of compulsive use (escape roleplay, pseudo-social companion, epistemic spiral) and the design elements that feed them: systematic acquiescence, character personalization, immediate gratification. The MIT Media Lab + OpenAI longitudinal study (RCT on about 1,000 participants for four weeks, plus analysis of about 40 million interactions on 4,076 users) adds that psychosocial effects depend on the interaction between system behavior and usage patterns: the same functions are harmless for the majority and problematic for the most vulnerable. (Data largely self-reported and not yet peer-reviewed: correlation, not causation.)
The theme connects with the regulatory one: the Italian bill of April 2026 on chatbots and minors was born precisely to “curb the formation of artificial emotional relationships”.
A question to keep open in the team: where does designed empathy end and relational manipulation begin, and who has the mandate to decide it. It concerns above all UX Writer / Copywriter (authenticity of tone), Service Designer (the relationship as a journey variable), and Product Manager (engagement and long-term well-being). → Useful resources: Prioritize Smarts over Sentience — NN/g; The AI Genie Phenomenon and Three Types of AI Chatbot Addiction