Write a lengthy, comprehensive and detailed research proposal. Detail study designs with…
- User
Write a lengthy, comprehensive and detailed research proposal. Detail study designs with tasks that involves humans proposing implementations of concrete energy reduction plans, the accuracy of which might depend on accuracy of knowledge about the energy-use of specific appliances. Focus in particular on designs and tasks which present the participant with some high-level energy savings goal (e.g., saving X amount of money, or reducing usage by Y amount of kwh), and then challenge the participant to propose more concrete action plans for how one might actually implement such a reduction goal. Include discussion of how a task might be set up to reinforce participants to generate feasible, or realistic plans. Also comment on how findings from prior research (e.g., Attari 2010; Marghetis et al. 2019) might be incorporated to make the task more realistic. Then, discuss different ways that a large language model might be incorporated into the task, such as by critiquing participant plans, proposing alternate plans, or by proposing initial plans etc. etc. Detail several different ways that such tasks might be set up (by detail, I mean describe in a way similar to what might be found in a methods or procedure section of a cognitive psychology journal article). Theoretical justifications should build from one or more of the following literatures: 1) bounded rationality; 2) mental models of AI, and LLM mental models or theory of mind ; 3) Trust and reliance in automation. 4) Perceptions of uncertainty in LLM output.
Some of your research proposals should involve combining the past methods and results from the work of research like Attari, Kantenbacher, Marghetis and Landy - into novel study ideas that involve using LLM's to assist humans on energy decision making. For instance, having LLM's that incorporate the expert heuristics outlined in Kantenbacher & Attari 2021. Or by using LLM's to try to infer which novice heursitics participants are using, and then attempting to give helpful feedback based on that info.
Further develop, and find extensive literature support for discussing these Theoretical Foundations 1) Bounded Rationality in Energy Decision-Making; 2) Mental Models of AI and LLM Capabilities; 3) Trust and Reliance in Automation; 4) Perceptions of Uncertainty in LLM Output. The proposals should also connect with the cognitive science literature on goal setting or planning.
For the proposals, only assume that researchers have access to general purpose LLMs. Also assume that experiments must be conducted online. Provide a general literature review for the core topics, and then separate, focused lit review and theoretical rationale for each study proposal. Each proposal should have hypotheses that clearly follow from the literature review.
Propose research-backed methods of measuring constructs such as plan accuracy, plan feasibility, appliance knowledge, knowledge change, LLM reliance/interaction and trust. Detail how these measures will be used to infer aspects of the participants mental models. Also suggest alternative ways of gauging users ability to plan.
Provide a full literature that address the following pertinent questions:
Cognitive Psychology Questions:
LLM Critique vs. Information: Does receiving feasibility critiques and accuracy checks from an LLM lead participants to generate energy reduction plans that are more realistic (aligned with household constraints) and accurate (closer to achievable savings) compared to having no assistance or only access to information upon request? Goal Format Influence: How does the cognitive representation of the energy savings goal – as absolute energy units (kWh), monetary value (), or a relative percentage (%) – influence the strategies participants use to construct their plans and the effectiveness of LLM assistance? Does one format facilitate better integration of LLM feedback? Knowledge Correction: Can interacting with an LLM that provides accurate energy information and critiques based on common misperceptions lead to measurable improvements in participants' domain knowledge or a reduction in their endorsement of common energy myths (e.g., overestimating savings from unplugging chargers)? Trust in Critics: How do users perceive and interact with the LLM when its primary role shifts from a passive information source to an active critic? Does proactive critique foster more trust and reliance, or does it potentially create reactance or reduce perceived autonomy? Heuristic Detection: Can an LLM be effectively trained or prompted to analyze participant responses (e.g., energy estimates, justifications for choices in a plan) and infer, with reasonable accuracy, the probability that the participant is relying on specific novice energy judgment heuristics identified by Kantenbacher & Attari (2021)? Targeted Feedback Effectiveness: Does providing participants with targeted feedback, specifically designed to address the heuristic inferred by the LLM, lead to greater improvements in energy judgment accuracy or planning quality compared to providing generic feedback based on expert heuristics or no feedback at all? Heuristic Shift: Over repeated trials, does receiving heuristic-specific feedback lead to a measurable reduction in participants' reliance on inaccurate novice heuristics and an increased adoption of more accurate expert heuristics? LLM Role Comparison: How do distinct LLM assistance paradigms – acting as an initial plan generator, a post-hoc critic, or an interactive collaborator – differentially affect the user's planning strategies, the allocation of cognitive effort, and the objective quality (accuracy, feasibility, potentially diversity) of the final energy reduction plan? Anchoring Effects: Does providing users with an LLM-generated initial plan (which they then edit) lead to anchoring effects, where users insufficiently adjust from the initial suggestion? Does this approach reduce user effort and cognitive load but potentially hinder deeper engagement and learning compared to approaches where the user takes the primary role in generation? User Perceptions of Roles: Which LLM interaction role (generator, critic, collaborator) do users perceive as most helpful, trustworthy, usable, and satisfying? How do these subjective perceptions align with objective measures of plan quality and evidence of user learning (e.g., knowledge gain)? Goal Format & LLM Data: Can access to an LLM providing detailed, granular energy data for specific appliances and concrete actions mitigate potential difficulties associated with less intuitive goal formats (e.g., helping users translate a goal into specific kWh-saving actions)? Does LLM access level the playing field across formats? Task Detail Demand: How does the cognitive demand of specifying concrete implementation details (e.g., "Set winter thermostat to 68°F during the day and 65°F at night" rather than just "Reduce heating") impact planning performance (accuracy, completeness) across different goal formats and LLM availability conditions?
Further develop, and find extensive literature support for discussing these Theoretical Foundations:
Bounded Rationality in Energy Decision-Making
Energy planning exemplifies bounded rationality-the concept that human decision-making is constrained by limited information, cognitive capacity, and time. This manifests in energy contexts through several documented phenomena:
Kantenbacher and Attari (2021) identified fundamental differences between expert and novice approaches to energy judgments. While novices rely on perceptual cues (e.g., "larger devices use more energy"), experts employ functional heuristics (e.g., "devices that heat or cool use more energy than devices with a primary function involving motion"). These cognitive patterns help explain the "energy efficiency gap"-the disparity between economically optimal energy decisions and actual consumer behavior.
Mental Models of AI and LLM Capabilities
A user's mental model of LLM capabilities-their conceptualization of what the system can and cannot do-significantly influences interaction quality and outcomes. Recent research reveals several patterns relevant to energy planning tasks:
- Capability overestimation: Users often overestimate LLM accuracy and expertise, especially in domains where they themselves lack knowledge.
- Differentiated perception: Mental models of AI abilities are increasingly differentiated from self-perceived abilities through interaction, though misconceptions persist.
- Pattern recognition vs. understanding: While users may anthropomorphize LLMs, attributing human-like reasoning, these systems operate through pattern recognition rather than true understanding or domain expertise.
This research suggests that effective LLM assistance in energy planning depends not only on accurate information provision but also on managing user expectations and promoting appropriate reliance.
Trust and Reliance in Automation
Trust in AI systems is multidimensional, extending beyond accuracy to include perceived intentions, fairness, and transparency. In energy planning contexts, several trust dimensions are particularly relevant:
- Competence perception: Users assess an LLM's ability to provide accurate energy information-a judgment that may be challenging given widespread misconceptions about energy usage.
- Intention assessment: Beyond accuracy, users evaluate whether the LLM prioritizes their interests (e.g., providing genuine energy-saving advice vs. promoting specific products).
- Calibrated trust: Both overtrust (excessive reliance) and undertrust (insufficient reliance) can lead to suboptimal outcomes in energy planning.
Trust calibration-aligning trust levels with actual system capabilities-is crucial for effective human-AI collaboration. Design elements that promote critical thinking, such as requiring justifications for decisions, can reduce overtrust in AI systems.
Perceptions of Uncertainty in LLM Output
How LLMs communicate uncertainty and how users perceive this uncertainty significantly impact decision quality[^3]. Recent research by Steyvers et al. (2025) identified a "calibration gap" between LLM confidence levels and user perception of LLM accuracy, with several implications for energy planning:
- Uncertainty language: The specific language used to express uncertainty significantly influences user confidence, regardless of actual accuracy[^3].
- Explanation effects: Longer explanations increase user confidence even when they don't improve accuracy, potentially leading to overconfidence in LLM energy suggestions[^3].
- Discrimination difficulty: Users struggle to distinguish between correct and incorrect LLM outputs without explicit uncertainty signals[^3].
Literature to incorporate when forming hypothesis justifications and theoretical connections:
Attari, S. Z., DeKay, M. L., Davidson, C. I., & Bruine De Bruin, W. (2010). Public perceptions of energy consumption and savings. Proceedings of the National Academy of Sciences, 107(37), 16054–16059. Borzino, N., Hiepler, B., Schmitt, K., Schmitz, J., Schubert, R., & Tiefenbeck, V. (2025). Switching Off: Energy Saving Goals Outshine Incentives—Evidence from a Field Experiment. Environmental and Resource Economics. Kantenbacher, J., & Attari, S. Z. (2021). Better rules for judging joules: Exploring how experts make decisions about household energy use. Energy Research & Social Science, 73, 101911. Li, Z., Tong, S., Liu, Y., Peng, K., & Wang, C. (2025). Potential of large language model-powered nudges for promoting daily water and energy conservation (No. arXiv:2503.11531). arXiv. https://doi.org/10.48550/arXiv.2503.11531 Marghetis, T., Attari, S. Z., & Landy, D. (2019). Simple interventions can correct misperceptions of home energy use. Nature Energy, 4(10), 874–881. Schille-Hudson, E. B., Margehtis, T., Miniard, D., Landy, D., & Attari, S. Z. (2019). Big, hot, or bright? Integrating cues to perceive home energy use. Proceedings of the 41st Cognitive Science Society. Attari, S. Z. (2021). Transforming energy use. Current Opinion in Behavioral Sciences, 42, 104–108. https://doi.org/10.1016/j.cobeha.2021.04.008 Barnby, J. M., Dayan, P., & Bell, V. (2023). Formalising social representation to explain psychiatric symptoms. Trends in Cognitive Sciences, 27(3), 317–332. Bittner, E. A. C., Oeste-Reiß, S., Kirmse, R., Poser, M., & Wiethof, C. (2024). Cognitive Load Theory Approach to Hybrid Intelligence: Tackling the Dual Aim of Task Performance and Learning. Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21. https://doi.org/10.1145/3449287 Callaway, F., Hardy, M., & Gri, T. L. (2023). Optimal nudging for cognitively bounded agents: A framework for modeling, predicting, and controlling the effects of choice architectures. Psychological Review, 130(6), 1457–1491.45
Assistant
Okay, here's a detailed research proposal addressing your requirements, incorporating the literature you provided and expanding on key theoretical frameworks.
Research Proposal: LLM-Assisted Energy Reduction Planning Under Bounded Rationality
1. Introduction and Literature Review
This research investigates how Large Language Models (LLMs) can assist individuals in creating and implementing effective energy reduction plans within their households. Addressing climate change requires widespread adoption of energy-efficient behaviors, yet consumers often struggle to make optimal decisions due to limited information, cognitive constraints, and systematic biases. This project aims to develop and test LLM-based interventions that improve energy planning by:
- Reducing the Impact of Bounded Rationality: LLMs can provide readily accessible information, perform complex calculations, and offer personalized recommendations to overcome cognitive limitations.
- Improving Mental Models of Energy Use: By providing accurate feedback and correcting common misperceptions, LLMs can foster more accurate mental models of energy consumption and savings.
- Calibrating Trust and Reliance: This research will explore how to design LLM interfaces that promote appropriate trust in AI assistance, avoiding both over-reliance and under-utilization.
- Addressing Uncertainty: The project will also investigate methods for communicating LLM uncertainty to users, ensuring informed decision-making even when the AI's confidence is limited.
Core Literature Review:
- Energy Efficiency Gap: The "energy efficiency gap" refers to the discrepancy between economically rational energy-saving investments and actual consumer behavior. Factors contributing to this gap include:
- Bounded Rationality: Consumers often lack the time, information, and cognitive capacity to make fully informed decisions (Simon, 1957; Kantenbacher & Attari, 2021).
- Behavioral Biases: Cognitive biases such as present bias (prioritizing immediate costs over long-term savings), loss aversion, and the status quo bias can impede the adoption of energy-efficient measures (Tversky & Kahneman, 1979).
- Information Asymmetries: Consumers may lack accurate information about appliance energy consumption, the costs and benefits of energy-saving actions, and the availability of rebates or incentives (Attari et al., 2010).
- Mental Models of Energy Use: People often rely on inaccurate mental models when making energy decisions. Research by Attari et al. (2010) demonstrates that the public often misperceives the relative energy consumption of various appliances and actions. For example, people tend to overestimate the savings from phantom loads (unplugging chargers) and underestimate the impact of behaviors like adjusting thermostat settings (Marghetis et al., 2019). Further research from Schille-Hudson et al. (2019) demonstrates the various heuristic cues that novices tend to use when determining appliance energy use.
- LLMs and Decision Support: LLMs possess the potential to provide personalized decision support in various domains, including energy consumption. By integrating domain-specific knowledge and reasoning capabilities, LLMs can offer tailored recommendations, answer user queries, and generate customized energy reduction plans (Li et al., 2025).
- Trust in AI: User trust is a crucial factor in the adoption and effective use of AI systems. Trust is influenced by factors such as perceived accuracy, reliability, transparency, and fairness (Lee & Moray, 1994). However, users can also exhibit overtrust or undertrust in AI systems, leading to suboptimal outcomes (Buçinca, Malaya, & Gajos, 2021).
- Uncertainty Communication: LLMs can express uncertainty in their outputs through various means, such as hedging language, confidence scores, or alternative suggestions (Steyvers et al., 2025). However, research suggests that users may struggle to interpret and appropriately respond to LLM uncertainty, potentially leading to miscalibrated trust and suboptimal decisions (Aridhi et al., 2021).
2. Research Questions and Hypotheses
This research seeks to answer the following overarching questions:
- How do different LLM assistance paradigms (critic, generator, collaborator) affect the quality (accuracy, feasibility) and diversity of user-generated energy reduction plans?
- Can targeted feedback from an LLM, tailored to address specific novice energy judgment heuristics, improve energy planning outcomes more effectively than generic feedback or no feedback?
- How does the format of the energy savings goal (kWh, $, %) influence the strategies users employ and their ability to effectively integrate LLM assistance?
- What are the key factors influencing user trust and reliance on LLMs in energy planning, and how can these factors be leveraged to promote appropriate trust calibration?
Theoretical Foundations:
1. Bounded Rationality in Energy Decision-Making
Energy planning decisions are inherently complex and often made under conditions of uncertainty and limited cognitive resources. As Kantenbacher and Attari (2021) point out, novices tend to rely on easily accessible but often inaccurate perceptual cues (e.g., size heuristic: "larger devices use more energy"), while experts employ more accurate functional heuristics (e.g., "devices that heat or cool use more energy"). The complexity involved with making energy decisions is compounded by the fact that the average user often does not have enough energy-related financial literacy to make good decisions (Agarwal et al., 2015). Given that users are boundedly rational, they often have difficulty calculating how much money a particular appliance will cost them, and are thus likely to choose the appliance with the low initial price, even if it has higher energy costs in the long term (Frederick et al., 2002).
- Literature Support: Simon (1957); Kantenbacher & Attari (2021); Tversky & Kahneman (1979); Attari et al. (2010); Frederick et al., 2002
2. Mental Models of AI and LLM Capabilities
A user's mental model of an LLM shapes their expectations and interactions with the system. Recent research indicates that people often overestimate the capabilities of AI, particularly in domains where they lack expertise (e.g., energy consumption). It is thus important that people gain a realistic understanding of LLM capabilities. That way, they can appropriately calibrate their trust. Incomplete or inaccurate mental models can lead to misuse or disuse of the technology (Norman, 2014). Furthermore, understanding the underlying technology is insufficient to develop good mental models. People need to understand how the AI behaves and what its capabilities are (Goodman et al., 2020).
- Literature Support: Norman (2014); Goodman et al., 2020;
3. Trust and Reliance in Automation
Trust in AI is a multifaceted construct encompassing competence, integrity, and benevolence (Mayer et al., 1995). In the context of energy planning, users need to trust the LLM's ability to provide accurate information and prioritize their energy-saving goals. However, trust must be calibrated to match the actual capabilities of the system. Overtust can lead to uncritical acceptance of inaccurate advice, while undertrust can result in missed opportunities for energy savings (Lee & See, 2004). Therefore, it is important to achieve trust calibration, where users' level of trust is aligned with the AI's actual capabilities (Lee & Moray 1994).
- Literature Support: Mayer et al. (1995); Lee & See (2004); Lee & Moray (1994); Buçinca, Malaya, and Gajos (2021)
4. Perceptions of Uncertainty in LLM Output
LLMs can express uncertainty through various linguistic cues (e.g., hedging, disclaimers) or numerical confidence scores. However, users often struggle to interpret and appropriately respond to this uncertainty information. Furthermore, LLM-generated responses often appear so confident, that users are misled (Yoffe et al., 2025). Recent research suggests that longer explanations from LLMs increase user confidence, even if they don't improve accuracy, which may lead to overconfidence in LLM energy suggestions.
- Literature Support: Steyvers et al. (2025); Yoffe et al., 2025;
Hypotheses:
- H1 (LLM Critique): Participants receiving feasibility critiques and accuracy checks from an LLM will generate more realistic and accurate energy reduction plans compared to those receiving no assistance or only information upon request.
- H2 (Heuristic Detection & Feedback): Participants receiving targeted feedback addressing inferred novice heuristics will show greater improvements in energy judgment accuracy and planning quality than those receiving generic feedback or no feedback.
- H3 (Goal Format): Participants presented with energy savings goals in monetary value ($) will generate more feasible plans compared to those presented with goals in absolute energy units (kWh) or relative percentages (%), particularly when LLM support is limited. This is because monetary values directly relate to user experience.
- H4 (Trust Calibration): Increased transparency in LLM outputs, including explanations of reasoning and explicit indications of uncertainty, will lead to better calibrated trust, resulting in more appropriate reliance on LLM assistance.
3. Study Designs and Methods
The following are several proposed study designs to investigate the research questions. All studies will be conducted online using a between-subjects design unless otherwise indicated.
Study 1: LLM Role and Planning Quality
- Objective: To examine the impact of different LLM assistance paradigms on the quality and diversity of user-generated energy reduction plans.
- Participants: A sample of 200 adults recruited through an online platform (e.g., MTurk, Prolific).
- Procedure:
- Pre-test: Participants will complete a questionnaire assessing their baseline energy knowledge, attitudes, and behaviors.
- Task: Participants will be presented with a scenario involving a hypothetical household and a specific energy savings goal (e.g., "Reduce your energy bill by 20%").
- Conditions: Participants will be randomly assigned to one of four conditions:
- Control: Participants generate an energy reduction plan without LLM assistance.
- LLM Generator: Participants are provided with an initial energy reduction plan generated by the LLM and are instructed to edit and improve it.
- LLM Critic: Participants generate an initial energy reduction plan, which is then critiqued by the LLM in terms of feasibility, accuracy, and potential drawbacks. Participants then revise their plan based on the LLM's feedback.
- LLM Collaborator: Participants interact with the LLM in a collaborative manner, iteratively proposing actions and receiving feedback until they develop a final energy reduction plan.
- LLM Implementation: A general purpose LLM will be prompted with instructions specific to the condition. The LLM will be primed with information about common household appliances and their energy consumption.
- Post-test: Participants will complete a questionnaire assessing their perceived workload, trust in the LLM, and satisfaction with the planning process.
- Measures:
- Plan Accuracy: The estimated energy savings of the proposed plan, calculated based on standard energy consumption values for appliances and behaviors.
- Plan Feasibility: A subjective rating of the plan's realism and practicality, assessed by independent raters. Raters will be asked to consider factors such as the participant's lifestyle, household constraints, and the likelihood of successful implementation.
- Plan Diversity: The range of energy-saving actions included in the plan, measured by the number of distinct categories of actions (e.g., appliance upgrades, behavioral changes, smart home technologies).
- LLM Reliance: The extent to which participants incorporated LLM-generated suggestions into their final plan, measured by the proportion of actions adopted from the LLM.
- Subjective Measures: Workload will be assessed using the NASA Task Load Index (TLX). Trust in the LLM will be measured using the Trust in Automation Scale (Jian, 2000), adapted for the context of LLM assistance. Satisfaction will be measured using a Likert scale.
- Hypotheses:
- H1: Participants in the LLM Critic and LLM Collaborator conditions will generate more accurate and feasible energy reduction plans compared to those in the Control and LLM Generator conditions.
- H2: Participants in the LLM Collaborator condition will exhibit higher plan diversity compared to those in the other conditions.
Study 2: Targeted Feedback Based on Novice Heuristics
- Objective: To investigate whether targeted feedback from an LLM, tailored to address specific novice energy judgment heuristics, can improve energy planning outcomes.
- Participants: A sample of 150 adults recruited through an online platform.
- Procedure:
- Pre-test: Participants will complete a questionnaire assessing their baseline energy knowledge and their reliance on common novice heuristics identified by Kantenbacher & Attari (2021). This will be measured using a series of questions designed to assess participants' beliefs about energy consumption, such as "Which uses more energy: a plasma TV or an LCD TV?" and "Do you agree with the statement: 'Larger appliances always use more energy than smaller appliances'?"
- Task: Participants will be presented with a series of energy planning scenarios and asked to propose actions to achieve specific savings goals.
- LLM Heuristic Detection: The LLM will be trained to analyze participant responses and infer the probability that the participant is relying on specific novice heuristics. This will be achieved through a combination of few-shot learning and fine-tuning on a dataset of labeled responses.
- Conditions: Participants will be randomly assigned to one of three conditions:
- Control: Participants receive no feedback on their plans.
- Generic Feedback: Participants receive generic feedback based on expert heuristics (e.g., "Consider focusing on actions that reduce heating and cooling energy consumption").
- Targeted Feedback: Participants receive targeted feedback specifically designed to address the heuristic inferred by the LLM (e.g., "It seems you are relying on the size heuristic. Remember that appliances that generate heat, like water heaters, can use a lot of energy even if they are small").
- Post-test: Participants will complete a questionnaire assessing their energy knowledge, their reliance on novice heuristics, and their perceived usefulness of the feedback.
- Measures:
- Energy Judgment Accuracy: The accuracy of participants' estimates of energy consumption for various appliances and actions.
- Heuristic Reliance: The extent to which participants endorse common novice heuristics, measured using the same questionnaire as the pre-test.
- Planning Quality: Assessed by independent raters based on the accuracy, feasibility, and completeness of the proposed plans.
- Subjective Measures: Perceived usefulness of the feedback, measured using a Likert scale.
- Hypotheses:
- H1: Participants receiving targeted feedback will show greater improvements in energy judgment accuracy and planning quality compared to those receiving generic feedback or no feedback.
- H2: Participants receiving targeted feedback will exhibit a greater reduction in their reliance on inaccurate novice heuristics compared to those in the other conditions.
Study 3: Goal Format and LLM Assistance
- Objective: To examine how the format of the energy savings goal (kWh, $, %) influences the strategies users employ and their ability to effectively integrate LLM assistance.
- Participants: A sample of 180 adults recruited through an online platform.
- Procedure:
- Pre-test: Participants will complete a questionnaire assessing their baseline energy knowledge and numeracy skills.
- Task: Participants will be presented with a scenario involving a hypothetical household and a specific energy savings goal, framed in one of three formats:
- kWh: "Reduce your energy consumption by 500 kWh per month."
- *50 per month."
- %: "Reduce your energy consumption by 20% per month."
- Conditions: Participants will be randomly assigned to one of three conditions:
- No LLM: Participants generate an energy reduction plan without LLM assistance.
- LLM Information: Participants have access to an LLM that provides detailed energy information for specific appliances and actions upon request.
- Post-test: Participants will complete a questionnaire assessing their perceived difficulty of the task and their satisfaction with the planning process.
- Measures:
- Plan Accuracy: The estimated energy savings of the proposed plan, calculated based on standard energy consumption values.
- Plan Feasibility: A subjective rating of the plan's realism and practicality, assessed by independent raters.
- Strategy Use: The types of energy-saving actions included in the plan (e.g., appliance upgrades, behavioral changes), categorized based on their cost and complexity.
- LLM Interaction: The number and types of queries submitted to the LLM, analyzed to understand how participants used the information provided.
- Subjective Measures: Perceived difficulty will be measured using a Likert scale. Satisfaction will be measured using a Likert scale.
- Hypotheses:
- H1: Participants presented with energy savings goals in monetary value ($) will generate more feasible plans compared to those presented with goals in absolute energy units (kWh) or relative percentages (%), particularly in the No LLM condition.
- H2: Access to the LLM will mitigate the difficulties associated with less intuitive goal formats (kWh, %), leading to more accurate and feasible plans across all goal format conditions.
- H3: Participants presented with goals in monetary value will exhibit more efficient use of the LLM, as they will need to convert their understanding of money into which appliances to cut back on.
Study 4: Trust Calibration and Uncertainty Communication
- Objective: To investigate how LLM uncertainty communication affects user trust and reliance in energy planning.
- Participants: A sample of 150 adults recruited through an online platform.
- Procedure:
- Pre-test: Participants will complete a questionnaire assessing their baseline trust in AI and their tolerance for uncertainty.
- Task: Participants will be presented with a series of energy planning scenarios and asked to solicit advice from an LLM.
- Conditions: Participants will be randomly assigned to one of three conditions:
- No Uncertainty: The LLM provides recommendations without explicitly communicating its uncertainty.
- Confidence Score: The LLM provides recommendations with a numerical confidence score indicating its certainty.
- Verbal Uncertainty: The LLM provides recommendations with verbal expressions of uncertainty (e.g., "I am fairly confident that...", "I am unsure, but I suggest...").
- LLM Implementation: The LLM will be trained to generate recommendations with varying levels of uncertainty, based on its internal confidence levels.
- Post-test: Participants will complete a questionnaire assessing their trust in the LLM, their perceived accuracy of the recommendations, and their willingness to follow the LLM's advice.
- Measures:
- Trust Calibration: The alignment between participants' trust in the LLM and the actual accuracy of its recommendations. This will be measured by correlating participants' trust ratings with the LLM's error rate for the specific scenarios they encountered.
- LLM Reliance: The extent to which participants followed the LLM's advice, measured by the proportion of recommendations adopted.
- Subjective Measures: Trust will be measured using the Trust in Automation Scale. Perceived accuracy will be measured using a Likert scale.
- Hypotheses:
- H1: Participants in the Confidence Score and Verbal Uncertainty conditions will exhibit better calibrated trust compared to those in the No Uncertainty condition.
- H2: Participants in the Confidence Score and Verbal Uncertainty conditions will demonstrate more appropriate reliance on the LLM, exhibiting greater caution when the LLM expresses high uncertainty.
4. Measurement Plan
Plan Accuracy: Calculated using standard energy consumption values for appliances and behaviors. These estimates will be cross-referenced with data from reputable sources like the U.S. Energy Information Administration (EIA) and ENERGY STAR. Plan accuracy can be measured using mean absolute percentage error (MAPE) (Hyndman & Koehler, 2006)
Plan Feasibility: Evaluated by independent raters with experience in energy efficiency or home energy auditing. Raters will assess feasibility based on a rubric that considers factors such as:
- Cost of implementation
- Lifestyle compatibility
- Household constraints
- Availability of incentives or rebates
- Likelihood of successful adoption
Appliance Knowledge: Assessed using a multiple-choice questionnaire covering basic concepts of energy consumption, appliance energy use, and energy-saving strategies.
Knowledge Change: Measured by comparing pre-test and post-test scores on the appliance knowledge questionnaire.
LLM Reliance/Interaction: Quantified by tracking:
- Number of queries submitted to the LLM
- Types of queries (e.g., information requests, advice seeking)
- Proportion of LLM suggestions incorporated into the final plan
- Time spent interacting with the LLM
Trust: Measured using validated scales such as the Trust in Automation Scale (Jian, 2000) and the Trust Scale for the AI Context (TAI) (Schaefer, 2021). These scales assess various dimensions of trust, including perceived competence, reliability, and transparency.
Mental Model Inference: Participants mental models will be inferred from the following measures:
- Appliance Knowledge Questionnaire: The results of the questionnaire will allow the researchers to determine if the participants use the correct facts when considering the appliance plans
- Think Aloud Protocols: By having participants think aloud as they respond to different stimuli, researchers can get a sense of how they are reasoning about the plans.
- Post Questionairres: These questionairres would help researchers get a more qualitative understanding of how the users reason about the AI plan.
- Error Analysis: By looking at the types of plans participants create (both with and without the help of AI) the researchers can see what types of actions users assume are more impactful.
5. Expected Outcomes and Significance
This research is expected to yield several significant outcomes:
- Improved Understanding of Human-AI Collaboration in Energy Planning: The studies will provide valuable insights into how different LLM assistance paradigms and uncertainty communication strategies affect user decision-making, trust, and learning.
- Design Guidelines for Effective LLM-Based Interventions: The findings will inform the development of practical guidelines for designing LLM interfaces that promote accurate energy planning, appropriate trust calibration, and sustainable energy behaviors.
- Enhanced Energy Literacy and Behavior Change: By providing personalized information and feedback, LLM-based interventions have the potential to improve energy literacy and motivate individuals to adopt energy-saving practices.
- Contribution to the Broader Field of Human-AI Interaction: The research will contribute to the growing body of knowledge on human-AI interaction, with implications for the design of AI systems in other domains involving complex decision-making and risk management.
- New Methods of Measuring Human Plans: The rubrics and questionnaires will contribute new ways of understanding human plans, particularly for boundedly rational users.
6. Timeline
- Month 1-3: Literature review, study design refinement, development of LLM prototypes, pilot testing.
- Month 4-9: Data collection for Studies 1-4.
- Month 10-12: Data analysis, interpretation of results, report writing, dissemination of findings.
7. Budget
- Participant compensation: $5,000
- LLM API usage: $1,000
- Software and data analysis tools: $500
- Total: $6,500
8. Literature Cited
(See the literature incorporated into the above sections)
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.