Experimental Analysis of Productive Interaction Strategy with ChatGPT: User Study on Function

Our study presents two project-level benchmark tasks that extend beyond function-level evaluations. It comes from a paper posted to arXiv on 9 September 2026. The application of Large Language Models (LLMs) is growing in the productive completion of Software Engineering tasks. Yet, studies investigating productive prompting techniques often employed a limited problem space, focusing primarily on well-known prompting patterns and targeting function-level SE practices. We identify significant gaps in real-world workflows that involve complexities beyond class-level (e.g., multi-class dependencies) and different features that can impact Human-LLM Interaction (HLI) processes in code generation. To address these issues, we designed an experiment to comprehensively analyze HLI features related to code generation productivity. We conducted a user study with 36 participants from diverse backgrounds, asking them to solve the assigned tasks by interacting with the GPT assistant using specific prompting patterns. We also examined participants' experiences and behavioral features during interactions by analyzing screen recordings and GPT chat logs. Our empirical investigation, based on statistical guidance, revealed (1) that three out of 15 HLI features emerged as consistently supported factors for productivity; (2) five primary guidelines for enhancing productivity for HLI processes; and (3) a taxonomy of 29 runtime and logic errors that can occur during HLI processes, along with suggested mitigation plans.

Why it counts

Our study presents two project-level benchmark tasks that extend beyond function-level evaluations.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.