还有 11 页未读 ,您可以 继续阅读 或 下载文档
报告派所有资源均由用户上传分享,仅供网友学习交流,未经上传用户书面授权,请勿作商用。
1、本文档共计 11 页,下载后文档不带水印,支持完整阅读内容或进行编辑。
2、当您下载文档后,并不意味着拥有了版权,文档仅供网友学习交流,不得用于其他商业用途。
3、本站所有内容均由合作方或网友上传,本站不对文档的完整性、权威性及其观点立场正确性做任何保证或承诺!
4、本本所有内容不构成任何投资建议,不具有任何指导和买卖意见。
5、如文档内容存在违规,或者侵犯商业秘密、侵犯著作权等,请点击“违规举报”。
analysis and insights for the audience.The overallsidered as the first benchmark for data analysisprocess is shown in Figure 1,which a data an-in NLP community.alyst is often asked to fulfill during work.Thisjob involves a relatively routine scope,which may2 Related Workbecome repetitive at times.It also requires sev-2.1 Related Tasks and Datasetseral technical skills,including but not limited toSQL,Python,data visualization,and data analysis,Since our task setting is new in NLP community.making it relatively expensive.As this job scopethere is no existing dataset that is fully suitable formay adhere to a relatively fixed pipeline,there is aour task.We explore the most relevant tasks andheated public debate about the possibility of an AIdatasets.First,the NvBench dataset (Luo et al..replacing a data analyst,which attracts significant2021)translates natural language (NL)queres toattention.corresponding visualizations (VIS),which coversIn this paper,we aim to answer the followingthe first half of the main job scope of a data an-research question:Is GPT-4 a good data analyst?alyst.This dataset has 153 databases along withTo answer this question,we conduct preliminary780 tables in total and covers 105 domains,andstudies on GPT-4 to demonstrate its potential capa-this task (NL2VIS)has attracted significant atten-bilities as a data analyst.We quantitatively evaluatetion in both commercial visualization vendors andthe pros and cons of LLM as a data analyst mainlyacademic researchers.Another popular subtask offrom the following metrics:performance,time,andthe NL2VIS task is called text-to-sql,which con-cost.In specific,we treat GPT-4 as a data analyst toverts natural language questions into SQL queriesconduct several end-to-end data analysis problems.(Zhong et al.,2017;Guo et al.,2019;Qiet al.,2022;Since there is no existing dataset for such data anal-Gao et al.,2022).Spider (Yu et al.,2018),SParCysis problems,we choose one of the most related(Yu et al..2019b)and CoSOL (Yu et al..2019a)aredataset NvBench,and add the data analysis partthree main benchmark datasets for text-to-sql tasks.on top.We design several automatic and humanSince this work is more focused on imitating theevaluation metrics to comprehensively evaluate theoverall process of the job scope of a data analyst,quality of the data extracted,charts plotted and datawe adopt the NL2VIS task which has one moreanalysis generated.Experimental results show thatstep forward than the text-to-sql task.GPT-4 can beat an entry level data analyst in termsFor the second part of data analysis,we also ex-of performance and have comparable performanceplore relevant tasks and datasets.Automatic chartto a senior level data analyst.In terms of cost andsummarization (Mittal et al.,1998:Ferres et al..time,GPT-4 is much cheaper and faster than hiring2013)is a task that aimed to explain a chart anda data analyst.summarize the key takeaways in the fomm of natu-However,since it is a preliminary study onral language.Indeed,generating natural languagewhether GPT-4 is a good data analyst,we pro-summaries from charts can be very helpful to infervide fruitful discussions on whether the conclu-key insights that would otherwise require a lot ofsions from our experiments are reliable in real lifecognitive and perceptual efforts.In terms of thebusiness from several perspectives,such as whetherdataset,the chart-to-text dataset (Kantharaj et al.,the questions are practical,whether the human data2022)aims to generate a short description of theanalysts we choose are representative,etc.To sum-given chart.This dataset also covers a wide rangemarize,our contributions include:of topics and chart types.Another relevant NLPtask is called data-to-text generation(Gardent et al.,We for the first time raise the research question2017:Dusek et al..2020:Koncel-Kedziorski et al..of whether GPT-4 is a good data analyst,and2019;Cheng et al.,2020).However,the output ofquantitatively evaluate the pros and cons.all these existing works are descriptions or sum-For such a typical data analyst job scope,wemaries in the form of one or a few sentences orpropose an end-to-end automatic frameworka short paragraph.In the practical setting of datato conduct data collection,visualization,andanalytics work,one should highlight the analysisanalysis.and insights in bullet points to make it clearer to theWe conduct systematic and professional humanaudience.Therefore,in this work,we aim to gen-evaluation on GPT-4's outputs.The data analy-erate the data analysis in the form of bullet pointssis and insights with good quality can be con-instead of a short paragraph.