{"id":331,"date":"2022-06-18T14:33:54","date_gmt":"2022-06-18T11:33:54","guid":{"rendered":"https:\/\/quecst.qcri.org\/blog\/?p=331"},"modified":"2022-09-03T12:57:30","modified_gmt":"2022-09-03T09:57:30","slug":"engineers-aware-commercial-tools-disagree-on-social-media-sentiment","status":"publish","type":"post","link":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/","title":{"rendered":"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment"},"content":{"rendered":"
\"\u00a0Engineers,
Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment<\/a><\/figcaption><\/figure>\n

For segmentation, one often need to use sentiment analysis services.<\/p>\n

Large commercial sentiment analysis tools are often deployed in software engineering due to their ease of use. However, it is not known how accurate these tools are, and whether the sentiment ratings given by one tool agree with those given by another tool.<\/p>\n

We use two datasets \u2013 (1) NEWS consisting of 5,880 news stories and 60K comments from four social media platforms: Twitter, Instagram, YouTube, and Facebook; and (2) IMDB consisting of 7,500 positive and 7,500 negative movie reviews \u2013 to investigate the agreement and bias of four widely used sentiment analysis (SA) tools: Microsoft Azure (MS), IBM Watson, Google Cloud, and Amazon Web Services (AWS).<\/p>\n

We find that the four tools assign the same sentiment on less than half (48.1%) of the analyzed content.<\/p>\n

We also find that AWS exhibits neutrality bias in both datasets;<\/p>\n

Google exhibits bi-polarity bias in the NEWS dataset but neutrality bias in the IMDB dataset, and IBM and MS exhibit no clear bias in the NEWS dataset but have bi-polarity bias in the IMDB dataset.<\/p>\n

Overall, IBM has the highest accuracy relative to the known ground truth in the IMDB dataset.<\/p>\n

Findings indicate that psycholinguistic features\u00a0 especially affect, tone, and use of adjectives \u2013 explain why the tools disagree. Engineers are urged caution when implementing SA tools for applications, as the tool selection affects the obtained sentiment labels.<\/p>\n

Jung, S.G., Salminen, J. and\u00a0Jansen, B. J.\u00a0(2022)\u00a0Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment. ACM SIGCHI Symposium on Engineering Interactive Computing Systems<\/a>\u00a0(EICS22), 21-24 June. Sophia Antipolis, France.\u00a0Article 153.<\/p>\n","protected":false},"excerpt":{"rendered":"

For segmentation, one often need to use sentiment analysis services. Large commercial sentiment analysis tools are often deployed in software engineering due to their ease of use. However, it is not known how accurate these tools are, and whether the sentiment ratings given by one tool agree with those given by another tool. We use… Continue reading Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[],"yoast_head":"\nEngineers, Aware! Commercial Tools Disagree on Social Media Sentiment - Team Acua<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment - Team Acua\" \/>\n<meta property=\"og:description\" content=\"For segmentation, one often need to use sentiment analysis services. Large commercial sentiment analysis tools are often deployed in software engineering due to their ease of use. However, it is not known how accurate these tools are, and whether the sentiment ratings given by one tool agree with those given by another tool. We use… Continue reading Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment\" \/>\n<meta property=\"og:url\" content=\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/\" \/>\n<meta property=\"og:site_name\" content=\"Team Acua\" \/>\n<meta property=\"article:published_time\" content=\"2022-06-18T11:33:54+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-09-03T09:57:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/quecst.qcri.org\/blog\/wp-content\/uploads\/2022\/06\/sentiment_cellphone-213x300.png\" \/>\n<meta name=\"author\" content=\"Jim Jansen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jim Jansen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/\"},\"author\":{\"name\":\"Jim Jansen\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/#\/schema\/person\/e3bb7a0b58349e548e8940716694c215\"},\"headline\":\"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment\",\"datePublished\":\"2022-06-18T11:33:54+00:00\",\"dateModified\":\"2022-09-03T09:57:30+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/\"},\"wordCount\":284,\"publisher\":{\"@id\":\"https:\/\/acua.qcri.org\/blog\/#organization\"},\"articleSection\":[\"Customer segmentation\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/\",\"url\":\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/\",\"name\":\"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment - Team Acua\",\"isPartOf\":{\"@id\":\"https:\/\/acua.qcri.org\/blog\/#website\"},\"datePublished\":\"2022-06-18T11:33:54+00:00\",\"dateModified\":\"2022-09-03T09:57:30+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/acua.qcri.org\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/#website\",\"url\":\"https:\/\/acua.qcri.org\/blog\/\",\"name\":\"Team Acua\",\"description\":\"Audience, Customer, and User Analytics\",\"publisher\":{\"@id\":\"https:\/\/acua.qcri.org\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/acua.qcri.org\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/#organization\",\"name\":\"Team Acua\",\"url\":\"https:\/\/acua.qcri.org\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/acua.qcri.org\/blog\/wp-content\/uploads\/2022\/10\/cropped-cropped-logo.png\",\"contentUrl\":\"https:\/\/acua.qcri.org\/blog\/wp-content\/uploads\/2022\/10\/cropped-cropped-logo.png\",\"width\":1466,\"height\":770,\"caption\":\"Team Acua\"},\"image\":{\"@id\":\"https:\/\/acua.qcri.org\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/#\/schema\/person\/e3bb7a0b58349e548e8940716694c215\",\"name\":\"Jim Jansen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/acua.qcri.org\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a4f97370631247bb1aed9a897d658981?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a4f97370631247bb1aed9a897d658981?s=96&d=mm&r=g\",\"caption\":\"Jim Jansen\"},\"sameAs\":[\"https:\/\/quecst.qcri.org\/blog\"],\"url\":\"https:\/\/acua.qcri.org\/blog\/author\/jjansenacm-org\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment - Team Acua","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/","og_locale":"en_US","og_type":"article","og_title":"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment - Team Acua","og_description":"For segmentation, one often need to use sentiment analysis services. Large commercial sentiment analysis tools are often deployed in software engineering due to their ease of use. However, it is not known how accurate these tools are, and whether the sentiment ratings given by one tool agree with those given by another tool. We use… Continue reading Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment","og_url":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/","og_site_name":"Team Acua","article_published_time":"2022-06-18T11:33:54+00:00","article_modified_time":"2022-09-03T09:57:30+00:00","og_image":[{"url":"https:\/\/quecst.qcri.org\/blog\/wp-content\/uploads\/2022\/06\/sentiment_cellphone-213x300.png"}],"author":"Jim Jansen","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Jim Jansen","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/#article","isPartOf":{"@id":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/"},"author":{"name":"Jim Jansen","@id":"https:\/\/acua.qcri.org\/blog\/#\/schema\/person\/e3bb7a0b58349e548e8940716694c215"},"headline":"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment","datePublished":"2022-06-18T11:33:54+00:00","dateModified":"2022-09-03T09:57:30+00:00","mainEntityOfPage":{"@id":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/"},"wordCount":284,"publisher":{"@id":"https:\/\/acua.qcri.org\/blog\/#organization"},"articleSection":["Customer segmentation"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/","url":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/","name":"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment - Team Acua","isPartOf":{"@id":"https:\/\/acua.qcri.org\/blog\/#website"},"datePublished":"2022-06-18T11:33:54+00:00","dateModified":"2022-09-03T09:57:30+00:00","breadcrumb":{"@id":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/acua.qcri.org\/blog\/engineers-aware-commercial-tools-disagree-on-social-media-sentiment\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/acua.qcri.org\/blog\/"},{"@type":"ListItem","position":2,"name":"Engineers, Aware! Commercial Tools Disagree on Social Media Sentiment"}]},{"@type":"WebSite","@id":"https:\/\/acua.qcri.org\/blog\/#website","url":"https:\/\/acua.qcri.org\/blog\/","name":"Team Acua","description":"Audience, Customer, and User Analytics","publisher":{"@id":"https:\/\/acua.qcri.org\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/acua.qcri.org\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/acua.qcri.org\/blog\/#organization","name":"Team Acua","url":"https:\/\/acua.qcri.org\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/acua.qcri.org\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/acua.qcri.org\/blog\/wp-content\/uploads\/2022\/10\/cropped-cropped-logo.png","contentUrl":"https:\/\/acua.qcri.org\/blog\/wp-content\/uploads\/2022\/10\/cropped-cropped-logo.png","width":1466,"height":770,"caption":"Team Acua"},"image":{"@id":"https:\/\/acua.qcri.org\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/acua.qcri.org\/blog\/#\/schema\/person\/e3bb7a0b58349e548e8940716694c215","name":"Jim Jansen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/acua.qcri.org\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a4f97370631247bb1aed9a897d658981?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a4f97370631247bb1aed9a897d658981?s=96&d=mm&r=g","caption":"Jim Jansen"},"sameAs":["https:\/\/quecst.qcri.org\/blog"],"url":"https:\/\/acua.qcri.org\/blog\/author\/jjansenacm-org\/"}]}},"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/331"}],"collection":[{"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/comments?post=331"}],"version-history":[{"count":2,"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/331\/revisions"}],"predecessor-version":[{"id":342,"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/posts\/331\/revisions\/342"}],"wp:attachment":[{"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/media?parent=331"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/categories?post=331"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/acua.qcri.org\/blog\/wp-json\/wp\/v2\/tags?post=331"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}