Guys,
I'm using the following sample request body message to get news sentiment for Apple:
{ "ExtractionRequest": { "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.NewsAnalyticsExtractionRequest", "ContentFieldNames": ["Headline", "Story Body", "Story Date Time", "Take Date Time", "Created Date", "Novelty Timestamp", "Attribution", "Products", "Topics", "Language", "Relevance", "Sentiment", "Sentiment - Negative", "Sentiment - Neutral", "Sentiment - Positive"], "IdentifierList": { "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList", "InstrumentIdentifiers": [ { "Identifier": "AAPL.O", "IdentifierType": "Ric" } ], "ValidationOptions": {"AllowHistoricalInstruments": true}, "UseUserPreferencesForValidationOptions": false }, "Condition": { "ReportDateRange": "Range", "QueryStartDate": "2019-01-01", "QueryEndDate": "2019-04-30", "NewsAnalyticsRelevanceOperator": "GreaterThanOrEqualTo", "NewsRelevanceValue": 0.5, "NewsAnalyticsPrevailingSentiment": "Positive", "NewsAnalyticsNovelty": "Novelty7Day", "NewsFilterNoveltyOperator": "GreaterThanOrEqualTo", "NewsNoveltyValue": 10, "NewsTopicsCodes":[ "CMPNY","TECH" ], "IncludeImbalace": true, "NewsAnalyticsSource": "ArticlesAndAlerts", "NewsItemsSource":"Selected", "NewsAttributionsCodes": [ "RTRS", "BSW" ] } } }
However, it seems some news record are the same, is there a way to distinguish/filter out duplicated news records?
e.g. the below 2 records in the response seems to be duplicated, what's the way to determine such duplicated news? Headline is the same, however, date time is different, sentiment being the same, maybe because the news content is the same?
{ "IdentifierType": "Ric", "Identifier": "AAPL.O", "Headline": "U.S. RESEARCH ROUNDUP- Apple, Merck & Co, Voya Financial", "Story Body": null, "Story Date Time": "03/26/2019 18:15:45", "Take Date Time": "03/26/2019 18:15:45", "Created Date": "03/26/2019 18:15:58", "Novelty Timestamp": "03/26/2019 18:15:45", "Attribution": "RTRS", "Products": "E U NAW C SOF PSC", "Topics": "RCH US REP BLR CMPNY RETE FOOD1 AMED MRCH PHAG COMS SOFW MDIA RSPC CYCS SHOP SHOPAL NCYC FOBE FOTB HECA HLTH HPRD PHMR PHAR TECH TEEQ TMT SWIT CCOS AMERS LEN RTRS", "Language": "EN", "Relevance": 1, "Sentiment": 1, "Sentiment - Negative": 0.0581011, "Sentiment - Neutral": 0.152388, "Sentiment - Positive": 0.789511 }, { "IdentifierType": "Ric", "Identifier": "AAPL.O", "Headline": "U.S. RESEARCH ROUNDUP- Apple, Merck & Co, Voya Financial", "Story Body": null, "Story Date Time": "03/26/2019 22:08:27", "Take Date Time": "03/26/2019 22:08:27", "Created Date": "03/26/2019 22:08:41", "Novelty Timestamp": "03/26/2019 22:08:27", "Attribution": "RTRS", "Products": "E U NAW C SOF PSC", "Topics": "RCH US REP BLR CMPNY RETE FOOD1 AMED MRCH PHAG COMS SOFW MDIA RSPC CYCS SHOP SHOPAL NCYC FOBE FOTB HECA HLTH HPRD PHMR PHAR TECH TEEQ TMT SWIT CCOS AMERS LEN RTRS", "Language": "EN", "Relevance": 1, "Sentiment": 1, "Sentiment - Negative": 0.0581011, "Sentiment - Neutral": 0.152388, "Sentiment - Positive": 0.789511 }