Use the IBM Watson Natural Language Understanding API, and need to write function extractEntities(url, source)

Question

Use the IBM Watson Natural Language Understanding API, and need to write function extractEntities(url, source) that takes as input a URL, and returns a list of dictionaries, with every dictionary containing the entity name, the URL where the entity appeared, the relevance of the entity to the article, and the sentiment about the entity. For example, after calling the function

url = 'http://www.politico.com/story/2017/05/23/infrastructure-transportation-trump-budget-238741'

extractEntities(url, 'politico')

we may get back a list of dictionaries, with each dictionary representing an entity, and looking like this: 

{

'entity': 'Donald Trump',

'url': 'http://www.politico.com/story/2017/05/23/infrastructure-transportation-trump-budget-238741',

'source': 'politico',

'relevance': 0.882851,

'sentiment': -0.25

}

The function extractEntities should return a list of dictionaries, with each dictionary in the list corresponding to one entity.

Answer 1

Here's a function that should work, though you should authenticate API before calling function. For entities, you'd want to use EntitiesOptions with sentiment set to True for Features:

CODE:

from ibm_watson import NaturalLanguageUnderstandingV1

from ibm_cloud_sdk_core.authenticators import IAMAuthenticator

from ibm_watson.natural_language_understanding_v1 import Features,EntitiesOptions

api_key = "<API_KEY_HERE>"

ibm_url = "<IBM_URL_HERE>" 

url = 'http://www.politico.com/story/2017/05/23/infrastructure-transportation-trump-budget-238741'

# authenticating:

 

authenticator = IAMAuthenticator(api_key)

natural_language_understanding = NaturalLanguageUnderstandingV1(

  version='2021-03-25',

  authenticator=authenticator

)

natural_language_understanding.set_service_url(ibm_url)

# actual function

def extractEntities(url, source):

 response = natural_language_understanding.analyze(

  url=url,

  features=Features(entities=EntitiesOptions(sentiment=True))).get_result()

 retrieved_url = response["retrieved_url"]

 entities = response["entities"]

 result = []

 for entity in entities:

  entity_dict = {}

  entity_dict["entity"] = entity["text"]

  entity_dict["url"] = retrieved_url

  entity_dict["source"] = source

  entity_dict["relevance"] = entity["relevance"]

  entity_dict["sentiment"] = entity["sentiment"]["score"]

  result.append(entity_dict)

 return result

extractEntities(url, "politico")

OUTPUT:

[{'entity': 'Trump',

 'relevance': 0.838583,

 'sentiment': -0.575215,

 'source': 'politico',

 'url': 'https://www.politico.com/story/2017/05/23/infrastructure-transportation-trump-budget-238741'},

 {'entity': 'Trump administration',

 'relevance': 0.589844,

 'sentiment': 0.634823,

 'source': 'politico',

 'url': 'https://www.politico.com/story/2017/05/23/infrastructure-transportation-trump-budget-238741'},

 {'entity': 'Transportation Committee',

 'relevance': 0.342742,

 'sentiment': -0.337258,

 'source': 'politico',

 'url': 'https://www.politico.com/story/2017/05/23/infrastructure-transportation-trump-budget-238741'},

 {'entity': 'Elaine Chao',

 'relevance': 0.320625,

 'sentiment': 0.861965,

 'source': 'politico',

 'url': 'https://www.politico.com/story/2017/05/23/infrastructure-transportation-trump-budget-238741'} 

….