How to Extract Bonds Information on Markets Insider via Python? Part 2

Previously on the How to Extract Corporate Bonds Information on Markets Insider via Python? Part 1, we discussed how to extract bond information and store it into SQLite DB. In this article, we will further discuss on how to extract hyper link for each item so that we can enter it to pull in-depth information.

We will leverage BeautifulSoup to parse the response and grab the hyper link information. Let’s take a page as an example:

import pandas as pd
import requests
from bs4 import BeautifulSoup

url = 'https://markets.businessinsider.com/bonds/finder?p=1&borrower=&maturity=midterm&yield=0&bondtype=2%2C3%2C4%2C16&coupon=0&currency=333&rating=&country=18'

df = pd.read_html(url)[0]

response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

We can find the target information is located under <table> -> <tr> -> <td>-><a> => href.

<table class="table">
...
  <tr class="table__tr">
    <td class="table__td">
    <a href="/bonds/dl-inflation-prot_secs_1828-Bond-2028-us912828y388">
    United States of America</a>
    </td>
  </tr>
  

As a result, we can parse it based on this structure to extract the link values as follows.

table = soup.find('table')

  links = []
  for tr in…