By SMJune


2020-03-26 03:15:43 8 Comments

I scraped a website that has hundreds of pages of badly organized HTML. I used BeautifulSoup to capture all the content of a div on each page. The excerpt of that list is:

mylist = [['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/30/2019<br/>09:00:00 AM<br/>12/31/2019<br/>09:00:00 AM<br/>92112<br/>Initiate<br/>Capacity Constraint<br/>12/29/2019<br/>03:02:38 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/30/2019<br/></div>'],
['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/29/2019<br/>09:00:00 AM<br/>12/30/2019<br/>09:00:00 AM<br/>92086<br/>Initiate<br/>Capacity Constraint<br/>12/28/2019<br/>02:55:39 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/29/2019<br/></div>'],
['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/28/2019<br/>09:00:00 AM<br/>12/29/2019<br/>09:00:00 AM<br/>92074<br/>Initiate<br/>Capacity Constraint<br/>12/27/2019<br/>03:14:16 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/28/2019<br/></div>']]

How do I capture what is in between the <br/> tags, including blanks when there is nothing between them?

I should have added that the output should become a list of lists with each item in separate by a <br/> tag is a item in the list. For example:

[['"006951446", "Algonquin Gas Transmission, LLC", "Critical notice", "12/30/2019", "09:00:00 AM", "12/31/2019", "09:00:00 AM", "92112", "Initiate", "Capacity Constraint", "12/29/2019", "03:02:38 PM", "No response required", "AGT Pipeline Conditions for 12/30/2019"'],
 ['"006951446", "Algonquin Gas Transmission, LLC", "Critical notice", "12/29/2019", "09:00:00 AM", "12/30/2019", "09:00:00 AM", "92086", "Initiate", "Capacity Constraint", "12/28/2019", "02:55:39 PM", "No response required", "AGT Pipeline Conditions for 12/29/2019"'],
 ['"006951446", "Algonquin Gas Transmission, LLC", "Critical notice", "12/28/2019", "09:00:00 AM", "12/29/2019", "09:00:00 AM", "92074", "Initiate", "Capacity Constraint", "12/27/2019", "03:14:16 PM", "No response required", "AGT Pipeline Conditions for 12/28/2019"']]

4 comments

@αԋɱҽԃ αмєяιcαη 2020-03-26 03:28:59

from bs4 import BeautifulSoup
mylist = [['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/30/2019<br/>09:00:00 AM<br/>12/31/2019<br/>09:00:00 AM<br/>92112<br/>Initiate<br/>Capacity Constraint<br/>12/29/2019<br/>03:02:38 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/30/2019<br/></div>'],
          ['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/29/2019<br/>09:00:00 AM<br/>12/30/2019<br/>09:00:00 AM<br/>92086<br/>Initiate<br/>Capacity Constraint<br/>12/28/2019<br/>02:55:39 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/29/2019<br/></div>'],
          ['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/28/2019<br/>09:00:00 AM<br/>12/29/2019<br/>09:00:00 AM<br/>92074<br/>Initiate<br/>Capacity Constraint<br/>12/27/2019<br/>03:14:16 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/28/2019<br/></div>']]

for item in mylist:
    soup = BeautifulSoup(*item, 'html.parser')
    print(*[a.get_text(strip=True, separator="|").split("|") for a in soup])

Output:

['006951446', 'Algonquin Gas Transmission, LLC', 'Critical notice', '12/30/2019', '09:00:00 AM', '12/31/2019', '09:00:00 AM', '92112', 'Initiate', 'Capacity Constraint', '12/29/2019', '03:02:38 PM', 'No response required', 'AGT Pipeline Conditions for 
12/30/2019']
['006951446', 'Algonquin Gas Transmission, LLC', 'Critical notice', '12/29/2019', '09:00:00 AM', '12/30/2019', '09:00:00 AM', '92086', 'Initiate', 'Capacity Constraint', '12/28/2019', '02:55:39 PM', 'No response required', 'AGT Pipeline Conditions for 
12/29/2019']
['006951446', 'Algonquin Gas Transmission, LLC', 'Critical notice', '12/28/2019', '09:00:00 AM', '12/29/2019', '09:00:00 AM', '92074', 'Initiate', 'Capacity Constraint', '12/27/2019', '03:14:16 PM', 'No response required', 'AGT Pipeline Conditions for 
12/28/2019']

@SMJune 2020-03-26 13:13:35

This is really close. But I need each item in the list to be a separate item in the list. Like: ``` [['"006951446", "Algonquin Gas Transmission, LLC", "Critical notice", "12/30/2019", "09:00:00 AM", "12/31/2019", "09:00:00 AM", "92112", "Initiate", "Capacity Constraint", "12/29/2019", "03:02:38 PM", "No response required", "AGT Pipeline Conditions for 12/30/2019"'], ['"006951446", "Algonquin Gas Transmission, LLC", "Critical notice", "12/29/2019", "09:00:00 AM", "12/30/2019", "09:00:00 AM", "92086", "Initiate", "Capacity Constraint"']] ```

@αԋɱҽԃ αмєяιcαη 2020-03-27 07:24:16

@SMJune code updated

@SMJune 2020-03-27 13:48:50

thanks! i noticed that right before the "No response required" part, there are some <br\> with nothing in them. In the whole data, there are times when those hold some numbers. How to capture those as empty items and with data when there is data.

@αԋɱҽԃ αмєяιcαη 2020-03-27 15:08:23

@SMJune you've receive an answer based on your example. good luck

@xkcdjerry 2020-03-26 03:23:10

Usually,when you use select on a BeatifulSoup object,you get a list of Tags.
And you can use select/getText on the Tags again.
For exsample:

SEP='(--*--SEP--*--)'
mylist=soup.select('div')
between_br=[[j for j in i.getText(SEP).split(SEP) if not j.isspace()] for i in mylist]

@SMJune 2020-03-26 13:19:50

Hi, am getting an error on this one that says: SelectorSyntaxError: Invalid character '/' position 2 line 1: br/

@xkcdjerry 2020-03-27 00:34:29

Fixed,it this your wanted output?paste.ubuntu.com/p/F5x4nmbN2z

@xkcdjerry 2020-03-27 00:35:27

It's to big to paste into a comment and the formatting will be terrable. Sorry :(

@dabingsou 2020-03-26 23:24:43

Solutions using library SimplifiedDoc.

from simplified_scrapy import SimplifiedDoc,req,utils
mylist = [['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/30/2019<br/>09:00:00 AM<br/>12/31/2019<br/>09:00:00 AM<br/>92112<br/>Initiate<br/>Capacity Constraint<br/>12/29/2019<br/>03:02:38 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/30/2019<br/></div>'],
          ['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/29/2019<br/>09:00:00 AM<br/>12/30/2019<br/>09:00:00 AM<br/>92086<br/>Initiate<br/>Capacity Constraint<br/>12/28/2019<br/>02:55:39 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/29/2019<br/></div>'],
          ['<div id="headingData">006951446<br/>Algonquin Gas Transmission, LLC<br/>Critical notice<br/>12/28/2019<br/>09:00:00 AM<br/>12/29/2019<br/>09:00:00 AM<br/>92074<br/>Initiate<br/>Capacity Constraint<br/>12/27/2019<br/>03:14:16 PM<br/> <br/><br/>No response required<br/> <br/> <br/>AGT Pipeline Conditions for 12/28/2019<br/></div>']]
values = []
# First way
for item in mylist:
  doc = SimplifiedDoc(item[0])
  tmp = doc.selects('br').nextText()
  tmp.insert(0,doc.div.firstText())
  values.append(tmp)
values = []
# Second way
for item in mylist:
  doc = SimplifiedDoc(item[0])
  brs = doc.selects('br')
  tmp = [br.previousText() for br in brs]
  values.append(tmp)
print(values)

Result:

[['006951446', 'Algonquin Gas Transmission, LLC', 'Critical notice', '12/30/2019', '09:00:00 AM', '12/31/2019', '09:00:00 AM', '92112', 'Initiate', 'Capacity Constraint', '12/29/2019', '03:02:38 PM', '', '', 'No response required', '', '', 'AGT Pipeline Conditions for 12/30/2019'], ['006951446', 'Algonquin Gas Transmission, LLC', 'Critical notice', '12/29/2019', '09:00:00 AM', '12/30/2019', '09:00:00 AM', '92086', 'Initiate', 'Capacity Constraint', '12/28/2019', '02:55:39 PM', '', '', 'No response required', '', '', 'AGT Pipeline Conditions for 12/29/2019'], ['006951446', 'Algonquin Gas Transmission, LLC', 'Critical notice', '12/28/2019', '09:00:00 AM', '12/29/2019', '09:00:00 AM', '92074', 'Initiate', 'Capacity Constraint', '12/27/2019', '03:14:16 PM', '', '', 'No response required', '', '', 'AGT Pipeline Conditions for 12/28/2019']]

@jammin0921 2020-03-26 03:24:20

Without seeing the rest of your code, it might be hard to give an exact answer but is a great package for this. You should be able to continue using the bs4 package to comb through the HTML using a combination of BeautifulSoup methods.(e.g. find/find_all/select etc.)

See this answer for help on br tags

Related Questions

Sponsored Content

14 Answered Questions

[SOLVED] Web-scraping JavaScript page with Python

28 Answered Questions

[SOLVED] How do I protect Python code?

1 Answered Questions

2 Answered Questions

[SOLVED] How to scrape HTML with BeautifulSoup?

3 Answered Questions

[SOLVED] Why does Python code run faster in a function?

1 Answered Questions

[SOLVED] HTML web scraping for a value

Sponsored Content