How to remove duplicate elements in HTML?

[Xiesxes]

Hello to all,

I´m trying to clean an html file that has repeated paragraphs within body. Below I show the input file and expected output.

Input.html https://jsfiddle.net/97ptc0Lh/4/

Output.html https://jsfiddle.net/97ptc0Lh/1/

I've been trying with the following code using BeautifulSoup but I don´t know why is not working, since the resultant list CleanHtml contains the repeated elements (paragraphs) that I´d like to remove. I already asked here, but still no much progress.

from bs4 import BeautifulSoup

fp = open("Input.html", "rb")
soup = BeautifulSoup(fp, "html5lib")

Uniques = set()
CleanHtml = []

for element in soup.html:
    if element not in Uniques:
        Uniques.add(element)
        CleanHtml.append(element)   

print (CleanHtml)

Thanks in advance for any help.

[Xiesxes]

Hi, is there something wrong with my question? It was my first question and nobody answered. Thanks

[Larz60+]

Hi, is there something wrong with my question? It was my first question and nobody answered. Thanks

I don't believe anyone is ignoring you, it's just that, speaking for myself, I think of html as a delivery medium that is generated on the fly, and as such cannot be thought of as static, so why bother.

That being said, you may have a legitimate reason for doing so, but again an unusual request.