Retrieve KEYWORDS from META tag in a HTML WebPage using JAVA_问答_开发者

Retrieve KEYWORDS from META tag in a HTML WebPage using JAVA

开发者 https://www.devze.com 2023-02-13 14:27 出处：网络

I want to retrieve all the content words from a HTML WebPage and all the keywords contained in the META TAG of the same HTML webpage using Java.

For example, consider this html source code:

<html>
<head>
<meta name = "keywords" content = "deception, intricacy, treachery">
</head>
<body>
My very short html document. 
<br>
It has just 2 'lines'.
</body>
</html>

The CONTENT WORDS here are: my, very, short, html, document, it, has, just, lines

Note: The punctuation and the number '2' are ruled out.

The KEYWORDS here are: deception, intricacy, treachery

I have created a class for this purpose called WebDoc, this is as far as I have been able to get.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;
import java.util.Set;
import java.util.TreeSet;

public class WebDoc {

    protected URL _url;
    protected Set<String> _contentWords;
    protected Set<String> _keyWords

    public WebDoc(URL paramURL) {
        _url = paramURL;
    }

    public Set<String> getContents() throws IOException {
        //URL url = new URL(url);
        Set<String> contentWords = new TreeSet<String>();
        BufferedReader in = new BufferedReader(new InputStreamReader(_url.openStream()));
        String inputLine;

        while ((inputLine = in.readLine()) != null) {
            // Process each line.
            contentWords.add(RemoveTag(开发者_开发问答inputLine));
            //System.out.println(RemoveTag(inputLine));
        }
        in.close();
        System.out.println(contentWords);
        _contentWords = contentWords;
        return contentWords;
    }    

    public String RemoveTag(String html) {
        html = html.replaceAll("\\<.*?>","");
        html = html.replaceAll("&","");
        return html;
    }



    public Set<String> getKeywords() {
        //NO IDEA !
        return null;
    }

    public URL getURL() {
        return _url;
    }

    @Override
    public String toString() {
        return null;
    }
}

So, after the answer from RedSoxFan about the meta-keywords, you only need to split your content lines. You can use a similar method there:

Instead of

contentWords.add(RemoveTag(inputLine));

use

contentWords.addAll(Arrays.asList(RemoveTag(inputLine).split("[^\\p{L}]+")));

.split(...) splits your line at all non-letters (I hope this works, please try and report), giving back an array of substrings, which each should contain only of letters, and some empty strings between.
Arrays.asList(...) wraps this array in a list.
addAll(...) adds all the elements of this array to the set, but not duplicates).

At the end you should delete the empty string "" from your contentWords-Set.

Process each line and use

public Set<String> getKeywords(String str) {
        Set<String> s = new HashSet<String>();
        str = str.trim();
        if (str.toLowerCase().startsWith("<meta ")) {
           if (str.toLowerCase().matches("<meta name\\s?=\\s?\"keywords\"\\scontent\\s?=\\s?\".*\"/?>")) {
               // Returns only whats in the content attribute (case-insensitive)
               str = str.replaceAll("(?i)<meta name\\s?=\\s?\"keywords\"\\scontent\\s?=\\s?\"(.*)\"/?>","$1");
               for (String st:str.split(",")) s.add(st.trim());
               return s;
           }
        }
        return null;
    }

If you need an explanation, let me know.