Unable to retrieve page 2 using POST request

dazzle Published at Dev

dazzle

I am trying to retrieve the HTML of Page 2 of this URL. I have added the values of required post form data like __EVENTTARGET and __EVENTARGUMENT, but still this is returning the 1st page only. Any clue what I maybe missing?

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.LinkedHashMap;
import java.util.Map;

public class HttpURLConnectionExample {
public static void main(String[] args) throws Exception {
    URL url = new URL(
            "http://www.themetaldirectory.com/?featured=0&country=USA");
    Map<String, Object> params = new LinkedHashMap<>();
    params.put("__EVENTTARGET", "ctl00%24ctl00%24ContentPlaceHolderDefault%24DirectoryDisplay_12%24GridView1");
    params.put("__EVENTARGUMENT", "Page$2");

    StringBuilder postData = new StringBuilder();
    for (Map.Entry<String, Object> param : params.entrySet()) {
        if (postData.length() != 0)
            postData.append('&');
        postData.append(URLEncoder.encode(param.getKey(), "UTF-8"));
        postData.append('=');
        postData.append(URLEncoder.encode(String.valueOf(param.getValue()),
                "UTF-8"));
    }
    byte[] postDataBytes = postData.toString().getBytes("UTF-8");

    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    conn.setRequestMethod("POST");
    conn.setRequestProperty("Content-Type",
            "application/x-www-form-urlencoded");
    conn.setRequestProperty("Content-Length",
            String.valueOf(postDataBytes.length));
    conn.setDoOutput(true);
    conn.getOutputStream().write(postDataBytes);

    Reader in = new BufferedReader(new InputStreamReader(
            conn.getInputStream(), "UTF-8"));
    for (int c; (c = in.read()) >= 0; System.out.print((char) c))
        ;
}

}

vzamanillo

You are sending an incorrect value for __EVENTTARGET param.

The expected value is

ctl00$ctl00$ContentPlaceHolderDefault$DirectoryDisplay_12$GridView1

but you are sending the value as URL encoded twice, first when you are setting the param value

params.put("__EVENTTARGET",
"ctl00%24ctl00%24ContentPlaceHolderDefault%24DirectoryDisplay_12%24GridView1");

and second when you encode the param values at

postData.append(URLEncoder.encode(String.valueOf(param.getValue()), "UTF-8"));

so you are really sending a double URL encoded value

ctl00%2524ctl00%2524ContentPlaceHolderDefault%2524DirectoryDisplay_12%2524GridView1

set the param value without URL encoding at first and should work

 params.put("__EVENTTARGET",
 "ctl00$ctl00$ContentPlaceHolderDefault$DirectoryDisplay_12$GridView1");

EDIT

If you want to save the resulting InputStream to a String you have a lot of possile ways, one of them as follows:

StringBuilder inputStringBuilder = new StringBuilder();
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));

String line;
while ((line = br.readLine()) != null) {
    inputStringBuilder.append(line);
}
String htmlString = inputStringBuilder.toString();

Collected from the Internet

Please contact [email protected] to delete if infringement.