InternetOpen + Tthreads + MaxConnectionsPerServer

[vexusdev]

Hello everyone, I am in desperate need of help.

I have coded a multi threaded application using TThreads that uses InternetOpen to download html pages.

No matter if I'm using 20 or 220 threads it still downloads only so many pages at a time. I am starting to think its the MaxConnectionsPerServer settings associated with http/wininet which defaults to 2,4. I changed both 1.1 and 1.0 to 99999999 in the registry and still it seems like my connections are being limited.

I don't have the code on hand right now but ill update it soon if you guys want to see it.

Pleaseeeeeeeeeeee help!!!

[WingedPanther]

Your OS may be limiting your connections, or it could be your code. Without the code, however, we can't offer much help.

[vexusdev]

Ok heres some code to help:

Heres my thread code (some of it):

procedure TBaseThread.addlist;
var
sender:Tobject;
begin
    form1.status.AddItem(inttostr(Proxi)+'Executing Thread #'+inttostr(inum),form1.ssender);
    form1.status.ItemIndex := form1.Status.Count - 1;
    form1.ProxyIn[inum] := true;
    FetchHTML(getphone,inum);
end;

procedure TBaseThread.Execute;
begin
FreeOnTerminate := True;
    Sleep(random(1000)*random(3));
    //sleep(1000+random(400));
  while not Terminated do begin
   addlist;
  end;
end;

Heres my function it calls fetchHtml (which has internetopen)(I removed the link to the site i call):

Procedure TBaseThread.FetchHTML(url:string; intproxy:integer);
var
  databuffer : array[0..4095] of char;
  ResStr : string;
  hSession, hfile, hRequest: hInternet;
  dwindex,dwcodelen,datalen,dwread,dwNumber:cardinal;
  dwcode : array[1..20] of char;
  res    : pchar;
  Str    : pchar;
  address:string;
  port:string;
  username:string;
  password:string;
  A:TStringList;
  intUserAgent:Integer;
  strCurrentProxy:String;
  intTimeOut:Integer;
  sender:Tobject;
  lcountr:integer;
  I:Integer;
  int1:Integer;
  int2:Integer;
  phone:string;
  strc:string;
  carrier:string;
  lcount:integer;
label
resend;
label
reproxy;
begin
lcount := form1.Proxy.items.count-1;
randomize;
intUserAgent := Random(form1.useragent.Count-1);
ResStr := '';
phone:=url;
url := 'linkremoved'+phone;

if (form1.ProxyChecked.Checked) Then begin
A:=TStringList.Create;
strCurrentProxy := form1.Proxy.Items[intproxy];
A:=Form1.Split(strCurrentProxy,':');
Address:=A[0];
Port:=A[1];
If A.Count > 2 then begin
UserName:=A[2];
Password:=A[3];
End;
A.Free;

    hSession := InternetOpen(pCHar(form1.useragent.Items[intUserAgent]),
       INTERNET_OPEN_TYPE_PROXY,PChar(address+':'+port), nil, 0);
End else begin
    hSession := InternetOpen(pCHAr(form1.useragent.Items[intUserAgent]),
       INTERNET_OPEN_TYPE_DIRECT,nil, nil, 0);
End;//proxy

If assigned(Hsession) then begin
      hfile := InternetOpenUrl(
           hsession,
           pchar(url),
           nil,
           0,
           INTERNET_FLAG_RELOAD,
           0);
      dwIndex  := 0;
      dwCodeLen := 10;
         resend:
      HttpSendRequest(hfile, nil, 0, nil, 0);
      HttpQueryInfo(hfile, HTTP_QUERY_STATUS_CODE,
              @dwcode, dwcodeLen, dwIndex);
      res := pchar(@dwcode);
      dwNumber := sizeof(databuffer)-1;
If res = '407' then begin //if authenticate
    InternetSetOption (hfile, INTERNET_OPTION_PROXY_USERNAME,
            pchar(username), length(username));
    InternetSetOption (hfile, INTERNET_OPTION_PROXY_PASSWORD,
            pchar(password), length(password));
goto resend;
End;
If (res ='200') or (res ='302') then begin//success start downloading
          while (InternetReadfile(hfile,@databuffer,dwNumber,DwRead)) do begin
              if dwRead =0 then break;
              databuffer[dwread]:=#0;
              Str := pchar(@databuffer);
              resStr := resStr + Str;
          end;
End;//if 200 or 302

//close clear
if assigned(hfile) then InternetCloseHandle(hfile);
InternetCloseHandle(hsession);
if Length(resStr) = 0 then exit;
  //parse here
If form1.intSent <> form1.l1.items.Count then begin
 int1 := pos('participating',resStr);
 if int1 > 0 then begin
  inc(form1.intFailed);
  inc(form1.intSent);
  form1.ProxyIn[inum] := false;
  exit;
 End;
 int1 := pos('<B>Carrier:</B>',resStr);
 If int1 > 0 then begin
  strc:=Copy(resStr,int1,70);
  carrier:=form1.GetCarrier(form1.SimpleParse(strc,'Carrier:</B>','</td>'));
  resStr := form1.l1.Items[form1.intsent] + carrier;
  If (length(carrier)>4) then begin
   form1.lstEmails.AddItem(resStr,sender);
   form1.lstEmails.ItemIndex := form1.lstEmails.Count - 1;
   inc(form1.intSent);
   form1.ProxyIn[inum] := false;
  end else begin
   inc(form1.intFailed);
  End;
 end else begin
  //check for verizon
   int2 := pos('verizon.com',resStr);
    if (int2 > 0) then begin
     resStr := form1.l1.Items[form1.intsent] + '@vtext.com';
     form1.lstEmails.AddItem(resStr,sender);
     form1.lstEmails.ItemIndex := form1.lstEmails.Count - 1;

     
     inc(form1.intSent);
     form1.ProxyIn[inum] := false;
     exit;
    end;
 End;
 end;//check
end;
 inc(form1.intSent);
 form1.ProxyIn[inum] := false;
end;

Heres how I change the connection limit with registry entries:

SetRegistryData(HKEY_CURRENT_USER,
  '\Software\Microsoft\Windows\CurrentVersion\Internet Settings',
  'MaxConnectionsPerServer', rdInteger, 99999999);
SetRegistryData(HKEY_CURRENT_USER,
  '\Software\Microsoft\Windows\CurrentVersion\Internet Settings',
  'MaxConnectionsPer1_0Server', rdInteger, 99999999);

[WingedPanther]

It looks like you're using an Windows API call:
Delphi - WinInet FTP programming

Ultimately, you'll need to do some research on how it works.

[vexusdev]

Could you be a little bit more clear? Any idea what I need to look into more?

[WingedPanther]

Since I haven't used these functions, not really. Delphi isn't always real good about documenting their functions, so research on the net is a must.