Subject: RE: http://www.fwtk.org/fwtk/faq/faq.html#2.4.5

From: Rick Murphy <[email-masked]>
Date: Thu, 30 Nov 2000 23:11:32 -0500

[To be removed from this list send the message "unsubscribe fwtk-users" in the
BODY of a mail message to [email-masked].]

At 10:33 AM 11/30/00 +0000, Tony Gale wrote:

>Ouch. This is gonna hurt.
>
>www.boston.com is running Netscape-Enterprise/3.6 SP3
>
>It is possibly the most broken server I've ever seen. If it doesn't
>receive the GET request and a Host header in the *same packet* it
>will simply reset the connection.

I've actually seen systems like this in the past where they really weren't 
a web server, they were a shell script or perl program that tried to 
emulate one.

Anything that's as broken as Tony describes isn't going to work with 
http-gw as it's currently designed.
         -Rick

---

From: "Larry Jackson" <[email-masked]>
Date: Mon, 4 Dec 2000 14:51:34 -0500

I'm having the same problem with http://www.newsmax.com 
It worked yesterday, but today I'm getting 
Network error: net_flags[4] set (read)

[email-masked]
----- Original Message ----- 
From: "Tony Gale" <[email-masked]>
> 
> The error discussed in the FAQ is due to improper propagation of an
> error condition. The real issue is, why was the connection closed in
> the first place?
> 
> I'll have a look into what is failing. I have an even wierder one to
> look into anyway.
> 
> 
> On 29-Nov-2000 Ross Bergman wrote:
> > 
> > We've been running FWTK for a number of years now, and recently
> > I've
> > received complaints from users trying to access www.boston.com and
> > odt.jamestower.com that they're unable to do so.  The error logged
> > is
> > described in the FAQ, as you've pointed out in numerous emails over
> > the years.
> > 
> > Is there a patch for this supposed "bug in the http-gw code"?  
> > 
> > Thank you.
> > 
> > --------------------------------------------------------------------
> > -----------
> >         Ross E. Bergman                  Director of Information
> > Technologies
> >         [email-masked]     PerkinElmer Detection
> > Systems
> > --------------------------------------------------------------------
> > -----------
> 
> ---
> E-Mail: Tony Gale <[email-masked]>
> Virtue does not always demand a heavy sacrifice -- only the willingness
> to make it when necessary.
> -- Frederick Dunn
>

---

From: Tony Gale <[email-masked]>
Date: Tue, 05 Dec 2000 10:42:57 -0000 (GMT)

This site looks ok to me. Make sure you have the JavaScript quoting
fix installed from the FAQ.

-tony

---
E-Mail: Tony Gale <[email-masked]>
A man who turns green has eschewed protein.

The views expressed above are entirely those of the writer
and do not represent the views, policy or understanding of
any other person or official body.

---

From: Tony Gale <[email-masked]>
Date: Tue, 05 Dec 2000 10:49:07 -0000 (GMT)

> http-gw as it's currently designed.

Thats true, but it doesn't look like a huge amount of work to
re-implement. I alrealy have all the code to do it in another piece
of (my) software. However, it isn't a 2 line patch.

-tony

---
E-Mail: Tony Gale <[email-masked]>
The way some people find fault, you'd think there was some kind of reward.

---

From: Tony Gale <[email-masked]>
Date: Wed, 06 Dec 2000 17:52:52 -0000 (GMT)

Here's an initial effort at solving the problem with broken web
servers as discussed previously.

*** WARNING: This patch could do untold harm to you, your system, and
your outlook on life. I post if here for review and comments. No
warranty, implied or otherwise with this patch.

It buffers the client request and headers as much as possible and
then writes them out at the end.

TODO:
 o say() / sayn() / net_write() should handle incomplete write()'s
   correctly
 o sort out the indenting

Thanks

-tony

---
E-Mail: Tony Gale <[email-masked]>
Passionate hatred can give meaning and purpose to an empty life.
		-- Eric Hoffer

The views expressed above are entirely those of the writer
and do not represent the views, policy or understanding of
any other person or official body.
--- /fwtk.dist/fwtk/http-gw/http-gw.c	Fri Feb  6 23:32:25 1998
+++ http-gw.c	Wed Dec  6 17:45:04 2000
@@ -568,7 +568,7 @@
 	if( (rem_type & TYPE_HTTP) ){
 		int hlen = 0;
 		if( http_protocol[0] != '\0'){
- 			copy_http_headers( -1, sockfd, 0, &hlen, G_NOPERM);
+ 			copy_http_headers( NULL, -1, sockfd, 0, &hlen, G_NOPERM);
 		}
 	}
 	if( rem_type & TYPE_FTP){
@@ -1178,8 +1178,8 @@
 	}else{
 		sprintf(go_request,"%s%s", http_method, buf);
 	}
-	if( rfd != -1)say(rfd, go_request);
-
+/* 	if( rfd != -1)say(rfd, go_request); */
+  
 frominternal:
 	if( http_protocol[0]){	/* Was there http protocol info? */
 		int hlen = 0;
@@ -1187,7 +1187,7 @@
 
 		if( (permissions & G_PLUS) == 0)
 			rfd = -1;
- 		if( copy_http_headers(rfd, sockfd, (rem_type&TYPE_WRITE), &hlen, G_NOPERM) &
&
+ 		if( copy_http_headers(go_request, rfd, sockfd, (rem_type&TYPE_WRITE), &hlen,
 G_NOPERM) &&
 			(rem_type&TYPE_WRITE)){
 			rfd = rsaved;
 			trans_html_file(rfd, sockfd, "http", hlen);
@@ -1198,12 +1198,15 @@
 			return 0;
 		}
 		rfd = rsaved;
-	}
-
+	} else {
+    if( rfd != -1)
+      say(rfd, go_request);
+  }
+    
 /* Get possible response line (maybe an old server)*/
 	if( get_http_response(sockfd, rfd)){
 
-		if( copy_http_headers(sockfd, rfd, 1, NULL, logging)){
+		if( copy_http_headers(NULL, sockfd, rfd, 1, NULL, logging)){
 			trans_html_file(sockfd, rfd, "http", -1);
 		}
 	}else if( rem_type & TYPE_HEAD){
@@ -1230,7 +1233,10 @@
 	return 0;
 }
 
-int copy_http_headers(sockfd, rfd,expect_data, lptr, logging)
+#define REQ_BUFF_LEN 8192 /* SECURITY: Must be bigger than MAX_URL_LEN */
+
+int copy_http_headers(request, sockfd, rfd,expect_data, lptr, logging)
+char *request;
 int sockfd, rfd;
 int expect_data, *lptr, logging;
 {	int len, cnt;
@@ -1238,12 +1244,26 @@
 	int n;
 	static char content_len[80];
 	static char content_type[513];
+  static char request_buff[REQ_BUFF_LEN];
+  int request_buff_len;
 	int	tmpX;
 	char	*tmpP;
 
+  request_buff_len = 0;
+  if ( request && (strlen(request) > REQ_BUFF_LEN-3)) {
+    goto broken;
+  }
+  
+  if (request) {
+    strncpy(request_buff, request, strlen(request));
+    request_buff_len += strlen(request);
+    strncpy(request_buff+request_buff_len, "\r\n", 2);
+    request_buff_len += 2;
+  }
+  
 	content_len[0] = '\0';
 	content_type[0]= '\0';
-
+  
 	if( lptr)
 		*lptr = 0;
 
@@ -1263,7 +1283,16 @@
 		}else if( checkBrowserType &&
 		      !strncasecmp(go_request, "user-agent:", 11) ) {
 			if( sockfd != -1){
-				 say(sockfd, go_request);
+        if (strlen(go_request)+2 > REQ_BUFF_LEN) {
+          request_buff[request_buff_len-2] = '\0';
+          say(sockfd, request_buff);
+          request_buff_len = 0;
+        }
+        strncpy(request_buff+request_buff_len, go_request, strlen(go_request));
+        request_buff_len += strlen(go_request);
+        strncpy(request_buff+request_buff_len, "\r\n", 2);
+        request_buff_len += 2;
+/* 				 say(sockfd, go_request); */
 			}
 			if (nojava == 2)
 			    for (tmpX=0;
@@ -1287,7 +1316,16 @@
 			cpystr(content_type, go_request, 512);
 			content_type[511] = '\0';
 			if( sockfd != -1){
-				 say(sockfd, go_request);
+        if (strlen(go_request)+2 > REQ_BUFF_LEN) {
+          request_buff[request_buff_len-2] = '\0';
+          say(sockfd, request_buff);
+          request_buff_len = 0;
+        }
+        strncpy(request_buff+request_buff_len, go_request, strlen(go_request));
+        request_buff_len += strlen(go_request);
+        strncpy(request_buff+request_buff_len, "\r\n", 2);
+        request_buff_len += 2;
+/* 				 say(sockfd, go_request); */
 			}
 			if( logging & G_CONTYPE){
 				syslog(LLEV,"content-type=%.512s", &go_request[13]);
@@ -1299,11 +1337,29 @@
 			if( dont_touch(p) == 0){
 				sprintf(errbuf, "Location: http://%s:%u/%s", ourname, ourport, p);
 				if( sockfd != -1){
-					say(sockfd, errbuf) ;
+          if (strlen(go_request)+2 > REQ_BUFF_LEN) {
+            request_buff[request_buff_len-2] = '\0';
+            say(sockfd, request_buff);
+            request_buff_len = 0;
+          }
+          strncpy(request_buff+request_buff_len, errbuf, strlen(errbuf));
+          request_buff_len += strlen(errbuf);
+          strncpy(request_buff+request_buff_len, "\r\n", 2);
+          request_buff_len += 2;
+/* 					say(sockfd, errbuf) ; */
 				}
 			}else{
 				if( sockfd != -1){
-                                	 say(sockfd, go_request) ;
+          if (strlen(go_request)+2 > REQ_BUFF_LEN) {
+            request_buff[request_buff_len-2] = '\0';
+            say(sockfd, request_buff);
+            request_buff_len = 0;
+          }
+          strncpy(request_buff+request_buff_len, go_request, strlen(go_request)
);
+          request_buff_len += strlen(go_request);
+          strncpy(request_buff+request_buff_len, "\r\n", 2);
+          request_buff_len += 2;
+/*           say(sockfd, go_request) ; */
 				}
 			}
 		}else if( !strncasecmp(go_request, "connection:", 11)){
@@ -1312,10 +1368,23 @@
 			continue;
 		}else {
 			if( sockfd != -1){
-                                 say(sockfd, go_request) ;
+        if (strlen(go_request)+2 > REQ_BUFF_LEN) {
+          request_buff[request_buff_len-2] = '\0';
+          say(sockfd, request_buff);
+          request_buff_len = 0;
+        }
+        strncpy(request_buff+request_buff_len, go_request, strlen(go_request));
+        request_buff_len += strlen(go_request);
+        strncpy(request_buff+request_buff_len, "\r\n", 2);
+        request_buff_len += 2;
+/*         say(sockfd, go_request) ; */
 			}
 		}
 	}
+  if ( sockfd != -1) {
+    request_buff[request_buff_len-2] = '\0';
+    say(sockfd, request_buff);
+  }
 	len = -1;
 	if( content_len[0]){
 		p = index(content_len, ':');

---

From: Ross Bergman <[email-masked]>
Date: Wed, 6 Dec 2000 14:01:11 -0500 (EST)

I've applied this patch and it does appear to be doing the trick!
thank you very much, Tony.  It appears that there was a lot of work in
order to make this work.

P.S.  There were two lines in your diff file which had carriage
returns inserted by your mailer (or some other intervening program).
Any line that doesn't begin with one of (regexp) ^[ +-@] should be
attached to the preceding line.

 > any other person or official body.

-------------------------------------------------------------------------------
        Ross E. Bergman                  Director of Information Technologies
        [email-masked]     PerkinElmer Detection Systems
-------------------------------------------------------------------------------
--- /fwtk.dist/fwtk/http-gw/http-gw.c	Fri Feb  6 23:32:25 1998
+++ http-gw.c	Wed Dec  6 17:45:04 2000
@@ -568,7 +568,7 @@
  	if( (rem_type & TYPE_HTTP) ){
  		int hlen = 0;
  		if( http_protocol[0] != '\0'){
- 			copy_http_headers( -1, sockfd, 0, &hlen, G_NOPERM);
+ 			copy_http_headers( NULL, -1, sockfd, 0, &hlen, G_NOPERM);
  		}
  	}
  	if( rem_type & TYPE_FTP){
@@ -1178,8 +1178,8 @@
  	}else{
  		sprintf(go_request,"%s%s", http_method, buf);
  	}
-	if( rfd != -1)say(rfd, go_request);
-
+/* 	if( rfd != -1)say(rfd, go_request); */
+  
  frominternal:
  	if( http_protocol[0]){	/* Was there http protocol info? */
  		int hlen = 0;
@@ -1187,7 +1187,7 @@

  
  
+  request_buff_len = 0;
  
@@ -1263,7 +1283,16 @@

---

From: David Lang <[email-masked]>
Date: Wed, 6 Dec 2000 13:29:36 -0800 (PST)

As this is running https it should be routed through the http proxy on
nFirewall assuming that the code was developed useing the libraries that
support proxies. We need to check with the product team to see if this is
possible. If it is possible we need to find out how to configure it to
talk through a proxy. Initially we will want to talk to the FWD on helium
and then hit the HTTP proxy on nFirewall. Eventually it will hit a HTTP
proxy on greenlantern, then a HTTP proxy on nFirwewall which will be a
different configuration on the webserver. 

If the software does not support proxies then we need to push development
to get it to support proxies and in the meantime put in the GSP they
origionally requested.

David Lang

  On Wed, 6 Dec 2000, Tony Gale wrote:

 > Date: Wed, 06 Dec 2000 17:52:52 -0000 (GMT)
 > From: Tony Gale <[email-masked]>
 > To: Tony Gale <[email-masked]>
 > Cc: Ross Bergman <[email-masked]>, [email-masked],
 >      Rick Murphy <[email-masked]>
 > Subject: patch candidate (was RE: http://www.fwtk.org/fwtk/faq/faq.html#2
 > 
 > 

  

+  request_buff_len = 0;
  
@@ -1263,7 +1283,16 @@

---

From: Tony Gale <[email-masked]>
Date: Thu, 07 Dec 2000 17:17:50 -0000 (GMT)

Version .2

Changes:
 o Try harder to send data to remote connections
        - This patch in hmain.c may conflict with existing patches,
          but it is quite easy to sort out. 

 o Sort out indenting

(should not have a wrap problem this time)

Thanks

-tony

---
E-Mail: Tony Gale <[email-masked]>
There is hopeful symbolism in the fact that flags do not wave in a vacuum.
		-- Arthur C. Clarke

---

From: Ross Bergman <[email-masked]>
Date: Fri, 8 Dec 2000 11:18:56 -0500 (EST)

Installed and working, with the exception you noted in hmain.c.