Subject: RE: net_flags[4]

From: Pascal Cimon <[email-masked]>
Date: Wed, 9 Aug 2000 06:09:04 -0400

[To be removed from this list send the message "unsubscribe fwtk-users" in the
BODY of a mail message to [email-masked].]

 Actually it's more then just messages in the logs. Some sites don't come up
at all in the browser. When I access www.perl.com or www.xml.com with
http-gw they don't come up and I get the server unavailable error int the
browser and the netflag error in the fw logs. If I access the same site from
my firewall it works fine so it's not an access problem. If I point my
firewall to the http-gw it stops working for those sites. All other sites
work fine and I've only noticed those two so far.

Pascal Cimon
PCSNC for Royal Canadian Mounted Police

-----Original Message-----
From: Tony Gale
To: Larry Jackson
Cc: Mario Misic; [email-masked]
Sent: 8/8/00 12:30 PM
Subject: Re: net_flags[4]

[To be removed from this list send the message "unsubscribe fwtk-users"
in the
BODY of a mail message to [email-masked].]

On 08-Aug-2000 Larry Jackson wrote:
> From: "Mario Misic" <[email-masked]>
>>
>> I already saw that even others have problem with http-gw.
>>
>> I have http-gw V2.1 and I got this error while I am trying to
>> access
>> www.xml.com .
>>
>> Network error: net_flags[4] set (read)
>>
>> I read in FAQs that it is probably bug in http-gw software, can
>> someone
>> advice me where to find patch if it exist, I tried to find it
>> somewhere
> but
>> I didn't.
>>
>> Regards!
>> Mario Misic
> 
> I also see these error messages in my logs.
> Any progress on fixing the problem?

I was looking at this this morning, and frankly, it aint worth the
effort.

Basically, the code is bad at propagating error conditions, and this
message just catches this.

I did try fixing it, but you just end up chasing function return
values through the code.

So, ignore the messages, and if they bother you comment them out of
gio.c

-tony

---
E-Mail: Tony Gale <[email-masked]>
The shortest distance between any two puns is a straight line.

The views expressed above are entirely those of the writer
and do not represent the views, policy or understanding of
any other person or official body.

---

From: Tony Gale <[email-masked]>
Date: Wed, 09 Aug 2000 11:49:26 +0100 (BST)

They both work for me through http-gw. If you haven't applied all the
patches on www.fwtk.org, I suggest you try them, especially the
quoting fix.

All the net_flags message indicates is that either the server or
client has *already* closed the connection, but that http-gw didn't
propagate that fact properly.

-tony

PS to everyone. I'm always interested in sites that genuinely don't
work through http-gw. Just please make sure you have applied all the
patches first.

---
E-Mail: Tony Gale <[email-masked]>
The finest eloquence is that which gets things done.

---

From: Ted Keller <[email-masked]>
Date: Sat, 12 Aug 2000 10:02:44 -0400 (EDT)

I'm wondering if this is a similar error condition I found with the Apache
proxy server.

I had the situation where a number of IIS 4.0 sites (but not all of them)
were not displaying behind my proxy server.  In investigating this
situation, I noted that the TCP/IP session was never terminated with a the
trailing <fin> sequence.

What this translated to was the following....(loosly speaking)

The proxy server issues a network read request.  As each buffer was read,
it was transferred to an output buffer.  When the the terminating <fin>
was received, the read request completed with an End of File condition.
At that time the output buffer is transferred to the client.

In researching the problem, I also looked at the http-gw code and noted
that it used very similar logic - that is - it required the EOF condition
to complete the transmission to the client machine.

With out the trailing <fin> everything goes into a BIG wait status until
a TCP/IP timeout occurs - or your proxy detects an application timeout.

Well, you ask, why don't clients that are connected to the internet
directly have this problem?

I believe the clients understand the data contents of each type of item
that they are downloading.  For example, they detect the </HTML> at athe
end of an <HTML? tag - and then know that the html transfer is complete -
hence don't look for the terminating EOF character.  The same thing goes
for gif, jpeg, .... other type of files.

How to get around this....

In apache - I created a set of patches to specifically NOT buffer input
and output read/write requests.  That is, each time I recieved a block of
characters, I wrote them without buffering to the client.  I think a
similar thing can be implemented in http-gw.

Note - the final read (waiting for the <fin> still was not satisfied until
a tcp/ip timeout occurs.  I worked around this by picking some arbritrary
"timeout" period (120 seconds in most cases) which I would assume the
other end went away - and terminated my connection.

Note - the client was then fat, dumb, and happy.  The proxy server had a
few extra open connections (note - these can really increase significantly
if a site sends no terminating <fin>s for all objects). - but my users
were happy.

hope this helps

ted keller

---

From: "Larry Jackson" <[email-masked]>
Date: Sat, 12 Aug 2000 15:33:32 -0400

Is there a way to patch http-gw to fix this condition?
Or is this a HTTP server problem?

[email-masked]

From: "Ted Keller" <[email-masked]>
To: "Pascal Cimon" <[email-masked]>
Cc: "'Tony Gale '" <[email-masked]>;
       "'Larry Jackson '" <[email-masked]>;
       "'Mario Misic '" <[email-masked]>; <[email-masked]>
Sent: Saturday, August 12, 2000 10:02 AM
Subject: RE: net_flags[4]

> I'm wondering if this is a similar error condition I found with
> the Apache proxy server.
>
> > >> access www.xml.com .
> > >> I read in FAQs that it is probably bug in http-gw software,
> > >> can someone advice me where to find patch if it exist,
> > >> I tried to find it somewhere but I didn't.
> > >>
> > >> Regards!
> > >> Mario Misic
> > >
> > I was looking at this this morning, and frankly,
> > it aint worth the effort.
> >
> > Basically, the code is bad at propagating error conditions,
> > and this message just catches this.
> >
> > So, ignore the messages, and if they bother you
> > comment them out of gio.c
> >
> > Tony Gale <[email-masked]>

---

From: Ted Keller <[email-masked]>
Date: Sun, 13 Aug 2000 18:05:10 -0400 (EDT)

Larry,

This is a server problem - however - you can patch http-gw to get around
the symptoms and show your users generally good response.

In taking a quick look at the code (note - I haven't analyzed it heavily)
- I would probably start off by adding an fflush after each net_write call
in the http-gw program.  That way, you shouldn't have data hanging around
in the buffers not sent to the client.

By doing this, a couple of things will happen....

first - the client, once it thinks it has seen everything, will hang up on
this connection. You will have some dangling processes until a timeout
occurs.

Seceond - the default timeout is probaby not appropriate.  It should be
reduced down to something smaller - say 120 seconds or so.

Third - Get ready for a lot of processes in a wait state when you hit a
site with a number of objects that you get stuck in the timeout process.
Your system should handle it - you will just see a lot of them.

ted keller

---

From: Eberhard Mattes <[email-masked]>
Date: Sat, 26 Aug 2000 13:01:16 +0200 (MET DST)

> situation, I noted that the TCP/IP session was never terminated with a the

With HTTP/1.1, a transaction doesn't have to end with FIN.

> directly have this problem?

They obey Content-Length.

I think there are two solutions:

- let the proxy send "Connection: Close" to the server

- make sure that all data is sent to the client after processing all
  octets as indicated by Content-Length

My squid-gw does both.  Can you give some examples of URLs which show
the problem, so that I can verify if this really helps?

-- 
  Eberhard Mattes <[email-masked]>

---

From: Ted Keller <[email-masked]>
Date: Sun, 27 Aug 2000 07:29:17 -0400 (EDT)

Sure 

www.rati.com

support.dell.com

www.nai.com

It's been a while since I looked at the dumps... but I don't remember a
content length in the data streams coming back.  If memory server me
correctly (and it's very early on a Sunday am which means it's quite
fuzzy) I only saw content lengths listed in http 0.9 versions - not in the
1.0 or 1.1 versions.  May be wrong here.  

Let me know what you find.

tnx
tek

---

From: Eberhard Mattes <[email-masked]>
Date: Wed, 30 Aug 2000 21:50:46 +0200 (MET DST)

> www.nai.com

These seem to work with squid-gw and Squid.

---

From: Tony Gale <[email-masked]>
Date: Thu, 31 Aug 2000 09:18:18 +0100 (BST)

> These seem to work with squid-gw and Squid.

And they work for me with http-gw

-tony

---
E-Mail: Tony Gale <[email-masked]>
There is hopeful symbolism in the fact that flags do not wave in a vacuum.
		-- Arthur C. Clarke