Subject: RE: http1.1 problem

From: "Overdijk, Harrie" <[email-masked]>
Date: Tue, 11 Jul 2000 14:18:26 +0200

[To be removed from this list send the message "unsubscribe fwtk-users" in the
BODY of a mail message to [email-masked].]

Dear Tony,

You wrote us the following:

> -----Original Message-----
> From:	Tony Gale [SMTP:[email-masked]]
> Sent:	Tuesday, July 11, 2000 12:42 PM
> To:	Overdijk, Harrie
> Cc:	[email-masked]; Rick Murphy; Mark C Farone; Joseph S D Yao
> Subject:	RE: http1.1 problem
> 
> 
> http-gw doesn't follow the RFCs on this. When I find the right RFC
> that lists all the unsafe URL characters I'll make a patch - can't
> remember the RFC number at the moment (hint).
> 
	[Harrie]  The hint accepted. ;-) It is RFC-1738.
	[Harrie]  It contains a part listing unsafe characters.
	[Harrie]  If you want the text, mail me privately then
	[Harrie]  I'll send it to you.

> I've looked at this before, so know where in the code to do it.
> 
	[Harrie]  Great, it saves me from searching the code!

	Greetings, 
	          Harrie

Harrie Overdijk, ECN, Postbox 1, 1755 ZG  Petten (NH), The Netherlands.
Internet: [email-masked]              Noisenet: +31 224 56 4597
Fidonet : 2:500/43.1902  (At home!)    FAXnet  : +31 224 56 1864
ICP/Postmaster/UseNet-News-manager/WWW-manager/Security-manager of ECN.

---

From: Mark C Farone <[email-masked]>
Date: Tue, 11 Jul 2000 08:48:30 -0400

>	[Harrie]  I'll send it to you.

Hint gladly accepted.
FWIW, RFC1808 updates some aspects of RFC1738 and specifies the 
standard for relative urls.
-- 
Mark C. Farone <[email-masked]>   In the future,
Systems Analyst, Gainesville Sun        everything will work.

---

From: Tony Gale <[email-masked]>
Date: Tue, 11 Jul 2000 14:03:23 +0100 (BST)

Right, here goes. http-gw runs the requested URL through
auth.c:unesc() which unescapes the %nn sequences in the URL. Now, it
has a list of reserved characters which it doesn't unescape, which
are listed at the top of auth.c (per protocol, although only HTTP
is actually supported):

struct res_chars        res_char
        { TYPE_HTTP, "()%:+/;?=&\r\n"},
        { 0, NULL}
};

Now, there are a few characters missing from this list (# being an
obvious one)

My suggested change is to:

        { TYPE_HTTP, "#{}|\\^~[] <>\"()%:+/;?=&\r\n"},

Now, as I don't happen to have URLs lying around with those
characters escaped in them, it is slightly hard to test. I have
though been using http-gw with # added for ages with no problems, and
it fixed a user reported problem.

A patch to add those listed above is included.

Comments?

-tony

> ICP/Postmaster/UseNet-News-manager/WWW-manager/Security-manager of

---
E-Mail: Tony Gale <[email-masked]>
Don't put off for tomorrow what you can do today because if you enjoy it today,
you can do it again tomorrow.

The views expressed above are entirely those of the writer
and do not represent the views, policy or understanding of
any other person or official body.

---

From: Mark C Farone <[email-masked]>
Date: Tue, 11 Jul 2000 10:30:33 -0400

>A patch to add those listed above is included.

Thanks, Tony.
That definitely is the place.

Using your suggestion and re-reading rfc1738, I interpret it to mean 
that for those characters which are "unsafe" or "reserved" we never 
want to unescape them  where:
  "unsafe" { <>"#%{}|\^~[]`}
  "reserved" {;/?:@=&}
and additionally, crlf

However, I think maybe we shouldn't unescape what it calls "special" 
characters (those which can exist unencoded in the URL) if they've 
already been escaped for whatever reason:
"special" {$-_.+!*'(),}

So, when I change that same line in auth.c to:
{ TYPE_HTTP, " <>\"#%{}|\\^~[]`;/?:@=&$-_.+!*'(),\r\n"},

I am able to correctly bring up this URL through http-gw:
http://www.eshopflorida.com/MALL/Catalog/StoreRedir/asp/url/http%3A%2F%2Fwww%2Eaaadiscountsports%2Ecom/company_id/1000011398/name/AAA+Discount+Sports%2Ecom/mall_id/1000010557.html

and get the same data as if I went direct.

Any comments, especially as this would relate to security?

Mark

---

From: Tony Gale <[email-masked]>
Date: Tue, 11 Jul 2000 16:19:43 +0100 (BST)

(Rick especially, please see comments at end)

> already been escaped for whatever reason:

This is where is gets tricky. To some extent this will depend upon
how broken peoples cgi scripts are. And there are a *LOT* of broken
cgi scripts around. In theory, I don't believe that the specials are
required, but they are probably a good idea.

> http://www.eshopflorida.com/MALL/Catalog/StoreRedir/asp/url/http%3A%
> 2F%2Fwww%2Eaaadiscountsports%2Ecom/company_id/1000011398/name/AAA+Di
> scount+Sports%2Ecom/mall_id/1000010557.html
> 

The only thing that comes to mind:

  - the "-dest" option, which does destination checking

but I don't think many people use it. Assuming that it is a problem,
it was there already, but just got worse.

-tony

---
E-Mail: Tony Gale <[email-masked]>
Q:	What does a WASP Mom make for dinner?
A:	A crisp salad, a hearty soup, a lovely entree, followed by
	a delicious dessert.