From McManus@appliedtheory.com Thu Jan 25 15:52:48 2001 Return-Path: Received: from localhost (IDENT:mcmanus@localhost [127.0.0.1]) by justice.atc-bos.com (8.9.3/8.8.7) with ESMTP id PAA11703 for ; Thu, 25 Jan 2001 15:52:48 -0500 Received: from localhost by localhost with POP3 (fetchmail-5.0.0) for mcmanus@localhost (single-drop); Thu, 25 Jan 2001 15:52:48 -0500 (EST) Received: from justice.atc-bos.com (ip-216-73-149-66.vantas.net [216.73.149.66]) by franklin.appliedtheory.com (8.10.0/8.10.0(ATC)) with ESMTP id f0PKpmX10104 for ; Thu, 25 Jan 2001 15:51:49 -0500 (EST) Received: (from mcmanus@localhost) by justice.atc-bos.com (8.9.3/8.8.7) id PAA11677; Thu, 25 Jan 2001 15:51:23 -0500 Date: Thu, 25 Jan 2001 15:51:23 -0500 From: Patrick McManus To: Bernhard Horstmann Cc: patrick mcmanus Subject: Re: Squid Gzip Encoding Message-ID: <20010125155123.A11656@AppliedTheory.com> Reply-To: mcmanus@appliedtheory.com References: <000701c086ac$cab32990$89010a36@spiegel.de> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.2i In-Reply-To: <000701c086ac$cab32990$89010a36@spiegel.de>; from Horstmann@spiegel.de on Thu, Jan 25, 2001 at 09:56:35AM +0100 X-UIDL: ",n"!nL`!!Zn"#!o$_"! Status: RO Content-Length: 57249 Lines: 2220 bernhard, the patches do transfer encodings according to client negotiation (as opposed to content encodings by user agent negotiation).. They are appendeded here.. I'm sorry that I've lost any other documentation, but I'll forward the posting to the squid-dev list I made at the time to you... they are likely to be very out of date, and I don't maintain them at this time. -Pat [Bernhard Horstmann: Thu, Jan 25, 2001 at 09:56:35AM +0100] > Hi Patrick, > > in a Squid newslist I did see a posting > http://www.squid-cache.org/mail-archive/squid-users/200001/0409.html > claining that you have done some work to have squid zip/unzip > content_encoded webpages according to the agents preference. The link in the > above posting to more information at > http://proximate.appliedtheory.com/transfer-encoding/ doesn´t work for me. > Could you please point me someplace where this squid problem is solved? > > Thanks a lot > > Bernhard > > _______________________________ > Bernhard Horstmann > Technischer Leiter / CTO > SPIEGELnet AG > Tel.: 040 / 3 80 80 - 262 > Fax: 040 / 3 80 80 - 223 > Index: doc/TE-template.txt =================================================================== RCS file: TE-template.txt diff -N TE-template.txt --- /dev/null Thu Aug 24 05:00:32 2000 +++ /tmp/cvsYB4z8e Thu Jan 25 15:48:05 2001 @@ -0,0 +1,33 @@ +Patrick McManus 08181999 +mcmanus@AppliedTheory.com + +The template transfer encoding. + +defines the token "template" for use in transfer-encoding and TE +headers of response and request, respectively. + +rfc 2616 - 14.41 explicitly allows other entity-header fields to be +used in conjunction with tokens in transfer-encodings. "template" +introduces the hop by hop header transfer-template that requires +exactly one absoulteURI and an etag.. as this is a hop-to-hop header, +transfer-template must be listed in the Connection header so as not to +break proxy chains. + +transfer-template: [absoluteURI|"self"] etag + +entity content is created from a template encoding by applying the +message contents as a patch to the entity body of the absoulteURI +specified by the transfer-template header. It is hoped that that URI +is a resource with a very static expiration profile, which MUST be +retrieved using the If-Match header and strong validation on the etag +to prevent data corruption. Additionally, servers SHOULD make use of the +content-md5 header in conjunction with this transfer-encoding. + +servers may substitute the keyword "self" to mean the URI that the +request refers to. In this case the etag SHOULD be an etag provided by +the client in a If-Match or If-None-Match header (as it indicates the +client already has that entity). + +definition of patch to follow. + + Index: src/HttpHeader.c =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/HttpHeader.c,v retrieving revision 1.1.1.1 retrieving revision 1.4 diff -c -u -r1.1.1.1 -r1.4 --- src/HttpHeader.c 1999/08/10 14:36:24 1.1.1.1 +++ src/HttpHeader.c 1999/08/20 16:35:39 1.4 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * DEBUG: section 55 HTTP Header * AUTHOR: Alex Rousskov @@ -109,7 +109,10 @@ {"Retry-After", HDR_RETRY_AFTER, ftStr}, /* for now (ftDate_1123 or ftInt!) */ {"Server", HDR_SERVER, ftStr}, {"Set-Cookie", HDR_SET_COOKIE, ftStr}, + {"TE",HDR_TE, ftStr}, {"Title", HDR_TITLE, ftStr}, + {"Transfer-Encoding", HDR_TRANSFER_ENCODING,ftStr}, + {"Transfer-Template", HDR_TRANSFER_TEMPLATE,ftStr}, {"Upgrade", HDR_UPGRADE, ftStr}, /* for now */ {"User-Agent", HDR_USER_AGENT, ftStr}, {"Vary", HDR_VARY, ftStr}, /* for now */ @@ -141,13 +144,15 @@ HDR_CONNECTION, HDR_IF_MATCH, HDR_IF_NONE_MATCH, HDR_LINK, HDR_PRAGMA, - /* HDR_TRANSFER_ENCODING, */ + /* HDR_EXPECT, */ + HDR_TE, + HDR_TRANSFER_ENCODING, HDR_UPGRADE, HDR_VARY, HDR_VIA, /* HDR_WARNING, */ HDR_WWW_AUTHENTICATE, - /* HDR_EXPECT, HDR_TE, HDR_TRAILER */ + /* HDR_TRAILER, */ HDR_X_FORWARDED_FOR }; @@ -155,7 +160,6 @@ static http_hdr_type GeneralHeadersArr[] = { HDR_CACHE_CONTROL, HDR_CONNECTION, HDR_DATE, HDR_PRAGMA, - /* HDR_TRANSFER_ENCODING, */ HDR_UPGRADE, /* HDR_TRAILER, */ HDR_VIA @@ -166,7 +170,9 @@ { HDR_ALLOW, HDR_CONTENT_BASE, HDR_CONTENT_ENCODING, HDR_CONTENT_LANGUAGE, HDR_CONTENT_LENGTH, HDR_CONTENT_LOCATION, HDR_CONTENT_MD5, - HDR_CONTENT_RANGE, HDR_CONTENT_TYPE, HDR_ETAG, HDR_EXPIRES, HDR_LAST_MODIFIED, HDR_LINK, + HDR_CONTENT_RANGE, HDR_CONTENT_TYPE, HDR_ETAG, HDR_EXPIRES, + HDR_LAST_MODIFIED, HDR_LINK,HDR_TRANSFER_ENCODING, HDR_TRANSFER_TEMPLATE, + HDR_OTHER }; @@ -191,6 +197,7 @@ HDR_IF_MATCH, HDR_IF_MODIFIED_SINCE, HDR_IF_NONE_MATCH, HDR_IF_RANGE, HDR_MAX_FORWARDS, HDR_PROXY_CONNECTION, HDR_PROXY_AUTHORIZATION, HDR_RANGE, HDR_REFERER, HDR_REQUEST_RANGE, + HDR_TE, HDR_USER_AGENT, HDR_X_FORWARDED_FOR }; Index: src/HttpHeaderTools.c =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/HttpHeaderTools.c,v retrieving revision 1.1.1.1 retrieving revision 1.3 diff -c -u -r1.1.1.1 -r1.3 --- src/HttpHeaderTools.c 1999/08/10 14:36:25 1.1.1.1 +++ src/HttpHeaderTools.c 1999/08/11 19:23:16 1.3 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * DEBUG: section 66 HTTP Header Tools * AUTHOR: Alex Rousskov @@ -171,13 +171,59 @@ { const char *pos = NULL; const char *item; + int matchsize, len_m; + assert(list && m); - while (strListGetItem(list, del, &item, NULL, &pos)) { - if (!strcasecmp(item, m)) - return 1; + + /* bug mcmanus@appliedtheory.com 08111999 - + length of match not used for an explicit compare.. so in a + list of N items, items < N would never match through this + function */ + + len_m = strlen (m); + + while (strListGetItem(list, del, &item, &matchsize, &pos)) { + if ((len_m == matchsize) &&(!strncasecmp(item, m,matchsize))) + return 1; } return 0; } + + +int strListIsMember_q (const String * list, const char *m, char del) +{ + int rv; + const char *pos = NULL; + const char *item, *l; + int matchsize, len_m; + + assert(list && m); + len_m = strlen (m); + rv = 1000; + + while (strListGetItem(list, del, &item, &matchsize, &pos)) + { + if (((l=strchr (item,';'))) && + ((l-item) < matchsize)) + { + matchsize = l-item; + /* rtrim again */ + for (l--; (l>=item) && xisspace(*l);matchsize--); + l = strchr (item+matchsize,'q'); + if (l) l++; + for (;l && *l && xisspace(*l);l++); + if (l&&(*l== '=')) + rv = ((double)atof(l+1)) * 1000; + + } + + if ((len_m == matchsize) &&(!strncasecmp(item, m,matchsize))) + return rv; + } + return 0; +} + + /* returns true iff "s" is a substring of a member of the list */ int Index: src/client_side.c =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/client_side.c,v retrieving revision 1.1.1.1 retrieving revision 1.32 diff -c -u -r1.1.1.1 -r1.32 --- src/client_side.c 1999/08/10 14:36:40 1.1.1.1 +++ src/client_side.c 1999/08/27 17:52:03 1.32 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * DEBUG: section 33 Client-side Routines * AUTHOR: Duane Wessels @@ -34,6 +34,7 @@ */ #include "squid.h" +#define ATHY_COMPRESSION #if IPF_TRANSPARENT #if HAVE_SYS_IOCTL_H @@ -46,8 +47,12 @@ #include #endif +#ifdef ATHY_COMPRESSION +#include +#endif + #if LINGERING_CLOSE #define comm_close comm_lingering_close #endif @@ -93,6 +98,10 @@ static int httpAcceptDefer(void); static log_type clientProcessRequest2(clientHttpRequest * http); static int clientReplyBodyTooLarge(int clen); +static HttpReply *clientBuildReply(clientHttpRequest * http, + const char *buf, size_t size); +static int undo_template (char *ibuf, int ibufl, + char **obuf, int *obufl, void **d); static int checkAccelOnly(clientHttpRequest * http) @@ -433,7 +442,7 @@ CLIENT_SOCK_SZ, buf, clientHandleIMSReply, - http); + http); return; } } else if (clientGetsOldEntry(entry, http->old_entry, http->request)) { @@ -690,7 +699,16 @@ clientdbUpdate(conn->peer.sin_addr, http->log_type, PROTO_HTTP, http->out.size); } if (http->acl_checklist) - aclChecklistFree(http->acl_checklist); + aclChecklistFree(http->acl_checklist); + while (http->te_translations) + { + TE_list *tf; + + tf = http->te_translations; + http->te_translations= http->te_translations->next; + xfree (tf); + } + if (request) checkFailureRatio(request->err_type, http->al.hier.code); safe_free(http->uri); @@ -1118,6 +1136,969 @@ } } +/* Transfer Encoding Filters: They work like this.. + + filter (a,b,c,d,e) + a - is a character pointer to the data to be filtered + b - is length of a + c - is address of a character pointer where the resulting buffer should be stored + d - pointer to integer to store length of c in + e - pointer to a general data pointer that the filter may use to maintain state, + it begins life as NULL. + +the return value is a 4 bit mask + 0x01 - c is a newly allocated buffer and should be freed when the calling + function has completed the filter + 0x02 - reserved internally for chunking + 0x04 - reserved internally for chunking + 0x08 - set this bit if your funciton was called with input, but did not + produce any (for instance if you're buffering it in the context provided by e.) + prevents squid from thinking EOF has been reached. + + call sequence + first call: *e is NULL but a is not and b>0 + body calls : a is not null, b>0 and value of *e determined by previous calls + last call: b==0.. good time to clean up *e if you've stored stuff there.. you + may produce output if necessary, but last call will be repeated. +*/ + +/* this is a TE filter */ +static int unbiff (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + + *obuf = ibuf; + *obufl = ibufl; + + if (!ibufl) return 0; + + for (;ibufl>0;ibufl--) + if (ibuf[ibufl-1] == 'a') + ibuf[ibufl-1] ='b'; + else + if (ibuf[ibufl-1] == 'b') + ibuf[ibufl-1] ='a'; + return 0; +} + +/* this is a TE filter */ +static int unjones (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + + *obufl = ibufl; + *obuf = ibuf; + + if (!ibufl) return 0; + + for (;ibufl>0;ibufl--) + if (ibuf[ibufl-1] == 'c') + ibuf[ibufl-1] ='b'; + else + if (ibuf[ibufl-1] == 'b') + ibuf[ibufl-1] ='c'; + return 0; +} + +#ifdef ATHY_COMPRESSION + +/* this is a TE filter */ +static int dogzip (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + compression_t *m13; + char *tb; + int tbl, of,created; + + m13 = *((compression_t **) d); + + if (!m13) + { + if (!ibufl) /* no input, nothing to finish.. */ + { + *obufl = 0; + return 0; + } + + m13 = compression_init (COMPRESSION_GZIP,COMPRESSION_STD); + *d = (void *) m13; + } + + tbl = ibufl+512; + tb = xmalloc (tbl); + + if (ibufl > 0) + { + created = 0; + do + { + created += compression_shrink (m13,ibuf,ibufl,tb+created,tbl,&of); + if (of>0) + { + tbl = of+16; + tb = xrealloc (tb,created+tbl); + ibufl = 0; + } + } + while (of>0); + + } + else /* end signal */ + { + created = compression_free (m13,tb); + *d = NULL; + } + + + *obuf = tb; + *obufl = created; + + return 1; +} + +/* this is a TE filter */ +static int identity (char *ibuf, int ibufl, + char **obuf, int *obufl, void **d) +{ + *obuf = ibuf; + *obufl = ibufl; + return 0; +} + +static void patch_init (template_t *t, int header_sz) +{ + int rs; + char buffer[512]; + template_t *template = t; /* synonym */ + + t->patch = xmalloc (sizeof (patch_t)); + t->patch->fn = xstrdup(tmpnam(NULL)); + + /* write template to disk for gnu patch */ + t->patch->f = fopen (t->patch->fn,"w"); + + rs = template->rpos - header_sz; + fwrite (template->buf+header_sz,1,rs,t->patch->f); + + fclose (t->patch->f); + + snprintf (buffer,512,"/usr/bin/patch %s 2> /dev/null >/dev/null", + t->patch->fn); + t->patch->f = popen (buffer,"w"); + t->patch->readoff = 0; + + return; +} + + +static void patch_chunk (template_t *t,char *datai, int d) +{ + if (d>0) + fwrite (datai,1,d,t->patch->f); + return; +} + +static void patch_chunk_nomoreinput (template_t *t) +{ + pclose (t->patch->f); + t->patch->f = fopen (t->patch->fn,"r"); + return; +} + + +static int patch_getchunk (template_t *t, char *buf, int sz) +{ + int toget; + + toget = fread (buf,1,sz,t->patch->f); + return toget; + +} + +static void patch_free (template_t *t) +{ + fclose (t->patch->f); + unlink (t->patch->fn); + xfree (t->patch->fn); + xfree (t->patch); + t->patch = NULL; + return; +} + +static void template_free (template_t *template) +{ + + xfree (template->buf); + xfree (template->uri); + + if (template->e) + { + storeUnregister(template->e, template); + storeUnlockObject(template->e); + } + + xfree (template); + +} + +static void +undotemplate_callback(void *data, char *buf, ssize_t size) +{ + template_t *template; + ETag etag; + int waste; + + template = data; + + template->rpos += size; + + if ((template->alcd - template->rpos) < 4096) + { + template->alcd += 8192; + template->buf = xrealloc (template->buf,template->alcd); + } + + + if (size > 0) + storeClientCopy(template->e,template->rpos,template->rpos, + template->alcd - template->rpos, + template->buf+template->rpos, + undotemplate_callback, template); + else + { + char *ecmp; + + etag = httpHeaderGetETag(&template->e->mem_obj->reply->header, HDR_ETAG); + ecmp =etag.str?etag.str:""; + + /* TODO right now all etags match.. */ + printf ("etag is %s .. ",ecmp); + printf ("compare to %s ..",template->etag); + printf ("They %smatch.\n",!strcmp(ecmp,template->etag)?"":"don't "); + + if (strcmp(template->etag,ecmp)) + { /* CACHED WRONG COPY! */ + template->state =1; + xfree (template->buf); + storeUnregister(template->e, template); + storeUnlockObject(template->e); + template->e = NULL; + template->rpos = template->alcd = 0; + undo_template (NULL,0, NULL, &waste, &template); + } + else + { + template->state = 20; + template->http->oldte_rv &= 0xEF; + } + + } + return; +} + +static void wrappercbdataXfree(void *p, int unused) +{ + clientHttpRequest *http, *old; + + http = p; + printf ("process miss complete\n"); + + old=http->stack; + http->conn->chr = old; + + xfree(p); + + old->oldte_rv &= 0xEF; + old->spool_complete = 1; + clientWriteComplete(old->conn->fd, NULL, 0, COMM_OK, old); +} + +static void suppress_potentially_spool (clientHttpRequest *http, + char *data, int sz) +{ + char *b; + int l; + + if (!sz) return; + if (!http->suppress_writes) return; + + b = *(http->stack_spool); + l = *(http->stack_spool_len); + + if (!b) + b = xmalloc (l=sz); + else + { + l+=sz; + b = xrealloc (b,l); + } + *(http->stack_spool) = b; + *(http->stack_spool_len) = l; + + memcpy (b+l-sz,data,sz); + return; +} + + + +static int undo_template (char *ibuf, int ibufl, + char **obuf, int *obufl, void **d) +{ + template_t *template; + int rv = 0x00, zeroover=0,locsz; + clientHttpRequest *http; + request_t *REQUEST; + HttpHeader *hdr; + + template = *((template_t **) d); + + if (!template) /* already freed */ + return 0; + + printf ("undo %d %s\n", template->state,template->http->uri); + + + state_machine_start_01: + + switch (template->state) + { + + case 0: + template->e = storeGet(storeKeyPublic(template->uri,METHOD_GET)); + case 1: + if (template->state == 0) + { + template->chunk1 = xmalloc (ibufl); + template->chunk1l =ibufl; + memcpy (template->chunk1,ibuf,template->chunk1l); + } + else + if (ibufl) + { /* 1 */ + template->chunk1= xrealloc + (template->chunk1,template->chunk1l+ibufl); + memcpy (template->chunk1+template->chunk1l,ibuf,ibufl); + template->chunk1l += ibufl; + } + + + if (!template->e) /* NOT CACHED */ + { + REQUEST = urlParse(METHOD_GET, template->uri); + requestLink (REQUEST); + + http = xcalloc(1, sizeof(clientHttpRequest)); + cbdataAdd(http, wrappercbdataXfree, 0); + + http->conn = template->http->conn; + http->request = REQUEST; + + http->uri = xstrdup (template->uri); + http->log_uri = xstrndup (template->uri,MAX_URL); + + http->req_sz = 1; + + http->entry = template->e; + http->log_type = LOG_TCP_MISS; + http->http_ver = template->http->http_ver; + http->start = template->http->start; + http->range_iter.boundary = StringNull; + + http->suppress_writes = 1; + + http->stack_spool = &(template->http->local_spool); + http->stack_spool_len = &(template->http->local_spool_len); + + dlinkAdd(http, &http->active, &ClientActiveRequests); + rv = 0x08 | 0x10; + template->state = 50; + + http->stack = http->conn->chr; + http->conn->chr = http; + + hdr = &http->request->header; + + if (httpHeaderHas(hdr, HDR_IF_MODIFIED_SINCE)) + httpHeaderDelById(hdr, HDR_IF_MODIFIED_SINCE); + + httpHeaderDelById(hdr, HDR_IF_MATCH); + httpHeaderPutStr (hdr,HDR_IF_MATCH,template->etag); + + + clientInterpretRequestHeaders(http); + clientProcessMiss(http); + + break; + } + + template->buf = xmalloc (4096); + template->rpos = 0; + template->alcd = 4096; + + /* NO BREAK */ + + case 5: + storeLockObject(template->e); + storeCreateMemObject(template->e, template->uri, template->uri); + + storeClientListAdd(template->e, template); + storeClientCopy(template->e,0,0, template->alcd ,template->buf, + undotemplate_callback, template); + + rv = 0x08 | 0x10; + template->state = 15; + + break; + + case 15: + template->chunk1 = xrealloc (template->chunk1,template->chunk1l+ibufl); + memcpy (template->chunk1+template->chunk1l,ibuf,ibufl); + template->chunk1l += ibufl; + rv = 0x08 | 0x10; + + break; + + case 50: + /* this is purgatory.. a request hangs out here while squid is + retrieving the transfer-template in the response that + sadly wasn't in the cache */ + + template->chunk1 = xrealloc (template->chunk1,template->chunk1l+ibufl); + memcpy (template->chunk1+template->chunk1l,ibuf,ibufl); + template->chunk1l += ibufl; + + if (template->http->spool_complete) + { + HttpReply *rep; + + template->buf = template->http->local_spool; + template->rpos = template->alcd = template->http->local_spool_len; + template->state = 20; + template->http->oldte_rv &= 0xEF; + ibufl = 0; + + if ((rep = clientBuildReply(template->http, + template->buf, template->rpos))) + { + locsz = rep->hdr_sz; + httpReplyDestroy(rep); + } + + goto state_machine_start_01; + } + else + rv = 0x08 | 0x10; + break; + + + case 20: + printf ("clearing ONE %d:%d\n",template->http->conn->fd, + template->chunk1l); + + if (ibufl<1) + { + zeroover = 1; + *obuf = template->chunk1; + *obufl = template->chunk1l; + } + else + { + *obufl = template->chunk1l + ibufl; + *obuf = xmalloc(*obufl); + memcpy (*obuf,template->chunk1,template->chunk1l); + memcpy (*obuf+template->chunk1l,ibuf,ibufl); + xfree (template->chunk1); + } + template->chunk1 = NULL; + template->chunk1l = 0; + + ibuf = *obuf; + ibufl = *obufl; + rv |= 0x01; + + template->state = 25; /* no break! */ + patch_init (template, template->e ? + template->e->mem_obj->reply->hdr_sz : locsz); + + case 25: + + if (ibufl>0) + { + patch_chunk (template,ibuf,ibufl); + if (rv & 0x01) + xfree (ibuf); + *obufl =0; + *obuf = NULL; + if (!zeroover) + return 0x08; + } + template->state = 30; + patch_chunk_nomoreinput (template); + + case 30: + *obuf = xmalloc (4096); + rv = 0x01; + *obufl = patch_getchunk (template, *obuf, 4096); + if (!*obufl) + { + xfree (*obuf); + rv = 0x00; + patch_free (template); + template_free (template); + *d =NULL; + } + return rv; + break; + + + default: + case 60: + return identity (ibuf,ibufl,obuf,obufl,d) | rv; + break; + } + *obufl = 0; + return rv; +} + +static int undogzip (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + compression_t *m13; + char *tb; + int tbl, of,created; + + m13 = *((compression_t **) d); + if (!m13) + { + m13 = compression_init (COMPRESSION_GZIP,COMPRESSION_STD); + *d = (void *) m13; + } + + tbl = ibufl*2+512; + tb = xmalloc (tbl); + + created = 0; + if (ibuf) + { + do + { + created += compression_expand (m13,ibuf,ibufl,tb+created,tbl,&of); + if (of>0) + { + tbl = of+16; + tb = xrealloc (tb,created+tbl); + ibufl = 0; + } + } + while (of>0); + } + + else + { + do + { + created += compression_finish_expand (m13,tb+created,tbl,&of); + if (of>0) + { + tbl = of+16; + tb = xrealloc (tb,created+tbl); + } + } + while (of > 0); + + compression_free (m13,tb); + *d = NULL; + } + + + *obuf = tb; + *obufl = created; + + return 1; +} + + +/* this is a TE filter */ +static int dodeflate (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + compression_t *m13; + char *tb; + int tbl, of,created; + + m13 = *((compression_t **) d); + + if (!m13) + { + if (!ibufl) /* no input, nothing to finish.. */ + { + *obufl = 0; + return 0; + } + + m13 = compression_init (COMPRESSION_ZLIB,COMPRESSION_STD); + *d = (void *) m13; + } + + tbl = ibufl+512; + tb = xmalloc (tbl); + + if (ibufl > 0) + { + created = 0; + do + { + created += compression_shrink (m13,ibuf,ibufl,tb+created,tbl,&of); + if (of>0) + { + tbl = of+16; + tb = xrealloc (tb,created+tbl); + ibufl = 0; + } + } + while (of>0); + + } + else /* end signal */ + { + created = compression_free (m13,tb); + *d = NULL; + } + + + *obuf = tb; + *obufl = created; + + return 1; +} + +/* this is a TE filter */ +static int undodeflate(char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + compression_t *m13; + char *tb; + int tbl, of,created; + + m13 = *((compression_t **) d); + + if (!m13) + { + m13 = compression_init (COMPRESSION_ZLIB,COMPRESSION_STD); /* rfc 1950 */ + *d = (void *) m13; + } + + tbl = ibufl*2+512; + tb = xmalloc (tbl); + + created = 0; + if (ibuf) + { + do + { + created += compression_expand (m13,ibuf,ibufl,tb+created,tbl,&of); + if (of>0) + { + tbl = of+16; + tb = xrealloc (tb,created+tbl); + ibufl = 0; + } + } + while (of>0); + } + + else + { + do + { + created += compression_finish_expand (m13,tb+created,tbl,&of); + if (of>0) + { + tbl = of+16; + tb = xrealloc (tb,created+tbl); + } + } + while (of > 0); + + compression_free (m13,tb); + *d = NULL; + } + + + *obuf = tb; + *obufl = created; + + return 1; +} + +#endif + +static int hexvalue (char i) +{ + if ((i >= '0') && (i<='9')) + i= i - '0'; + else + i = toupper (i) - 'A' + 10; + return i; +} + + +/* this is a TE filter */ +static int dochunk (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + char *tb, *w; + int tbl, created, rv; + + tbl = ibufl+128; + tb = xmalloc (tbl); + rv = 0x00; + + if (ibufl > 0) + { + created = 0; + snprintf (tb,tbl,"%X\r\n", ibufl); + debug(33, 8) ("created chunk of %d\n",ibufl); + + w = tb + strlen (tb); + memcpy (w,ibuf,ibufl); + created = ibufl + (w-tb) + 2; + strcpy (w+ibufl,"\r\n"); + *d = (void *) 0x01; + } + else /* end signal */ + if (*d) + { + strcpy (tb,"0\r\n\r\n"); + created = 5; + debug(33,8) ("created chunk of %d\n",ibufl); + *d = NULL; + rv = 0x08; + } + else + { + *obuf = tb; + *obufl = 0; + return 0x02 | 0x01; /* signal that dochunk() is complete */ + } + + *obuf = tb; + *obufl = created; + + return rv | 0x01; +} + + +static int undochunked + (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + chunked_t *h; + char *tb, *w, *l; + int tbl, created,rv = 1,tr; + + h = *((chunked_t **) d); + + if (!h) + { + h = xmalloc (sizeof (chunked_t)); + h->state = 1; + *d = (void *) h; + } + + tbl = ibufl; + tb = xmalloc (tbl); + + created = 0; + w=ibuf; + l = ibuf+ibufl; + + state_machine_repeat: + + if (ibufl) + switch (h->state) + { + case 1: /* new chunk */ + h->pos = 0; + h->toread = 0; + h->half_crlf = 0; + + for (; (w < l) && !isxdigit (*w); w++); + if (w>=l) + break; + /* no default break! */ + h->state = 2; /* for 1 fallthru */ + + case 2: /* reading hexl */ + for (; (w < l) && isxdigit (*w); w++) + { + h->toread = h->toread << 4; + h->toread += hexvalue(*w); + } + if (w>=l) + break; + h->state = 3; + h->pos = h->toread; + debug(33,8) ("read chunk of %d\n",h->toread); + + + case 3: /* searching for CRLF */ + do + { + if (!h->half_crlf) + { + for (; (w < l) && (*w != '\r') ; w++); + if (w>=l) + break; + h->half_crlf = 1; + w++; + if (w>=l) + break; + } + if (*w != '\n') + h->half_crlf = 0; + else + { + h->state = 4; + } + w++; + if (w>=l) + break; + }while (h->state == 3); + + case 4: + /* gotta read h->toread bytes */ + if (h->toread) + { + tr = min (h->toread, l-w); + memcpy (tb+created, w, tr); + w += tr; + h->toread -= tr; + created += tr; + } + if (h->toread == 0) + { + h->state = 5; + h->half_crlf = 0; + } + if (w>=l) + break; + + case 5: /* seeking CRLF */ + do + { + if (!h->half_crlf) + { + for (; (w < l) && (*w != '\r') ; w++); + if (w>=l) + break; + h->half_crlf = 1; + w++; + if (w>=l) + break; + } + if (*(w++) != '\n') + h->half_crlf = 0; + else + { + debug(33,8) ("completed %d chunk\n",h->pos); + if (h->pos) + { + h->state = 1; + goto state_machine_repeat; + } + else + { + h->state =6; + h->half_crlf = 2; + } + } + + + + if (w>=l) + break; + }while (h->state == 5); + + case 6: + while (h->half_crlf < 4) + { + if (w>=l) + break; + if (!(h->half_crlf % 2)) + if (*w == '\r') + h->half_crlf++; + else + h->half_crlf =0; + else + if (*w == '\n') + h->half_crlf++; + else + h->half_crlf =0; + w++; + } + + /* chunked complete */ + h->state = 7; + rv = 0x04 | 0x01; /* signal that undochunk() is complete */ + + case 7: /* done, ignore */ + break; + } + else + { + xfree (h); + xfree (tb); + *d = NULL; + *obufl = 0; + return 0; + } + + + *obuf = tb; + *obufl = created; + + return rv; +} + + +/* this is a TE filter */ +int biff (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + /* like ROT13.. symmetrical mappings */ + return unbiff (ibuf,ibufl,obuf,obufl,d); +} + +/* this is a TE filter */ +int jones (char *ibuf, int ibufl, char **obuf, int *obufl, void **d) +{ + /* like ROT13.. symmetrical mappings */ + return unjones (ibuf,ibufl,obuf,obufl,d); +} + + +void new_xlat (clientHttpRequest *http, int beg_or_end, + int (*func)(char *, int , char **, int *, void **), + void *data) +{ + TE_list *tp,*tx; + + tp = xmalloc (sizeof (TE_list)); + tp->func = func; + tp->data = data; + + if (beg_or_end == 0) + { /* beginning */ + tp->next = http->te_translations; + http->te_translations = tp; + } + else + { + /* end */ + tp->next = NULL; + for (tx = http->te_translations; tx && tx->next; tx =tx->next); + if (!tx) /* root node */ + http->te_translations = tp; + else + tx->next = tp; + } + + return; +} + + /* * filters out unwanted entries from original reply header * adds extra entries if we have more info than origin server @@ -1126,26 +2107,239 @@ static void clientBuildReplyHeader(clientHttpRequest * http, HttpReply * rep) { - HttpHeader *hdr = &rep->header; - int is_hit = isTcpHit(http->log_type); - request_t *request = http->request; + String s_transfer_encoding, s_te, s_ce; + HttpHeader *hdr = &rep->header; + int is_hit = isTcpHit(http->log_type); + request_t *request = http->request; + const char *pos = NULL; + const char *item; + int y, may_apply_tes; + HttpHeader *ohdr; + const char *s_ct; + char added_te =0; + + ohdr = &request->header; + #if DONT_FILTER_THESE - /* but you might want to if you run Squid as an HTTP accelerator */ - /* httpHeaderDelById(hdr, HDR_ACCEPT_RANGES); */ - httpHeaderDelById(hdr, HDR_ETAG); -#endif - httpHeaderDelById(hdr, HDR_PROXY_CONNECTION); - /* here: Keep-Alive is a field-name, not a connection directive! */ - httpHeaderDelByName(hdr, "Keep-Alive"); - /* remove Set-Cookie if a hit */ - if (is_hit) - httpHeaderDelById(hdr, HDR_SET_COOKIE); + /* but you might want to if you run Squid as an HTTP accelerator */ + /* httpHeaderDelById(hdr, HDR_ACCEPT_RANGES); */ + httpHeaderDelById(hdr, HDR_ETAG); +#endif + httpHeaderDelById(hdr, HDR_PROXY_CONNECTION); + /* here: Keep-Alive is a field-name, not a connection directive! */ + httpHeaderDelByName(hdr, "Keep-Alive"); + /* remove Set-Cookie if a hit */ + if (is_hit) + httpHeaderDelById(hdr, HDR_SET_COOKIE); + + may_apply_tes = 0; + + if (httpHeaderHas (hdr,HDR_TRANSFER_ENCODING)) + { + s_transfer_encoding = httpHeaderGetList (hdr,HDR_TRANSFER_ENCODING); + + /* REMOVE ALL TRANSFER ENCODINGS */ + /* order is impt here.. want those entries at end of transfer-encoding + list to be undone first and they are listed in the order they are + applied .. TE's are HOP BY HOP */ + + pos = NULL; + while (strListGetItem(&s_transfer_encoding, ',', &item, &y, &pos)) + if (!strncasecmp(item, "biff",y)) + { + /* need to remove the TE.. and rewrite the header */ + debug(33,7) ("removing biff\n"); + new_xlat (http,0,unbiff,NULL); + } + else + if (!strncasecmp(item, "jones",y)) + { + debug(33,7) ("removing jones\n"); + new_xlat (http,0,unjones,NULL); + } +#ifdef ATHY_COMPRESSION + else + if (!strncasecmp(item, "gzip",y)) + { + debug(33,7) ("removing gzip\n"); + new_xlat (http,0,undogzip,NULL); + } + else + if (!strncasecmp(item, "deflate",y)) + { + debug(33,7) ("removing deflate\n"); + new_xlat (http,0,undodeflate,NULL); + } +#endif + else + if (!strncasecmp(item, "chunked",y)) + { + debug(33,7) ("removing chunked\n"); + new_xlat (http,0,undochunked,NULL); + } + else + if ((!strncasecmp(item, "template",y)) && + httpHeaderHas (hdr,HDR_TRANSFER_TEMPLATE)) + { + const char *s_tt; + template_t *temp; + char *et; + + s_tt = httpHeaderGetStr (hdr,HDR_TRANSFER_TEMPLATE); + debug(33,7) ("removing template: %s\n",s_tt); + temp= xmalloc (sizeof (template_t)); + + temp->state = 0; + temp->patch = NULL; + temp->uri = xstrdup (s_tt); + et = temp->uri; + if (xisspace (*et)) + for (;xisspace(*et) && *et;et++); + for (;(!xisspace(*et)) && *et;et++); + if (*et) + *(et++) = '\0'; + for (;xisspace(*et) && *et;et++); + temp->etag = et; + + temp->http = http; + new_xlat (http,0,undo_template,temp); + /* delheader should be redundant.. but just in case */ + httpHeaderDelById (hdr, HDR_TRANSFER_TEMPLATE); + } + + /* transfer-encoding should be cleared by it's inclusion in + the connection header.. but just in case */ + + httpHeaderDelById (hdr, HDR_TRANSFER_ENCODING); + stringClean(&s_transfer_encoding); + } + + if (httpHeaderHas (ohdr,HDR_TE) && + httpHeaderHas (hdr,HDR_CONTENT_TYPE) && + /* we do this length check to not compress payloads that will + likely fit in one packet.. though we don't really know what + MTU is.. this should also screen out simple 404's and redirects, + without breaking the algorithm if those responses do have big bodies + which they might*/ + ((rep->content_length > 1200) || (rep->content_length <0))) + + { + s_ct = httpHeaderGetStr (hdr,HDR_CONTENT_TYPE); + + y = strlen (s_ct); + if (((y>=5) && !strncasecmp(s_ct,"text/",5)) + || strstr (s_ct,"postscript")) + { + may_apply_tes = 1; + debug(33,8) ("Content-Type of Application: %s\n",s_ct); + if (httpHeaderHas (hdr,HDR_CONTENT_ENCODING)) + { + /* oh, base entity might be compressed.. don't want + to double *that* */ + s_ce = httpHeaderGetList (hdr,HDR_CONTENT_ENCODING); + + if ((strListIsMember(&s_ce, "gzip", ',')) || + (strListIsMember(&s_ce, "deflate", ',')) || + (strListIsMember(&s_ce, "x-deflate", ',')) || + (strListIsMember(&s_ce, "x-gzip", ',')) || + (strListIsMember(&s_ce, "x-compress", ',')) || + (strListIsMember(&s_ce, "compress", ',')) ) + may_apply_tes |= 0x02; /* 02 is the compress exception */ + + stringClean(&s_ce); + } + } + } + + + /* ADD NEW TRANSFER ENCODINGS.. check announcements */ + if (may_apply_tes && httpHeaderHas (ohdr,HDR_TE)) + { + char vlb[128]; + + vlb[0] = '\0'; + + s_te = httpHeaderGetList (ohdr,HDR_TE); + + if ((strListIsMember_q(&s_te, "chunked", ',')) || + (request->http_ver > 1.099)) + { + /* chunked is known.. we can add te's. */ + + if (strListIsMember_q(&s_te, "biff", ',')) + { + debug(33,7) ("will add biff\n"); + new_xlat (http,1,biff,NULL); + + if (added_te) + strcat (vlb,", "); + else + added_te = 1; + strcat (vlb, "biff"); + } + + if (strListIsMember_q(&s_te, "jones", ',')) + { + debug(33,7) ("will add jones\n"); + new_xlat (http,1,jones,NULL); + + if (added_te) + strcat (vlb,", "); + else + added_te = 1; + strcat (vlb, "jones"); + } + + +#ifdef ATHY_COMPRESSION + if ((strListIsMember_q(&s_te, "deflate", ',')) + && (!(may_apply_tes & 0x02))) /* the ce: compressed */ + { + debug(33,7) ("will add deflate\n"); + new_xlat (http,1,dodeflate,NULL); + + if (added_te) + strcat (vlb,", "); + else + added_te = 1; + strcat (vlb, "deflate"); + } + else /* ONLY IF NOT DEFLATE */ + if ((strListIsMember_q(&s_te, "gzip", ',')) + && (!(may_apply_tes & 0x02))) /* the ce: compressed */ + { + debug(33,7) ("will add gzip\n"); + new_xlat (http,1,dogzip,NULL); + + if (added_te) + strcat (vlb,", "); + else + added_te = 1; + strcat (vlb, "gzip"); + } +#endif + + if (added_te) + { + debug(33,7) ("will add chunked\n"); + new_xlat (http,1,dochunk,NULL); + strcat (vlb,", chunked"); + httpHeaderPutStr(hdr, HDR_TRANSFER_ENCODING, vlb); + httpHeaderDelById (hdr, HDR_CONTENT_LENGTH); + } + + } + stringClean(&s_te); + } + + + /* handle Connection header */ if (httpHeaderHas(hdr, HDR_CONNECTION)) { - /* anything that matches Connection list member will be deleted */ - String strConnection = httpHeaderGetList(hdr, HDR_CONNECTION); - const HttpHeaderEntry *e; - HttpHeaderPos pos = HttpHeaderInitPos; + /* anything that matches Connection list member will be deleted */ + String strConnection = httpHeaderGetList(hdr, HDR_CONNECTION); + const HttpHeaderEntry *e; + HttpHeaderPos pos = HttpHeaderInitPos; /* * think: on-average-best nesting of the two loops (hdrEntry * and strListItem) @?@ @@ -1155,6 +2349,7 @@ * from strConnection first? */ while ((e = httpHeaderGetEntry(hdr, &pos))) { + if (strListIsMember(&strConnection, strBuf(e->name), ',')) httpHeaderDelAt(hdr, pos); } @@ -1197,6 +2392,10 @@ httpHeaderPutStr(hdr, http->flags.accel ? HDR_CONNECTION : HDR_PROXY_CONNECTION, request->flags.proxy_keepalive ? "keep-alive" : "close"); + + if (added_te) + httpHeaderPutStr(hdr,HDR_CONNECTION, "Transfer-Encoding"); + #if ADD_X_REQUEST_URI /* * Knowing the URI of the request is useful when debugging persistent @@ -1205,7 +2404,7 @@ * debugger [hdr->entries.count-1]. */ httpHeaderPutStr(hdr, HDR_X_REQUEST_URI, - http->entry->mem_obj->url ? http->entry->mem_obj->url : http->uri); + http->entry->mem_obj->url ? http->entry->mem_obj->url : http->uri); #endif } @@ -1216,6 +2415,7 @@ if (httpReplyParse(rep, buf)) { /* enforce 1.0 reply version */ rep->sline.version = 1.0; + /* do header conversions */ clientBuildReplyHeader(http, rep); /* if we do ranges, change status to "Partial Content" */ @@ -1548,6 +2748,49 @@ return 0; } + +static int perform_te (TE_list *xlat, char *ibuf, int ibufl, + char **obuf, int *obufl) +{ + int rv = 0, made_new, orig_len; + + *obuf = ibuf; + *obufl = ibufl; + orig_len = ibufl; + + made_new = 0; + + while (xlat) + { + /* xlat func is likely to be unbiff or dogzip() or something */ + + rv &= 0xFE; /* turn off low bit, for reuse, while + preserving chunk indicators */ + + rv |= (*(xlat->func)) ((char *)ibuf, ibufl, obuf, obufl,&(xlat->data)); + + /* FREE SOME STUFF HERE if (rv & 0x01) */ + if ((made_new) && (rv & 0x01)) + xfree (ibuf); + + made_new = made_new | (rv & 0x01); + + ibuf = *obuf; + ibufl = *obufl; + xlat = xlat->next; + } + + if ((rv & 0x08) /* a 0 length chunk was created */ + && (orig_len > 0)) /* and there was actually input */ + { + *obufl = 0; /* don't write the chunk, as it means EOT */ + debug(33,8) ("EOT PREVENTION\n"); + } + + + return rv | made_new; +} + /* * accepts chunk of a http message in buf, parses prefix, filters headers and * such, writes processed message to the client's socket @@ -1561,17 +2804,24 @@ int fd = conn->fd; HttpReply *rep = NULL; const char *body_buf = buf; - ssize_t body_size = size; + char *te_body_buf; + ssize_t body_size = size, te_body_size; + int free_te; MemBuf mb; ssize_t check_size = 0; debug(33, 5) ("clientSendMoreData: %s, %d bytes\n", http->uri, (int) size); + printf ("clientSendMoreData: %s, %d bytes\n", http->uri, (int) size); + + suppress_potentially_spool (http,buf,size); + + free_te = 0x00; assert(size <= CLIENT_SOCK_SZ); assert(http->request != NULL); dlinkDelete(&http->active, &ClientActiveRequests); dlinkAdd(http, &http->active, &ClientActiveRequests); debug(33, 5) ("clientSendMoreData: FD %d '%s', out.offset=%d \n", fd, storeUrl(entry), (int) http->out.offset); - if (conn->chr != http) { + if ((conn->chr != http) && ! http->suppress_writes) { /* there is another object in progress, defer this one */ debug(33, 1) ("clientSendMoreData: Deferring %s\n", storeUrl(entry)); memFree(buf, MEM_CLIENT_SOCK_BUF); @@ -1653,17 +2903,21 @@ assert(rep || (body_buf && body_size)); /* init mb; put status line and headers if any */ if (rep) { + if (!http->suppress_writes) mb = httpReplyPack(rep); - http->out.offset += rep->hdr_sz; - check_size += rep->hdr_sz; - httpReplyDestroy(rep); - rep = NULL; + else + memBufDefInit(&mb); + http->out.offset += rep->hdr_sz; + check_size += rep->hdr_sz; + httpReplyDestroy(rep); + rep = NULL; } else { memBufDefInit(&mb); } + /* append body if any */ if (http->request->range) { - /* Only GET requests should have ranges */ + /* Only GET requests should have ranges */ assert(http->request->method == METHOD_GET); /* clientPackMoreRanges() updates http->out.offset */ /* force the end of the transfer if we are done */ @@ -1672,7 +2926,26 @@ } else if (body_buf && body_size) { http->out.offset += body_size; check_size += body_size; - memBufAppend(&mb, body_buf, body_size); + if (body_size) + { + http->oldte_rv = free_te = + perform_te (http->te_translations,(char *)body_buf, + body_size, &te_body_buf, &te_body_size); + + + if (te_body_size != body_size) + debug(33,1) ("%s:%d:%d\n",http->log_uri, body_size,te_body_size); + if ((te_body_size) && !http->suppress_writes) + memBufAppend(&mb, te_body_buf, te_body_size); + if ((te_body_size) && http->suppress_writes) + http->oldte_rv |= 0x08; + } + + if (free_te & 0x01) + xfree (te_body_buf); + + if (free_te & 0x04) + http->flags.done_copying = 1; } if (!http->request->range && http->request->method == METHOD_GET) assert(check_size == size); @@ -1682,6 +2955,8 @@ memFree(buf, MEM_CLIENT_SOCK_BUF); } + + static void clientKeepaliveNextRequest(clientHttpRequest * http) { @@ -1730,7 +3005,12 @@ { clientHttpRequest *http = data; StoreEntry *entry = http->entry; - int done; + int done, stuckdatasize,free_te,datadone; + char *stuck_data; + MemBuf mb; + + datadone=0; + stuckdatasize = 0; http->out.size += size; debug(33, 5) ("clientWriteComplete: FD %d, sz %d, err %d, off %d, len %d\n", fd, size, errflag, (int) http->out.offset, entry ? objectLen(entry) : 0); @@ -1748,41 +3028,90 @@ comm_close(fd); /* yuk */ } else if (EBIT_TEST(entry->flags, ENTRY_ABORTED)) { comm_close(fd); - } else if ((done = clientCheckTransferDone(http)) != 0 || size == 0) { + } else if (((done = clientCheckTransferDone(http)) != 0 || + ((size == 0) && (!(http->oldte_rv & 0x08)))) && + (!(http->oldte_rv & 0x10))) + + { debug(33, 5) ("clientWriteComplete: FD %d transfer is DONE\n", fd); /* We're finished case */ + + http->oldte_rv = free_te = perform_te (http->te_translations,NULL,0, + &stuck_data, &stuckdatasize); + if (stuckdatasize > 0) + { + debug(33,1) ("%s:0:%d\n",http->log_uri,stuckdatasize); + + suppress_potentially_spool (http,stuck_data, stuckdatasize); + + memBufDefInit(&mb); + if (!http->suppress_writes) + memBufAppend(&mb, stuck_data, stuckdatasize); + else + http->oldte_rv |= 0x08; + + comm_write_mbuf(fd, mb, clientWriteComplete, http); + + /* don't need this, as write may happen async, it's dangerous + too.. plus, it is done after the write but the hook function + within it */ + /* memBufClean(&mb); */ + + + if (free_te & 0x01) + xfree (stuck_data); + return; + } + if (free_te & 0x02) /* signal from dochunk() that + data has been chunked */ + { + http->flags.done_copying = 1; + } + if (http->entry->mem_obj->reply->content_length < 0) { - debug(33, 5) ("clientWriteComplete: closing, content_length < 0\n"); - comm_close(fd); + debug(33, 5) ("clientWriteComplete: closing, content_length < 0\n"); + if (!http->suppress_writes) comm_close(fd); + datadone=1; } else if (!done) { - debug(33, 5) ("clientWriteComplete: closing, !done\n"); - comm_close(fd); + debug(33, 5) ("clientWriteComplete: closing, !done\n"); + if (!http->suppress_writes) comm_close(fd); + datadone=1; } else if (clientGotNotEnough(http)) { - debug(33, 5) ("clientWriteComplete: client didn't get all it expected\n"); - comm_close(fd); + debug(33, 5) ("clientWriteComplete: client didn't get all it expected\n"); + if (!http->suppress_writes) comm_close(fd); + datadone=1; } else if (http->request->flags.proxy_keepalive) { - debug(33, 5) ("clientWriteComplete: FD %d Keeping Alive\n", fd); - clientKeepaliveNextRequest(http); + debug(33, 5) ("clientWriteComplete: FD %d Keeping Alive\n", fd); + if (!http->suppress_writes) clientKeepaliveNextRequest(http); + datadone=1; } else { - comm_close(fd); + if (!http->suppress_writes) comm_close(fd); + datadone=1; } - } else if (clientReplyBodyTooLarge((int) http->out.offset)) { - comm_close(fd); - } else { + } else if (clientReplyBodyTooLarge((int) http->out.offset)) { + if (!http->suppress_writes) comm_close(fd); + datadone=1; + } else { /* More data will be coming from primary server; register with - * storage manager. */ + * storage manager. */ + http->oldte_rv &= 0xF7; + if (EBIT_TEST(entry->flags, ENTRY_ABORTED)) - debug(33, 0) ("clientWriteComplete 2: ENTRY_ABORTED\n"); + debug(33, 0) ("clientWriteComplete 2: ENTRY_ABORTED\n"); storeClientCopy(entry, - http->out.offset, - http->out.offset, - CLIENT_SOCK_SZ, - memAllocate(MEM_CLIENT_SOCK_BUF), - clientSendMoreData, - http); - } + http->out.offset, + http->out.offset, + CLIENT_SOCK_SZ, + memAllocate(MEM_CLIENT_SOCK_BUF), + clientSendMoreData, + http); + } + + if ((datadone) && http->suppress_writes) + httpRequestFree(http); } + /* * client issued a request with an only-if-cached cache-control directive; * we did not find a cached object that can be returned without @@ -1954,8 +3283,7 @@ /* * Prepare to fetch the object as it's a cache miss of some kind. */ -static void -clientProcessMiss(clientHttpRequest * http) +static void clientProcessMiss(clientHttpRequest * http) { char *url = http->uri; request_t *r = http->request; @@ -2006,6 +3334,7 @@ } if (http->flags.internal) r->protocol = PROTO_INTERNAL; + fwdStart(http->conn->fd, http->entry, r, http->conn->peer.sin_addr, http->conn->me.sin_addr); } @@ -2620,6 +3949,7 @@ #define SENDING_BODY 0 #define SENDING_HDRSONLY 1 +/* checks status of sending to original client */ static int clientCheckTransferDone(clientHttpRequest * http) { @@ -2641,10 +3971,10 @@ * objectLen(entry) will be set proprely. */ if (entry->store_status == STORE_OK) { - if (http->out.offset >= objectLen(entry)) - return 1; - else - return 0; + if (http->out.offset >= objectLen(entry)) + return 1; + else + return 0; } /* * Now, handle STORE_PENDING objects Index: src/comm.c =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/comm.c,v retrieving revision 1.1.1.1 retrieving revision 1.3 diff -c -u -r1.1.1.1 -r1.3 --- src/comm.c 1999/08/10 14:36:41 1.1.1.1 +++ src/comm.c 1999/08/26 18:47:25 1.3 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * DEBUG: section 5 Socket Functions * AUTHOR: Harvest Derived @@ -514,6 +514,7 @@ fde *F = &fd_table[fd]; close_handler *ch; debug(5, 5) ("commCallCloseHandlers: FD %d\n", fd); + while ((ch = F->close_handler) != NULL) { F->close_handler = ch->next; debug(5, 5) ("commCallCloseHandlers: ch->handler=%p\n", ch->handler); @@ -643,7 +644,8 @@ close_handler *new = xmalloc(sizeof(*new)); close_handler *c; debug(5, 5) ("comm_add_close_handler: FD %d, handler=%p, data=%p\n", - fd, handler, data); + fd, handler, data); + for (c = fd_table[fd].close_handler; c; c = c->next) assert(c->handler != handler || c->data != data); new->handler = handler; Index: src/dns_internal.c =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/dns_internal.c,v retrieving revision 1.1.1.1 retrieving revision 1.2 diff -c -u -r1.1.1.1 -r1.2 --- src/dns_internal.c 1999/08/10 14:36:44 1.1.1.1 +++ src/dns_internal.c 1999/08/10 17:04:38 1.2 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * DEBUG: section 78 DNS lookups; interacts with lib/rfc1035.c * AUTHOR: Duane Wessels @@ -116,12 +116,12 @@ nns = nns_alloc = 0; } -static void -idnsParseResolvConf(void) +static void idnsParseResolvConf(void) { FILE *fp; char buf[512]; char *t; + fp = fopen(_PATH_RESOLV_CONF, "r"); if (fp == NULL) { debug(78, 1) ("%s: %s\n", _PATH_RESOLV_CONF, xstrerror()); @@ -130,11 +130,11 @@ idnsFreeNameservers(); while (fgets(buf, 512, fp)) { t = strtok(buf, w_space); - if (strcasecmp(t, "nameserver")) + if ((!t) ||(strcasecmp(t, "nameserver"))) continue; t = strtok(NULL, w_space); if (t == NULL) - continue;; + continue; debug(78, 1) ("idnsParseResolvConf: nameserver %s\n", t); idnsAddNameserver(t); } Index: src/enums.h =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/enums.h,v retrieving revision 1.1.1.1 retrieving revision 1.3 diff -c -u -r1.1.1.1 -r1.3 --- src/enums.h 1999/08/10 14:36:44 1.1.1.1 +++ src/enums.h 1999/08/20 16:35:39 1.3 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * * SQUID Internet Object Cache http://squid.nlanr.net/Squid/ @@ -220,7 +220,10 @@ HDR_RETRY_AFTER, HDR_SERVER, HDR_SET_COOKIE, + HDR_TE, HDR_TITLE, + HDR_TRANSFER_ENCODING, + HDR_TRANSFER_TEMPLATE, HDR_UPGRADE, HDR_USER_AGENT, HDR_VARY, Index: src/http.c =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/http.c,v retrieving revision 1.1.1.1 retrieving revision 1.12 diff -c -u -r1.1.1.1 -r1.12 --- src/http.c 1999/08/10 14:36:51 1.1.1.1 +++ src/http.c 1999/08/24 14:30:07 1.12 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * DEBUG: section 11 Hypertext Transfer Protocol (HTTP) * AUTHOR: Harvest Derived @@ -39,6 +39,7 @@ */ #include "squid.h" +#define ATHY_COMPRESSION static const char *const crlf = "\r\n"; @@ -449,8 +450,10 @@ read_sz = delayBytesWanted(delay_id, 1, read_sz); #endif Counter.syscalls.sock.reads++; + len = read(fd, buf, read_sz); debug(11, 5) ("httpReadReply: FD %d: len %d.\n", fd, len); + if (len > 0) { fd_bytes(fd, len, FD_READ); #if DELAY_POOLS @@ -466,11 +469,11 @@ } if (!httpState->reply_hdr && len > 0) { /* Skip whitespace */ - while (len > 0 && xisspace(*buf)) + while (len > 0 && xisspace(*buf)) xmemmove(buf, buf + 1, len--); if (len == 0) { /* Continue to read... */ - commSetSelect(fd, COMM_SELECT_READ, httpReadReply, httpState, 0); + commSetSelect(fd, COMM_SELECT_READ, httpReadReply, httpState, 0); return; } } @@ -531,7 +534,7 @@ (void) 0; } else if (httpPconnTransferDone(httpState)) { /* yes we have to clear all these! */ - commSetDefer(fd, NULL, NULL); + commSetDefer(fd, NULL, NULL); commSetTimeout(fd, -1, NULL, NULL); commSetSelect(fd, COMM_SELECT_READ, NULL, NULL, 0); #if DELAY_POOLS @@ -693,6 +696,24 @@ if (Config.fake_ua && !httpHeaderHas(hdr_out, HDR_USER_AGENT)) httpHeaderPutStr(hdr_out, HDR_USER_AGENT, Config.fake_ua); + /* append TE */ + { + char strTE[128]; + + httpHeaderDelById(hdr_out, HDR_TE); /* hop by hop.. that's what last + hop could do*/ +#ifdef ATHY_COMPRESSION + strcpy (strTE,"biff;q=0.0, gzip;q=1.0, deflate;q=1.0, jones;q=0.0, chunked;q=1.0, template;q=1.0"); +#else + strcpy (strTE,"biff;q=1.0, gzip;q=0.0, deflate;q=0.0, jones;q=1.0, chunked;q=1.0"); +#endif + + httpHeaderPutStr (hdr_out,HDR_TE,strTE); + httpHeaderPutStr(hdr_out,HDR_CONNECTION, "TE"); /* its hop by hop */ + + } + + /* append Via */ { String strVia = httpHeaderGetList(hdr_in, HDR_VIA); Index: src/patch.c =================================================================== RCS file: patch.c diff -N patch.c --- /dev/null Thu Aug 24 05:00:32 2000 +++ /tmp/cvsELMn9S Thu Jan 25 15:48:06 2001 @@ -0,0 +1,4 @@ +#include +#inlcude + +/* COPYRIGHT APPLIEDTHEORY COMMUNICATIONS */ Index: src/protos.h =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/protos.h,v retrieving revision 1.1.1.1 retrieving revision 1.2 diff -c -u -r1.1.1.1 -r1.2 --- src/protos.h 1999/08/10 14:37:02 1.1.1.1 +++ src/protos.h 1999/08/11 19:23:16 1.2 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * * SQUID Internet Object Cache http://squid.nlanr.net/Squid/ @@ -378,6 +378,7 @@ extern void httpHeaderAddContRange(HttpHeader * hdr, HttpHdrRangeSpec spec, size_t ent_len); extern void strListAdd(String * str, const char *item, char del); extern int strListIsMember(const String * str, const char *item, char del); +extern int strListIsMember_q (const String * list, const char *m, char del); extern int strListIsSubstr(const String * list, const char *s, char del); extern int strListGetItem(const String * str, char del, const char **item, int *ilen, const char **pos); extern const char *getStringPrefix(const char *str, const char *end); Index: src/structs.h =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/structs.h,v retrieving revision 1.1.1.1 retrieving revision 1.9 diff -c -u -r1.1.1.1 -r1.9 --- src/structs.h 1999/08/10 14:37:15 1.1.1.1 +++ src/structs.h 1999/08/27 15:47:58 1.9 @@ -1,6 +1,6 @@ /* - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * * SQUID Internet Object Cache http://squid.nlanr.net/Squid/ @@ -804,6 +804,16 @@ HierarchyLogEntry hier; }; + +typedef struct _TE_list +{ + int (*func)(char *ibuf, int ibufl, char **obuf, int *obufl, void **data); + void *data; + struct _TE_list *next; +} +TE_list; + + struct _clientHttpRequest { ConnStateData *conn; request_t *request; /* Parsed URL ... */ @@ -837,7 +847,15 @@ http_status status; char *location; } redirect; - dlink_node active; + dlink_node active; + + TE_list *te_translations; + int oldte_rv; + char suppress_writes, spool_complete; + clientHttpRequest *stack; /* ok, it's a one level stack.. but hey */ + char **stack_spool,*local_spool; /* a place to spool the transfer-template + lookup */ + int *stack_spool_len, local_spool_len; }; struct _ConnStateData { @@ -1785,3 +1803,39 @@ int bad_log_op; int zero_object_sz; }; + +typedef struct chunked_tag +{ + int state; /* 1 - in between */ + int pos; /* 2 - reading hexl */ + int toread; /* 3 - seeking CRLF from hexl */ + int half_crlf; /* 4 - reading data */ + /* 5 - seeking CRLF from data */ + /* 6 - looking for double CRLF */ +} chunked_t; + +typedef struct patch_tag +{ + FILE *f; + int readoff; + char *fn; +} +patch_t; + + +typedef struct template_tag +{ + char state; + char *uri,*buf,*chunk1,*etag; + StoreEntry *e; + clientHttpRequest *http; + int rpos,alcd,chunk1l; + patch_t *patch; + + /* 0 - uninited + 10 - has uri from cache + 20 - fetching uri + 30 - performing translation + 40 - complete */ +} +template_t; Index: src/urn.c =================================================================== RCS file: /var/cvsroot/mcmanus/squid-2.3/src/urn.c,v retrieving revision 1.1.1.1 retrieving revision 1.2 diff -c -u -r1.1.1.1 -r1.2 --- src/urn.c 1999/08/10 14:37:20 1.1.1.1 +++ src/urn.c 1999/08/11 15:12:29 1.2 @@ -1,7 +1,7 @@ /* * - * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ + * $Id: squid-te-patches,v 1.1.1.1 2002/04/09 19:04:55 mcmanus Exp $ * * DEBUG: section 52 URN Parsing * AUTHOR: Kostas Anagnostakis @@ -338,3 +338,14 @@ debug(52, 3) ("urnParseReply: Found %d URLs\n", i); return list; } + + + + + + + + + + +