If you are looking for the epic motorcycle journey blog that I've written, please see the Miles By Motorcycle site I put together. 
  • Ian
    Debugging a Website with Telnet
    06/08/2013 2:15AM

    Telnet and modern-day websites aren't often considered together, but that's how I recently solved a problem with a server I now work with.  To make matters a little extra complicated, the server runs the Solaris 10 operating system and I do not have 'root' access to it.

     Previously, this server was just acting as a reverse proxy and handling traffic to and from Glassfish Java application servers, and there was no problem with that.  However, when I installed AWstats to provide web stats reporting functionality, I ran into a problem.  It just didn't work.  Attempts to load the AWstats pages in a browser resulted in nothing, as if the browser had not been asked to do anything.  No browser console error messages, nothing.  Firefox, IE, Chrome, even 'wget', all the same.  Finally, using the text-based browser 'elinks' from my home machine I got a "bad headers" message.  Aha!  I tried the "Live HTTP Headers" plugin for Firefox, which showed me... nothing.

     I could see in the logs on the web server that it was generating a 200 ("OK") response and could even see the size of the output in the logs, but got nothing at the browser.  Most peculiar.  Eventually I tried telnet.

    So you use telnet to connect to a server on port 80 and then manually make requests, just the way a browser does, and it will happily show you the otherwise invisible headers that the browser receives, along with all of the "normal" data. After trying this a few times I realized that for any request where there was a "Content-Length" header, the output was fine, but for any request where the content did not have the length specified in the header, but rather used "chunked" transfer encoding, it would fail.

    The Apache web server can only issue a Content-Length header when it knows how long the content is that it is serving up.  If the content is generated by a script (dynamically), then it can't know exactly how much data will be output until after the headers have been sent, so it breaks the output into "chunks", and before each chunk it tells the browser how big the next chunk is, so it will know if it has received the whole thing.  The chunk size is reported using a hexadecimal value and is readily visible in the telnet output. MY server was not printing the chunk size as hexadecimal; it came out as "%lx" and that is what was causing the "bad headers".

    I've got bigger fish to fry, so I haven't researched this in the greatest detail.  Suffice it to say "Apache was not compiled properly on this machine."  I recompiled Apache and the problem was gone.

     It's been a while since I worked on servers where we compiled software like Apache instead of installing pre-compiled packages, but it's interesting in that it takes me back to the old days of internet server management.

    For the curious, here's how to make a query to a web server using telnet. I've bolded the user input, the transfer encoding line, and the hexadecimal chunk size.  Note that you must hit Enter one more time after the "Connection:" line before the web server will respond.

     $ telnet yml.com 80
    Trying 108.18.135.196...
    Connected to yml.com.
    Escape character is '^]'.
    GET /homepage.html HTTP/1.1
    Host: yml.com
    Connection: close


    HTTP/1.1 200 OK
    Date: Sat, 08 Jun 2013 06:00:18 GMT
    Server: Apache/2.2.3 (CentOS)
    X-Powered-By: PHP/5.3.5
    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
    P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
    Set-Cookie: f261cfbdfb9f612=gs758ac6uentfva7233o5p9lc3; path=/; domain=yml.com
    Expires: Thu, 19 Nov 1981 08:52:00 GMT
    Pragma: no-cache
    Vary: Accept-Encoding,User-Agent
    Connection: close
    Transfer-Encoding: chunked
    Content-Type: text/html; charset=iso-8859-1

    2203
    html output follows...

  • Comment By:
    Yermo
    06/08/2013 12:03PM
    Wow. I have never run across an improperly compiled apache before. That's some good detective work. %lx is the printf() specification for a long hexadecimal so clearly something is very very broken.
  • Comment By:
    Ian
    06/08/2013 3:03PM
    I looked over the document that was made, along the lines of "how to install Apache on our servers", which I believe was prepared by a co-op student. There appeared to be some questionable instructions in it, such as using a combination of GNU and Solaris tools to compile it. I don't know exactly what was done. Anyway, it was unusual; Googling a variety of combinations of terms, I only found one other reference to this problem.
You must be logged into an account to post comments.