This file documents the URL loading package.

Copyright (C) 1996, 1997, 1998, 1999, 2002, 2004  Free Software Foundation Copyright (C) 1993, 1994, 1995, 1996  William M. Perry

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with the Invariant Sections being ``GNU GENERAL PUBLIC LICENSE''.  A copy of the license is included in the section entitled ``GNU Free Documentation License.''

URL
***

Getting Started
***************

Uniform Resource Locators (URLs) are a specific form of Uniform Resource Identifiers (URI) described in RFC 2396 which updates RFC 1738 and RFC 1808.  RFC 2016 defines uniform resource agents.

URIs have the form SCHEME:SCHEME-SPECIFIC-PART, where the SCHEMEs supported by this library are described below. See Supported URL Types.

FTP NFS, HTTP, HTTPS, rlogin, telnet, tn3270, IRC and gopher URLs all have the form

     SCHEME://[USERINFO@]HOSTNAME[:PORT][/PATH]

where [ and ] delimit optional parts. USERINFO sometimes takes the form USERNAME:PASSWORD but you should beware of the security risks of sending cleartext passwords.  HOSTNAME may be a domain name or a dotted decimal address.  If the :PORT is omitted then the library will use the `well known' port for that service when accessing URLs.  With the possible exception of telnet, it is rare for ports to be specified, and it is possible using a non-standard port may have undesired consequences if a different service is listening on that port (e.g., an HTTP URL specifying the SMTP port can cause mail to be sent)., but see url-bad-port-list. The meaning of the PATH component depends on the service.

Configuration
=============

 -- Variable: url-configuration-directory
     The directory in which URL configuration files, the cache etc., reside.  Default ~/.url.

Parsed URLs
===========

The library functions typically operate on parsed versions of URLs.  These are actually vectors of the form:

     [TYPE USER PASSWORD HOST PORT FILE TARGET ATTRIBUTES FULL]

where

type
     is the type of the URL scheme, e.g., http

user
     is the username associated with it, or nil;

password
     is the user password associated with it, or nil;

host
     is the host name associated with it, or nil;

port
     is the port number associated with it, or nil;

file
     is the `file' part of it, or nil.  This doesn't necessarily actually refer to a file;

target
     is the target part, or nil;

attributes
     is the attributes associated with it, or nil;

full
     is t for a fully-specified URL, with a host part indicated by // after the scheme part.

These attributes have accessors named url-PART, where PART is the name of one of the elements above, e.g., url-host.  Similarly, there are setters of the form url-set-PART.

There are functions for parsing and unparsing between the string and vector forms.

 -- Function: url-generic-parse-url url
     Return a parsed version of the string URL.

 -- Function: url-recreate-url url
     Recreates a URL string from the parsed URL.

Retrieving URLs
***************

 -- Function: url-retrieve-synchronously url
     Retrieve URL synchronously and return a buffer containing the data.  URL is either a string or a parsed URL structure.  Return nil if there are no data associated with it (the case for dired, info, or mailto URLs that need no further processing).

 -- Function: url-retrieve url callback &optional cbargs
     Retrieve URL asynchronously and call CALLBACK with args CBARGS when finished.  The callback is called when the object has been completely retrieved, with the current buffer containing the object and any MIME headers associated with it.  URL is either a string or a parsed URL structure.  Returns the buffer URL will load into, or nil if the process has already completed.

Supported URL Types
*******************

http and https
==============

The scheme http is Hypertext Transfer Protocol.  The library supports version 1.1, specified in RFC 2616.  (This supersedes 1.0, defined in RFC 1945) HTTP URLs have the following form, where most of the parts are optional:

     http://USER:PASSWORDHOST:PORT/PATH?SEARCHPART#FRAGMENT

The scheme https is a secure version of http, with transmission via SSL.  It is defined in RFC 2069.  Its default port is 443.  This scheme depends on SSL support in Emacs via the ssl.el library and is actually implemented by forcing the ssl gateway method to be used.  See Gateways in general.

 -- User Option: url-honor-refresh-requests
     This controls honouring of HTTP Refresh headers by which servers can direct clients to reload documents from the same URL or a or different one.  nil means they will not be honoured, t (the default) means they will always be honoured, and otherwise the user will be asked on each request.

Cookies
-------

 -- User Option: url-cookie-file
     The file in which cookies are stored, defaulting to cookies in the directory specified by url-configuration-directory.

 -- User Option: url-cookie-confirmation
     Specifies whether confirmation is require to accept cookies.

 -- User Option: url-cookie-multiple-line
     Specifies whether to put all cookies for the server on one line in the HTTP request to satisfy broken servers like http://www.hotmail.com.

 -- User Option: url-cookie-trusted-urls
     A list of regular expressions matching URLs from which to accept cookies always.

 -- User Option: url-cookie-untrusted-urls
     A list of regular expressions matching URLs from which to reject cookies always.

 -- User Option: url-cookie-save-interval
     The number of seconds between automatic saves of cookies to disk. Default is one hour.

Language and Encoding Preferences
---------------------------------

HTTP allows clients to express preferences for the language and encoding of documents which servers may honour.  For each of these variables, the value is a string; it can specify a single choice, or it can be a comma-separated list in descending order of preference. Each element can be followed by ;q=PRIORITY to specify its preference level; e.g., for url-mime-language-string, "de, en-gb;q=0.8, en;q=0.7".

 -- User Option: url-mime-charset-string
     This variable specifies a preference for character sets when documents can be served in more than one encoding.

     HTTP allows specifying a series of MIME charsets which indicate your preferred character set encodings, e.g., Latin-9 or Big5, and these can be weighted.  The default series is generated automatically from the associated MIME types of all defined coding systems, sorted by the coding system priority specified in Emacs.  See Recognizing Coding Systems (The GNU Emacs Manual).

 -- User Option: url-mime-language-string
     A string specifying the preferred language when servers can serve files in several languages.  Use RFC 1766 abbreviations, e.g., en for English, de for German.

     The string can be "*" to get the first available language (as opposed to the default).

HTTP URL Options
----------------

HTTP supports an OPTIONS method describing things supported by the URL.

 -- Function: url-http-options url
     Returns a property list describing options available for URL.  The property list members are:

     methods
          A list of symbols specifying what HTTP methods the resource supports.

     dav
          A list of numbers specifying what DAV protocol/schema versions are supported.

     dasl
          A list of supported DASL search types supported (string form).

     ranges
          A list of the units available for use in partial document fetches.

     p3p
          The Platform For Privacy Protection description for the resource. Currently this is just the raw header contents.

Dealing with HTTP documents
---------------------------

HTTP URLs are retrieved into a buffer containing the HTTP headers followed by the body.  Since the headers are quasi-MIME, they may be processed using the MIME library.  See Emacs MIME (The Emacs MIME Manual).  The URL package provides a function to do this in general:

 -- Function: url-decode-text-part handle &optional coding
     This function decodes charset-encoded text in the current buffer.  In Emacs, the buffer is expected to be unibyte initially and is set to multibyte after decoding. HANDLE is the MIME handle of the original part.  CODING is an explicit coding to use, overriding what the MIME headers specify. The coding system used for the decoding is returned.

     Note that this function doesn't deal with http-equiv charset specifications in HTML <meta> elements.

file and ftp
============

     ftp://USER:PASSWORD@HOST:PORT/FILE
     file://USER:PASSWORD@HOST:PORT/FILE

These schemes are defined in RFC 1808. ftp: and file: are synonymous in this library.  They allow reading arbitrary files from hosts.  Either ange-ftp (Emacs) or efs (XEmacs) is used to retrieve them from remote hosts.  Local files are accessed directly.

Compressed files are handled, but support is hard-coded so that jka-compr-compression-info-list and so on have no affect. Suffixes recognized are .z, .gz, .Z and .bz2.

 -- User Option: url-directory-index-file
     The filename to look for when indexing a directory, default "index.html".  If this file exists, and is readable, then it will be viewed instead of using dired to view the directory.

info
====

     info:FILE#NODE

Info URLs are not officially defined.  They invoke Info-goto-node with argument (FILE)NODE. #NODE is optional, defaulting to Top.

mailto
======

A mailto URL will send an email message to the address in the URL, for example mailto:foo@bar.com would compose a message to foo@bar.com.

 -- User Option: url-mail-command
     The function called whenever url needs to send mail.  This should normally be left to default from MAIL-USER-AGENT.  See Mail-Composition Methods (The GNU Emacs Manual).

An X-Url-From header field containing the URL of the document that contained the mailto URL is added if that URL is known.

RFC 2368 extends the definition of mailto URLs in RFC 1738. The form of a mailto URL is

     mailto:MAILBOX[?HEADER=CONTENTS[&HEADER=CONTENTS]]

where an arbitrary number of HEADERs can be added.  If the HEADER is body, then CONTENTS is put in the body otherwise a HEADER header field is created with CONTENTS as its contents.  Note that the URL library does not consider any headers `dangerous' so you should check them before sending the message.

Email messages are defined in RFC822.

news, nntp and snews
====================

The network news URL scheme take the following forms following RFC 1738 except that for compatibility with other clients, host and port fields may be included in news URLs though they are properly only allowed for nntp an snews.

news:NEWSGROUP
     Retrieves a list of messages in NEWSGROUP;

news:MESSAGE-ID
     Retrieves the message with the given MESSAGE-ID;

news:*
     Retrieves a list of all available newsgroups;

nntp://HOST:PORT/NEWSGROUP
nntp://HOST:PORT/MESSAGE-ID
nntp://HOST:PORT/*
     Similar to the news versions.

:PORT is optional and defaults to :119.

snews is the same as nntp except that the default port is :563. (It is tunneled through SSL.)

An nntp URL is the same as a news URL, except that the URL may specify an article by its number.

 -- User Option: url-news-server
     This variable can be used to override the default news server. Usually this will be set by the Gnus package, which is used to fetch news. It may be set from the conventional environment variable NNTPSERVER.

rlogin, telnet and tn3270
=========================

These URL schemes from RFC 1738 for logon via a terminal emulator have the form

     telnet://USER:PASSWORD@HOST:PORT

but the :PASSWORD component is ignored.

To handle rlogin, telnet and tn3270 URLs, a rlogin, telnet or tn3270 (the program names and arguments are hardcoded) session is run in a terminal-emulator buffer. Well-known ports are used if the URL does not specify a port.

irc
===

Internet Relay Chat (IRC) is handled by handing off the IRC session to a function named in url-irc-function.

 -- User Option: url-irc-function
     A function to actually open an IRC connection. This function must take five arguments, HOST, PORT, CHANNEL, USER and PASSWORD.  The CHANNEL argument specifies the channel to join immediately, this can be nil.  By default this is url-irc-zenirc.

 -- Function: url-irc-zenirc host port channel user password
     Processes the arguments and lets zenirc handle the session.

data
====

     data:[MEDIA-TYPE][;BASE64],DATA

Data URLs contain MIME data in the URL itself.  They are defined in RFC 2397.

MEDIA-TYPE is a MIME Content-Type string, possibly including parameters.  It defaults to text/plain;charset=US-ASCII.  The text/plain can be omitted but the charset parameter supplied.  If ;base64 is present, the DATA are base64-encoded.

nfs
===

     nfs://USER:PASSWORD@HOST:PORT/FILE

The nfs: scheme is defined in RFC 2224.  It is similar to ftp: except that it points to a file on a remote host that is handled by the automounter on the local host.

 -- Variable: url-nfs-automounter-directory-spec

A string saying how to invoke the NFS automounter.  Certain % sequences are recognized:

%h
     The hostname of the NFS server;

%n
     The port number of the NFS server;

%u
     The username to use to authenticate;

%p
     The password to use to authenticate;

%f
     The filename on the remote server;

%%
     A literal %.

Each can be used any number of times.

cid
===

RFC 2111

about
=====

ldap
====

The LDAP scheme is defined in RFC 2255.

imap
====

RFC 2192

man
===

     man:PAGE-SPEC

This is a non-standard scheme.  PAGE-SPEC is passed directly to the Lisp man function.

Defining New URLs
*****************

Naming conventions
==================

Required functions
==================

Optional functions
==================

Asynchronous fetching
=====================

Supporting file-name-handlers
=============================

General Facilities
******************

Disk Caching
============

The disk cache stores retrieved documents locally, whence they can be retrieved more quickly.  When requesting a URL that is in the cache, the library checks to see if the page has changed since it was last retrieved from the remote machine.  If not, the local copy is used, saving the transmission over the network. Currently the cache isn't cleared automatically.

 -- User Option: url-automatic-caching
     Setting this variable non-nil causes documents to be cached automatically.

 -- User Option: url-cache-directory
     This variable specifies the directory to store the cache files.  It defaults to sub-directory cache of url-configuration-directory.

 -- User Option: url-cache-creation-function
     The cache relies on a scheme for mapping URLs to files in the cache. This variable names a function which sets the type of cache to use. It takes a URL as argument and returns the absolute file name of the corresponding cache file.  The two supplied possibilities are url-cache-create-filename-using-md5 and url-cache-create-filename-human-readable.

 -- Function: url-cache-create-filename-using-md5 url
     Creates a cache file name from URL using MD5 hashing. This is creates entries with very few cache collisions and is fast if you have the md5 function as a primitive (Emacs 21 and XEmacs).

          (url-cache-create-filename-using-md5 "http://www.example.com/foo/bar")
            => "/home/fx/.url/cache/fx/http/com/example/www/b8a35774ad20db71c7c3409a5410e74f"

 -- Function: url-cache-create-filename-human-readable url
     Creates a cache file name from URL more obviously connected to URL than for url-cache-create-filename-using-md5, but more likely to conflict with other files.

          (url-cache-create-filename-human-readable "http://www.example.com/foo/bar")
            => "/home/fx/.url/cache/fx/http/com/example/www/foo/bar"

Proxies and Gatewaying
======================

Proxy servers are commonly used to provide gateways through firewalls or as caches serving some more-or-less local network.  Each protocol (HTTP, FTP, etc.) can have a different gateway server.  Proxying is conventionally configured commonly amongst different programs through environment variables of the form PROTOCOL_proxy, where PROTOCOL is one of the supported network protocols (http, ftp etc.).  The library recognizes such variables in either upper or lower case.  Their values are of one of the forms:

  * HOST:PORT

  * A full URL;

  * Simply a host name.

The NO_PROXY environment variable specifies URLs that should be excluded from proxying (on servers that should be contacted directly). This should be a comma-separated list of hostnames, domain names, or a mixture of both.  Asterisks can be used as wildcards, but other clients may not support that.  Domain names may be indicated by a leading dot.  For example:

     NO_PROXY="*.aventail.com,home.com,.seanet.com"

says to contact all machines in the aventail.com and seanet.com domains directly, as well as the machine named home.com.  If NO_PROXY isn't defined, no_PROXY and no_proxy are also tried, in that order.

Proxies may also be specified directly in Lisp.

 -- User Option: url-proxy-services
     This variable is an alist of URL schemes and proxy servers that gateway them.  The items are of the form (SCHEME . HOST:PORTNUMBER), says that the URL SCHEME is gatewayed through PORTNUMBER on the specified HOST.  An exception is the pseudo scheme "no_proxy", which is paired with a regexp matching host names not to be proxied.  This variable is initialized from the environment as above.

          (setq url-proxy-services
                '(("http"     . "proxy.aventail.com:80")
                  ("no_proxy" . "^.*\\(aventail\\|seanet\\)\\.com")))

Gateways in General
===================

The library provides a general gateway layer through which all networking passes.  It can both control access to the network and provide access through gateways in firewalls.  This may make direct connexions in some cases and pass through some sort of gateway in others. (Proxies (which only operate over HTTP) are implemented using this.)  The library's basic function responsible for making connexions is url-open-stream.

 -- Function: url-open-stream name buffer host service
     Open a stream to HOST, possibly via a gateway.  The other arguments are as for open-network-stream.  This will not make a connexion if url-gateway-unplugged is non-nil.

 -- Variable: url-gateway-local-host-regexp
     This is a regular expression that matches local hosts that do not require the use of a gateway.  If nil, all connexions are made through the gateway.

 -- Variable: url-gateway-method
     This variable controls which gateway method is used.  It may be useful to bind it temporarily in some applications.  It has values taken from a list of symbols.  Possible values are:

     telnet
          Use this method if you must first telnet and log into a gateway host, and then run telnet from that host to connect to outside machines.

     rlogin
          This method is identical to telnet, but uses rlogin to log into the remote machine without having to send the username and password over the wire every time.

     socks
          Use if the firewall has a SOCKS gateway running on it.  The SOCKS v5 protocol is defined in RFC 1928.

     native
          This method uses Emacs's builtin networking directly.  This is the default.  It can be used only if there is no firewall blocking access.

The following variables control the gateway methods.

 -- User Option: url-gateway-telnet-host
     The gateway host to telnet to.  Once logged in there, you then telnet out to the hosts you want to connect to.

 -- User Option: url-gateway-telnet-parameters
     This should be a list of parameters to pass to the telnet program.

 -- User Option: url-gateway-telnet-password-prompt
     This is a regular expression that matches the password prompt when logging in.

 -- User Option: url-gateway-telnet-login-prompt
     This is a regular expression that matches the username prompt when logging in.

 -- User Option: url-gateway-telnet-user-name
     The username to log in with.

 -- User Option: url-gateway-telnet-password
     The password to send when logging in.

 -- User Option: url-gateway-prompt-pattern
     This is a regular expression that matches the shell prompt.

 -- User Option: url-gateway-rlogin-host
     Host to rlogin to before telnetting out.

 -- User Option: url-gateway-rlogin-parameters
     Parametres to pass to rsh.

 -- User Option: url-gateway-rlogin-user-name
     User name to use when logging in to the gateway.

 -- User Option: url-gateway-prompt-pattern
     This is a regular expression that matches the shell prompt.

 -- User Option: socks-server
     This specifies the default server, it takes the form ("Default server" SERVER PORT VERSION) where VERSION can be either 4 or 5.

 -- Variable: socks-password
     If this is nil then you will be asked for the password, otherwise it will be used as the password for authenticating you to the SOCKS server.

 -- Variable: socks-username
     This is the username to use when authenticating yourself to the SOCKS server.  By default this is your login name.

 -- Variable: socks-timeout
     This controls how long, in seconds, to wait for responses from the SOCKS server; it is 5 by default.

 -- User Option: socks-nslookup-program
     This the nslookup program.  It is "nslookup" by default.

Suppressing Network Connexions
------------------------------

In some circumstances it is desirable to suppress making network connexions.  A typical case is when rendering HTML in a mail user agent, when external URLs should not be activated, particularly to avoid `bugs' which `call home' by fetch single-pixel images and the like.  To arrange this, bind the following variable for the duration of such processing.

 -- Variable: url-gateway-unplugged
     If this variable is non-nil new network connexions are never opened by the URL library.

History
=======

The library can maintain a global history list tracking URLs accessed. URL completion can be done from it.  The history mechanism is set up automatically via url-do-setup when it is configured to be on. Note that the size of the history list is currently not limited.

The history `list' is actually a hash table, url-history-hash-table.  It contains access times keyed by URL strings.  The times are in the format returned by current-time.

 -- Function: url-history-update-url url time
     This function updates the history table with an entry for URL accessed at the given TIME.

 -- User Option: url-history-track
     If non-nil, the library will keep track of all the URLs accessed.  If is is t, the list is saved to disk at the end of each Emacs session.  The default is nil.

 -- User Option: url-history-file
     The file storing the history list between sessions.  It defaults to history in url-configuration-directory.

 -- User Option: url-history-save-interval
     The number of seconds between automatic saves of the history list. Default is one hour.  Note that if you change this variable directly, rather than using Custom, after url-do-setup has been run, you need to run the function url-history-setup-save-timer.

 -- Function: url-history-parse-history &optional fname
     Parses the history file FNAME (default url-history-file) and sets up the history list.

 -- Function: url-history-save-history &optional fname
     Saves the current history to file FNAME (default url-history-file).

 -- Function: url-completion-function string predicate function
     You can use this function to do completion of URLs from the history.

Customization
*************

Environment Variables
=====================

The following environment variables affect the library's operation at startup.

TMPDIR
     If this is defined, URL-TEMPORARY-DIRECTORY is initialized from it.

General User Options
====================

The following user options, settable with Customize, affect the general operation of the package.

 -- User Option: url-debug
     Specifies the types of debug messages the library which are logged to the *URL-DEBUG* buffer. t means log all messages. A number means log all messages and show them with message. If may also be a list of the types of messages to be logged.

 -- User Option: url-personal-mail-address

 -- User Option: url-privacy-level

 -- User Option: url-uncompressor-alist

 -- User Option: url-passwd-entry-func

 -- User Option: url-standalone-mode

 -- User Option: url-bad-port-list

 -- User Option: url-max-password-attempts

 -- User Option: url-temporary-directory

 -- User Option: url-show-status

 -- User Option: url-confirmation-func
     The function to use for asking yes or no functions.  This is normally either y-or-n-p or yes-or-no-p, but could be another function taking a single argument (the prompt) and returning t only if an affirmative answer is given.

 -- User Option: url-gateway-method
     A symbol specifying the type of gateway support to use fro connexions from the local machine.  The supported methods are:

     telnet
          Run telnet in a subprocess to connect;

     rlogin
          Rlogin to another machine to connect;

     socks
          Connect through a socks server;

     ssl
          Connect with SSL;

     native
          Connect directly.