Site Logging.

IMPORTANT: This logging system is now obsolete; replaced by daily logs sent via email.

We have developed new logging software exclusive to tvScience Hosting.

• To add logging to your site add the following code snippet to your shtml file(s) ^2,3:

  <!--#include virtual="/cgi-bin/mail-log.cgi"-->

  This code is formatted as an HTML comment and should thus never appear when your pages are displayed in a browser. If it does, you have made a mistake. N.B: You must enter this text using the 'HTML View' of your page editing software (or use a simple text editor).
• To test its operation:
  1. Upload the page to your site.
  2. Load that page from your site using your browser. The page should appear exactly as normal, with no error messages shown.
  3. View the source code from your site for that page. The code snippet should have disappeared.
  4. Using an FTP programme (see uploading via FTP), navigate into the /example.com/cgi-bin/ directory. Check that the log file has been updated, the date and time of the file should reflect the time that you accessed your pages in Step 2 above.
• When the log file has grown to a certain size then it will be mailed to webmaster@example.com. The number of entries before mailing can be altered, see below.

Customisation.

The format of the logs can be changed. The only field that must be recorded for each hit, is the date. All others can either be:

• Recorded.
• Recorded only if non-blank (available for certain fields).
• Omitted altogether.

This option can be set in two ways:

1. Adding or changing options to your cgi.conf file⁴, which is in your /example.com/cgi-bin/ directory.
2. Adding a query string to the <!--#include virtual="/cgi-bin/mail-log.cgi" --> line in your web-pages ².

Option.	Default Value.	Field Affected.	Use in cgi.conf	Use in Query String.	Range.	Action.
log_max_entries	20	-	Y	N	5 - 200	Entries to log before mailing.
log_periodic	n	-	Y	N	m | w | d | n	Enable periodic mailing.
log_page	1	Page:	Y	Y	0 | 1	Log page accessed.
log_page_fqdn	0	Page:	Y	Y	0 | 1	Prepend the domain to the pagename.
log_pagename	Path to page.	Page:	N	Y	Any string.	The name of the page to be recorded, see log_page.
log_referrer	X	From:	Y	Y	0 | 1 | X	Log referring URL.
log_ip	1	IP:	Y	Y	0 | 1	Log IP address.
log_host	1	Host:	Y	Y	0 | 1 | X	Log host (lookup IP: address in DNS).
log_useragent	0	With:	Y	Y	0 | 1 | X	Log browser details.
log_domain	0	Domain:	Y	Y	0 | 1	Log own domain.
log_check_host	1	-	Y	N	0 | 1	Filter out page reloads.
log_interval	600	-	Y	N	10 - 1000	Reload interval (seconds), see log_check_host.
log_email	0	E-mail:	Y	Y	0 | 1 | X	Log any e-mail address.
log_newline	1	-	Y	N	0 | 1	Add newline at the end of each entry.
log_csv	0	All	Y	N	0 | 1	Log as CSV data.
log_datefmt	%d/%m/%Y%_%T	Time:	Y	N	Valid date format.	Format for the date: dd/mm/yyyy hh:mm:ss
log_logsto	webmaster@example.com	-	Y	N	Valid e-mail address.	Address to send logs.
log_ip_exclude	none	-	Y	N	Vertical-bar (|) separated list of IP4 addresses.	IP addresses to exclude from logs.
log_subj	www.example.com	-	Y	N	Any string.	Start of subject line for e-mails.
log_recent	0	-	Y	N	0 | 1	Order the log with most recent first.
log_error	1	ERROR:	Y	N	0 | 1	Place page errors in the log.
log_jump	1	Destn:	Y	N	0 | 1	Place external jumps in the log.

 Many values can be set in either the query string or in cgi.conf Any value given in the query string will override the value in cgi.conf

To set value(s) in the query string, change your HTML to something like ³:

<!--#include virtual="/cgi-bin/mail-log.cgi?log_page=0&log_host=0"-->

Not the question mark (?). Separate commands with an ampersand (&). Do not include any spaces in the command (i.e. inside the double quotes).

To add values to cgi.conf, include lines like:
# This is a comment line as it starts with a #
log_page = 0
log_host = 0
log_subj = My%_own%_string
log_logsto = logs@example.com

Don't include spaces in strings on the right hand side. Instead use the escape sequence: %_

 Description of Options.

• log_max_entries=n

This gives the minimum number of entries in each mailing.

If periodic mailings are enabled then a log will be sent if either of these two options require it. See log_periodic below.

Setting this to zero will disable logging on a site-wide basis, except for pages where the logger's query string overrides this value.

• log_periodic=m|w|d|n

This enables periodic mailings. Options are:

Value.		Action.
n	-	Never - Disable periodic mailings.
m	-	Monthly mailings.
w	-	Weekly mailings.
d	-	Daily mailings.

 

Setting this option will cause logs to be sent at regular intervals. You should set an interval long enough to guarantee at least one hit!

The log will be mailed on the first hit after the time period has expired. A site with few hits may not receive the log at the expected time.

When the log reaches its maximum permitted size it will be mailed regardless.

If log_max_entries is set then a log will be sent if either of these two options require it. See above.

• log_page=0|1

Should the Page: entry be included in the log? See also log_pagename below.

• log_pagename=<name>

If log_page=1 then you may use this string as the substitute name for the page.

Do not include spaces.

• log_page_fqdn=0|1

  Should the Page: entry include the domain name too? Do not use log_pagename with this option.

  log_referrer=0|1|X

Should the From: entry be included in the log?

Value.		Action.
0	-	Do not log.
1	-	Always log, even if empty.
X	-	Log only if non-blank.

Any X option is forced to a 1 if log_csv is set.

• log_ip=0|1

Should the IP: entry be included in the log? This identifies your visitor as far as that is possible (more information).

• log_host=0|1|X

Should the Host: entry be included in the log?

If enabled the logger will attempt to resolve the IP: into a host-name.

Possible options.

• log_useragent=0|1|X

Should the With: entry be included in the log?

Possible options.

• log_domain=0|1

Should the Domain: entry be included in the log?

This will be the domain name of your web-site.

Useful if the logged page(s) serve different websites. See also log_logsto below.

• log_check_host=0|1

Should the logger attempt to filter page reloads? See also log_interval.

• log_interval=n

Time in seconds which is used to filter out reloads of a page.

If the same log_useragent from the same log_ip load the same log_page within log_interval.
Then it will be considered a re-load and won't be entered in the log.

• log_email=0|1|X

Some browsers or robots supply an e-mail address to the server. If they do it can be logged.

Possible options.

• log_newline=0|1

Should the logger append a blank line to each log entry?

Ignored if log_csv is set.

• log_csv=0|1

Should the logs be in CSV format for importation into a database/spreadsheet?

If this is set then any X options will be forced to a '1'. log_recent is forced to a '0'.

The log will be returned with entries like this:

"16/08/2003 10:34:30","http://www.lycos.co.uk/cgi-bin/pursuit?mtemp=main&etemp=error&query=tv+components&cat=brit",212.198.114.35,thelonius.tv-science.co.uk,"Mozilla/4.0 (compatible; MSIE 4.01; Windows 95; Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)","http://www.example.com/index.htm?option=yes&index=22",spamplease@visitor.com,http://www.example.com/

Each hit will be a single line, fields will be enclosed in double quotes where appropriate.

The data can then cut and pasted into another program for further analysis.

• log_datefmt=<format>

A format string for the date field.

Interpreted sequences are:

Value.		Action.
%_	-	a space (a percent sign, followed by an underscore character).
%a	-	locale's abbreviated weekday name (Sun..Sat)
%A	-	locale's full weekday name, variable length (Sunday..Saturday)
%b	-	locale's abbreviated month name (Jan..Dec)
%B	-	locale's full month name, variable length (January..December)
%c	-	locale's date and time (Sat Nov 04 12:02:33 EST 1989)
%d	-	day of month (01..31)
%D	-	date (mm/dd/yy)
%e	-	day of month, blank padded (1..31)
%H	-	hour (00..23)
%I	-	hour (01..12)
%j	-	day of year (001..366)
%k	-	hour (0..23)
%l	-	hour (1..12)
%m	-	month (01..12)
%M	-	minute (00..59)
%p	-	locale's AM or PM
%r	-	time, 12-hour (hh:mm:ss [AP]M)
%s	-	seconds since 00:00:00, Jan 1, 1970
%S	-	second (00..60)
%T	-	time,  24-hour (hh:mm:ss)
%U	-	week number of year with Sunday as first day of week (00..53)
%V	-	week number of year with Monday as first day of week (01..52)
%w	-	day of week (0..6); 0 represents Sunday
%W	-	week number of year with Monday as first day of week (00..53)
%x	-	locale's date representation (mm/dd/yy)
%X	-	locale's time representation (%H:%M:%S)
%y	-	last two digits of year (00..99)
%Y	-	year (1970...)
%Z	-	time zone (e.g. GMT, BST)

Useful examples:

%d/%m/%Y%_%T	- yields -	28/02/2002 19:00:01
%e%_%b%_%Y%_%l:%M%_%r	- yields -	1 Feb 2003 6:20 AM

Do not use quotes of any kind (" or ' or `) or the dollar sign ($) in your string.

An invalid format may result in an error.

• log_logsto=<e-mail>

The e-mail address to send logs.

A valid address might be: access.logs_2@tv-science.co.uk

Do not include brackets of any description, spaces, quotes, comments or any other RFC 2822 junk.

An invalid address will result in an error.

If you are logging a page which serves multiple domains then set this variable.

Be sure to request a destination for the default of webmaster@example.com, otherwise logs will be lost.

• log_ip_exclude=<IP_address>

Exclude these IP addresses from your logs. Use this option to exclude your own page accesses.

Typical entry: 123.456.789.10|111.22.3.4

• log_subj=<string>

This is the string which starts the Subject Line of all the loggers' e-mails.

The default is the domain logged.

Do not use quotes of any kind (" or ' or `) or the dollar sign ($) in your string.

Include a space by using the sequence: %_

• log_recent=0|1

Should the logger place most recent entries into the log first.

Ignored if logging in CSV format.

• log_error=0|1

Log server errors?

Relevant error messages (broken links etc.) are placed in the log with this option.

• log_jump=0|1

Log external jumps?

Jumps to external sites can be logged with /cgi-bin/jumper.pl. Setting this option to 0 disables this feature site-wide. See the documentation on how to use/cgi-bin/jumper.pl.

The logging data for your site is held in the file: /cgi-bin/logfile

To log visitors as they leave your site through an external link, see the External Jump Logger.

Here is further information on the data contained in your logs.

You must give your files an .shtml extension to use this logging feature.

Return to the Top of This Page.

Notes:

1	Not to be confused with the logger's query string.
2	The previous form of <!--#exec cgi="/cgi-bin/mail-log.cgi"--> can no longer be used.
3	These examples can also be found in a text file.
4	Here is an example cgi.conf file. Be sure to preserve the Linux file permissions when editing your copy of this file.

 

Go to the TV Science Ltd. Homepage. Legal Information. Last update of this page: . E-Mail: webmaster@tv–science.co.uk