Pass non- text/* files onto indexing code IF there is a FileFilter
associated with the *extension* of the URL. Fixes the problem of not being
able to index, say, pdf files by using the FileFilter configuation option.
Fixed bug where nulls were stripped when using FileFilter with -S prog.
Catch by Greg Fenton. [moseley]
Filters (FileFilter directive) did not work correctly when spidering with
the -S http method. A new filter system was developed and now filtering of
documents (e.g. pdf->html or MSWord->text) is handled by the
When indexing with the -S http method only documents of content-type
``text/*'' are indexed. Other documents must be converted to text by using
the filter system.
Documentation is now included in the source distribution as .pod (perldoc)
files, and as HTML files. In addition, the distribution can now generate
PDF, postscript, and unix man pages from the source .pod files. See README for more information.
Pre 2.2 some internal data was stored in fixed locations within the index,
namely the file name, file size, and title. 2.2 introduced new internal
data such as the last modified date, and document summaries. This data is
considered meta data since it is data about a document.
Instead of adding new data to the internal structure of the index file, it
was decided to use the MetaNames and PropertyNames feature of Swish-e to
store this meta information. This allows for new meta data to be added at a
later time (e.g. Content-type), and provides an easy and customizable way
to print results with the -p switch and the new -x switch. In addition, search results can now be sorted and limited by
Swish-e now merges (and sorts) the results from multiple indexes when using -f to specify more than one index. This change effects the way maxhits (-m) works. Here's a summary of the way it works for the different versions.
1.3.2 - MaxHits returns first N results starting from the first index.
e.g. maxhits=20; 15 hits Index1, 40 hits Index2
All 15 from Index1 plus first five from Index2 = 20 hits.
2.0.0 - MaxHits returns first N results from each index.
e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
All 15 from Index1 plus 15 from Index2.
2.2.0 - Results are merged and first N results are returned.
e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
Results are merged from each index and sorted
(rank is the default sort) and only the first
20 are returned.
You can now use -S prog to use an external program to supply documents to
Swish-e. This external program can be used to spider web servers, index
it can be indexed by Swish-e. Examples are given in the
TranslateCharacters now is done before WordCharacters is checked. For
TranslateCharacters ˝ n
Now El Ni˝o will be indexed as El Nino (el and nino), even though ˝
is not listed in WordCharacters.
Previously, stopwords were checked after stemming and soundex conversions,
as well as most of the other word checks (WordCharacters, min/max length
and so on). This meant that the stopword list probably didn't work as
expected when using stemming.
The search parser was rewritten to correct a number of logic errors.
Swish-e did not differentiate between meta names, Swish-e operators and
search words when parsing the query. This meant, for example, that
metanames might be broken up by the WordCharacters setting, and that they
could be stemmed.
Swish-e operator characters "*()= can now be searched by escaping with a backslash. For example:
./swish-e -w 'this\=odd\)word'
will end up searching for the word this=odd)word. To search for a backslash character preceed it with a backslash.
Currently, searching for:
./swish-e -w 'this\*'
is the same as a wildcard search. This may be fixed in the future.
Searching for buzzwords with those characters will still require
backslashing. This also may change to allow some un-escaped operator
characters, but some will always need to be escaped (e.g. the double-quote
A bug was fixed in the parse_line() function (in string.c) where backslashes were not escaping the next character. parse_line() is used to parse a string of text into tokens (words). Normally splitting
is done at whitespace. You may use quotes (single or double) to define a
string (that might include whitespace) as a single parameter. The backslash
can also be used to escape the following character when *within* quotes
(e.g. to escape an embedded quote character).
ReplaceRules append "foo bar" <- define "foo bar" as a single word
ReplaceRules append "foo\"bar" <- escape the quotes
ReplaceRules append 'foo"bar' <- same thing
Previous versions of Swish-e included a configuration file called
user.config which contained examples of all directives. This has been replaced by a
series of example configuration files located in the
conf directory. The configuration directives are now described in
Multiple string properties of the same name in a document are now concatenated into one
property. A space character is added between the strings if needed. A
warning will be generated if multiple numeric or date properties are found
in the same document, and the additional properties will be ignored.
Previously, properties of the same name were added to the index, but could
not be retrieved.
To do: remove the next pointer, and allow user-defined character to place between properties.
To improve security Swish-e now uses the mkstemp(3) function to create temporary files. Temporary files are used while indexing
only. This may result in some portability issues, but the security issues
(Currently this does not apply to the -S http indexing method.)
mkstemp opens the temporary with O_EXCL|O_CREAT flags. This prevents overwriting
existing files. In addition, the name of the file created is a lot harder
to guess by attackers. The temporary file is created with only owner
Please report any portability issues on the Swish-e discussion list.
Swish-e now uses the environment variables TMPDIR, TMP, and
TEMP (in that order) to decide where to write temporary files. The configuration
setting of TmpDir will be used if none of the environment variables are set. Swish-e uses the
current directory otherwise; there is no default temporary directory.
Since the environment variables override the configuration settings, a
warning will be issued if you set TmpDir
in the configuration file and there's also an environment variable set.
Temporary files begin with the letters ``swtmp'' (which can be changed in
config.h), followed by two or more letters that indicate the type of temporary
file, and some random characters to complete the file name. If indexing is
aborted for some reason you may find these temporary files left behind.
The IndexContents directive assigns internal Swish-e document parsers to
files based on their file type. The DefaultContents directive assigns a
parser to be used on file that are not assigned a parser with
The builtin list of stopwords has been removed. Use of the SwishDefault
word will generate a warning, and no stop words will be used. You must now
specify a list of stopwords, or specify a file of stopwords.
A sample file stopwords.txt has been included in the conf/stopwords
directory of the distribution, and can be used by the directive:
Buzzwords are words that should be indexed as-is, without checking for
stopwords, word length, WordCharacters, or any other of the word limiting
features. This allows indexing of things like C++ when ``+'' is not listed in WordCharacters.
Currenly, IgnoreFirstChar and IgnoreLastChar will be stripped before
In the future we may use separate IgnoreFirst/Last settings for buzzwords
since, for example, you may wish to index all + within Swish-e words, but strip + from the start/end of Swish-e words, but not from the buzzword C++.
Before Swish-e 2.2 all user-defined document properties were stored in the
index as strings. PropertyNamesNumeric and PropertyNamesDate tell it that a
property should be stored in binary format. This allows for correct sorting
of numeric properties.
Currenly, only integers can be stored, such as a unix timestamp. (Swish-e
uses strtoul to convert the number to an unsigned long internally.)
PropertyNamesDate only indicates to Swish-e that a number is a unix
timestamp, and to display the property as a formatted time when printing
results. Swish does not currently parse date strings; you must provide a
If compiled with zlib, Swish-e uses this setting to control the level of
compression applied to properties. Properties must be long enough (defined
in config.h) to be compressed. Useful for StoreDescription.
SWISH-E is distributed with no warranty under the terms of the
GNU Public License,
Free Software Foundation, Inc.,
59 Temple Place - Suite 330, Boston, MA 02111-1307, USA
Public questions may be posted to
the SWISH-E Discussion.