Exclude.area Directive

Defines a prefix of URLs excluded from the area. A url is included in the area if it is a root (seed) of the area or matches (starts with) at least one inclusion and does not match any exclusions.

Example:

include.internal = http://www.mycom.com/internal
exclude.internal = http://www.mycom.com/internal/management
      

See also

• Index Area
• Area Directive
• Include.area Directive

Exclude.loglinks Directive

Defines a prefix of path of URLs ignored when processing logs of the site. Note that URLs are also ignored if they are rejected by filters in Nutch configuration.

Note that in this directive path prefixes are used instead of URL prefixes because the scope of this directive is limited to one site.

Example:

exclude.loglinks = /images/
exclude.loglinks = /css/
exclude.loglinks = /temp/
      

See also

• Loglinks Area
• Area Directive
• Exclude.area Directive

Facet Directive

Controls faceted search. Faceting directives set for sites override root configuration if parameter domain is used in the request.

Another method to override faceting directives is to add facet=true and other Solr faceting parameters to the request. If Arch finds a "facet field in request, it ignores configuration parameters. Set facet.sites and facet.areas to on if have more than one site and area.

Example:

facet.sites=on
      

See also

• Domain
• Faceting in Solr

File.bookmarks Directive

Bookmark files contain bookmark URLs to index. There can be several bookmark files used in one site. Bookmark URLs can point to third party sites.

Example:

file.bookmarks = /opt/arch/conf/arch/sites/mySite/bookmarks1.txt
file.bookmarks = /opt/arch/conf/arch/sites/mySite/bookmarks2.txt
      

See also

• Bookmarks Area
• Groupsread.bookmarks Directive
• Usersread.bookmarks Directive

Frontend.profile Directive

Front-end search profile parameter defines front-end id, password, sites and areas that the front-end is allowed to search and users and groups that are allowed to do search via this front-end. The fields are separated by |. The required parameters are the id and password. The rest can be left blank.

Note: frontent.profile parameter is expected by Arch reference authentication plugin. It is very likely that you will want to replace it with a plugin implementing authentication method used in your organization. Your plugin may use a different configuration parameters set.

Note: unless parameter domain is sent with request, front-end authentication related parameters are taken from the root configuration file and must be declared there. If domain is sent in request, it must match a site name. In this case all authentication related parameters are taken from that site configuration file and search is limited to that site data only.

Please note that regular expressions depend on whether IPv4 or IPv6 is used. The examples below are valid for IPv4.

Example:

frontend.profile = global | pass1
      

See also

• Security.enabled Directive

Groupsread.bookmarks Directive

Lists names of groups allowed to see indexed bookmarks.

Example:

groupsread.bookmarks = staff public
      

See also

• Bookmarks Area
• Usersread.bookmarks Directive

Hits.threshold Directive

If an IP address generates more than this number of hits within a capture interval, it is considered belonging to a robot and ignored in the future.

In this example, IP address is blocked if there are more than 30 accesses to text documents in a 5 minutes interval.

Example:

capture.interval = 300
hits.threshold = 30
      

See also

• Ip.filter Directive
• Capture.interval Directive
• Max.hits.ip.day Directive
• Max.hits.day Directive

Ignore.in.logs Directive

This directive is depricated and ignored.

Ignored URLs are now defined by filters in Nutch configuration, such as the regex URL filter and the suffix URL filter.

Arch is an extension of Nutch, and most Nutch confoguration options are effective in Arch.

See also

• Nutch tutorial

Include.area Directive

Defines a prefix of URLs included in the area. A url is included in the area if it is a root (seed) of the area or matches (starts with) at least one inclusion and does not match any exclusions.

Example:

include.internal = http://www.mycom.com/internal
      

See also

• Index Area
• Area Directive
• Exclude.area Directive

Interval.area Directive

Re-indexing interval (days) with weekdays on which re-indexing is allowed. For example, to re-index area approx. every 20 days, but only on weekends, see the example below. The default value is 0, which means that the area will be re-crawled each time Arch is started.

Example:

interval.documentation = 20, sat, sun
      

See also

• Index Area
• Area Directive
• Include.area Directive
• Exclude.area Directive
• Root.area Directive

Ip.filter Directive

Switches on or off attempts to identify and ignore log records caused by search engines and other non-human clients based on IP stats. All IP based filtering can be turned off by setting this parameter to off. Robots accesses can still be filtered out based on the client type, if this information is in the logs and robots do not masquerade as browsers.

Example:

ip.filter = on
      

See also

• History.size Directive
• Max.hits.day Directive
• Max.hits.ip.day Directive
• Max.score Directive

Log.format Directive

Value in a site configuration file, if provided, overrides value in the root configuration file.

This parameter must match format attribute (defined in plugin.xml file) of a log parser plugin that is able to process logs of this type. The default arch log parser works with logs in combined format.

Example:

log.format = combined
      

Log.length Directive

Defines effective log length, in days. If log records are available for longer then defined by this directive, the older records are ignored. E.g. if we have logs for the last 10 years, but the value of log.length is 365, only the records of the the latest 365 days are used to compute document scores.

Example:

log.length = 365
      

Log.repository Directive

Sometimes logs need merging before they can be processed correctly, for example, if site has both HTTP and HTTPS access. If log.repository option is provided, logs will be read from the log directories (as specified by the log option above), merged and placed into the first directory specified by the log.repository option. After that, they will be used for "normal" processing. Note that log masks are used exactly as they are used in the logs option. In fact, if there is no logs option provided and the default logs directory is empty or absent, log.repository option will be used instead, if present, and no log pre-processing done.

Example:

log.repository = file:///var/logs/www.atnf.csiro.au/ | ^merged_log_2019-.+ 
      

Log.repository Directive

Sometimes logs need merging before they can be processed correctly, for example, if site has both HTTP and HTTPS access. If log.repository option is provided, logs will be read from the log directories (as specified by the log option above), merged and placed into the first directory specified by the log.repository option. After that, they will be used for "normal" processing. Note that log masks are used exactly as they are used in the logs option. In fact, if there is no logs option provided and the default logs directory is empty or absent, log.repository option will be used instead, if present, and no log pre-processing done.

Example:

log.repository = file:///var/logs/www.atnf.csiro.au/ | ^merged_log_2019-.+ 
      

See also

• Logs Directive
• Merged retention Directive

Logs Directive

Location of log files. Can be a local or remote directory with a number of file masks defined by regular expressions. If no file masks are provided, all files match. Files found in this directory must be web server log files of this log site. Multiple locations can be used simultaneously.

Example:

logs = file:///var/logs/www.atnf.csiro.au/ | ^latest.+ | ^access\.2019-.+ 
logs = sftp://arch:mypass@myhost:22/var/log/www.atnf.csiro.au/ | ^access\.log-2019.+

      

See also

• Log.repository Directive

Max.hits.norm Directive

Number of hits from IP to URL per day beyond which the IP is placed on the list to ignore.

It should be used carefuly because some pages, e.g. home pages, may be requested often in normal use. The default number of 1000 prectically disables use of this parameter.

Example:

max.hits.norm = 1000
      

See also

• Ip.filter Directive
• Max.hits.ip.day Directive
• Max.hits.ip.day Directive

Max.hits.day Directive

Max number of accesses to a page per day that counts, the rest are ignored.

This parameter should be set to a value that is the highest reasonable estimate of accesses to a page per day, excluding accesses by automatic clients.

Example:

max.hits.day = 5000
      

See also

• History.size Directive
• Ip.filter Directive
• Max.hits.ip.day Directive
• Max.score Directive

Max.hits.ip.day Directive

Max number of accesses to a page from a single IP per day that counts, the rest are ignored.

Example:

max.hits.ip.day = 5
      

See also

• History.size Directive
• Ip.filter Directive
• Max.hits.day Directive

Max.ip.cache Directive

Max size of memory held IP address information cache. This cache is used when computing a list of ignored IP addresses with aim to count only accesses generated by human readers and exclude accesses generated by robots. A bigger cache speeds up ignored IPs list generation, but requires more memory.

Example:

max.ip.cache = 100000
      

See also

• Ip.filter Directive

Max.score Directive

Max document weight value that is stored with a document in Solr (Lucene) index. This parameter can be changed to tune the effect of document weights derived from logs on Solr results ranking. In our experiments, 5 was the optimal value.

Example:

max.score = 5
      

See also

• Max.hits.day Directive
• Max.score Directive

Max.url.length Directive

Too long URLs extracted from logs are often a sign of hacker activity and should be ignored together with IP addresses that generated them. Use this parameter to ignore URLs longer than its value. Set to -1 to not limit the length.

Example:

max.url.length = 300
      

Max.urls Directive

Default max number of urls to fetch at each crawling iteration. This is passed to Nutch as the topN parameter.

The total number of indexed URLs is limited to the crawling depth (number of crawling iterations) multiplied by max.urls.

If present in Arch crawling script, CRAWLING_MAX_URLS directive overrides this directive.

Example:

max.urls = 10000
      

See also

• CRAWLING_MAX_URLS Directive

Max.urls.area Directive

This directive is ignored. All areas are crawled with max URLs parameter defined in the root configuration file or the Arch crawling script.

See also

• CRAWLING_MAX_URLS Directive
• Max.urls Directive

Mail.host Directive

Sets the mail server to use to post mail.

Site configuration directives, if present, override root configuration directives.

Example:

mail.host = smtp.mycom.com
      

See also

• Mail.level Directive

Mail.level Directive

There are five levels: DEBUG, INFO, WARN, ERROR, OFF. The DEBUG level is most detailed, the OFF level switches mail off.

Note that, even if mail.level is not OFF, email will not be sent if other mail related parameters are missing or invalid.

Site configuration directives, if present, override root configuration directives.

Example:

mail.level = OFF
      

Mail.password Directive

Sets the mail server password to use if posting mail requires authentication.

Site configuration directives, if present, override root configuration directives.

Example:

mail.password = My.name.is.Bond,James.Bond.
      

See also

• Mail.transport.protocol Directive
• Mail.user Directive

Mail.transport.protocol Directive

Sets the mail transport protocol to use to post mail.

Site configuration directives, if present, override root configuration directives.

Example:

mail.transport.protocol = smtp
      

See also

• Mail.level Directive

Mail.recipient Directive

Sets email addresses to send messages to.

Site configuration directives, if present, override root configuration directives. If this directive is present in a site configuration file, email messages related to the site are sent to address defined in its configuration, and their text is not included in the combined message sent to the address configured in the root configuration file.

To cause sending a separate email for a site, define at least mail.recipient parameter in its configuration. The rest of the mail related parameters are optional, as long as they are defined in the global config file.

Example:

mail.recipient = address1@mycompany.com, address2@anothercompany.com
      

See also

• Mail.level Directive
• Mail.transport.protocol Directive

Mail.subject Directive

Sets the subject line for mail messages from Arch.

Site configuration directives, if present, override root configuration directives.

Example:

mail.subject = My message from Arch
      

See also

• Mail.level Directive
• Mail.transport.protocol Directive

Mail.user Directive

Sets the mail user name to use if posting mail requires authentication.

Site configuration directives, if present, override root configuration directives.

Example:

mail.user = bond007
      

See also

• Mail.password Directive
• Mail.transport.protocol Directive

Merged.retention Directive

Specifies for how long to store merged logs in log repository before deleting them. -1 means no time limit.

Example:

merged.retention = 365
      

See also

• Marged.repository Directive

Parallel.indexing Directive

If off, areas are crawled sequentially, one at a time. The sequential mode is recommended for troubleshooting a new installation or after a new site has been added. In this mode, if indexing fails, you can fix the problem and restart indexing. It will skip areas that have been processed successfully in the previous run. The parallel mode may significantly decrease the time of indexing. Note that bookmarks areas will be processed sequentially even if parallel processing is switched on.

In parallel indexing mode, all pre-configured crawling seeds (roots) together with all links found in logs are used as starting points of crawling.

Example:

parallel.indexing = off
      

See also

• Loglinks Area
• Bookmarks Area
• Parallel Indexing
• Sequential indexing

Permissions Directive

This parameter is used to set access permissions for a file or folder (and by default, it’s subfolders). These permissions make effect when affected documents are re-indexed.

In the syntax definition:

• 'f' or 'd' stand for "folder" or "document" respectively;
• groups-R/O – a space separated list of user groups having R/O access;
• groups-R/W – a space separated list of user groups having R/W access;
• users-R/O – a space separated list of users having R/O access;
• users-R/W – a space separated list of users having R/W access;
• owners – a space separated list of users having administrator access;
• 's' or 'i' stand for defined or inherited permissions mode respectively.

If inherited mode is set, all user and group lists, including owners, are inherited from the parent folder. This setting practically disables the permissions directive because all permissions are inherited from the parent, which is default mode.

By default, site root has an implicit permissions declaration like this:

f | http://mysite.com/ | public | null | guest | null | admin | s

All other folders and documents inherit permissions from their parent folders.

The example below limits access to the internal part of the site to the staff group only.

Example:

permissions = f | http://mysite.com/  | public | staff | admin | admin | admin | s 
permissions = f | http://mysite.com/internal/ | staff | staff | admin | admin | admin | s 
      

See also

• Allowed.ip.addresses Directive
• Blocked.ip.addresses Directive
• Allowed.users Directive
• Security.enabled Directive

Prune.content.types Directive

Server output of pages with these content types will be prunned.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

prune.content.types = text/html | text/javascript
      

See also

• Prune.file.types Directive
• Scan.content.types Directive
• Prune.content.types.after Directive
• Scan.enabled Directive

Prune.content.types.after Directive

Extracted (by document parsers) text content of pages of these content types will be cleaned before indexing by removing fragments defined by the scan.ignore.bits.after parameter.

Note that files of types listed in the prune.content.types parameter are processed before parsing the document. This kind of processing is relatively easy to apply to text based files, such as HTML and PHP files. The prune.content.types.after parameter lists content types of files that are to be pruned after parsing them (extracting textual content from them).

To tune pruning, set temporarily log4j.logger.au.csiro.cass.arch.security.BasicPruner to TRACE in conf/log4j.properties file to see what input the pruner gets.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

prune.content.types.after = application/msword | application/pdf
      

See also

• Prune.file.types Directive
• Scan.content.types Directive
• Prune.content.types Directive
• Scan.enabled Directive

Prune.file.types Directive

Server output of pages with these file types will be prunned.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

prune.file.types = htm html asp aspx do
prune.file.types = php php3 php4 php5
      

See also

• Prune.content.types Directive
• Scan.file.types Directive
• Scan.enabled Directive

Prune.file.types.after Directive

Extracted (by document parsers) text content of pages of these file types will be cleaned before indexing by removing fragments defined by the scan.ignore.bits.after parameter.

Note that files of types listed in the prune.file.types parameter are processed before parsing the document. This kind of processing is relatively easy to apply to text based files, such as HTML and PHP files. The prune.file.types.after parameter lists types of files that are to be pruned after parsing them (extracting textual content from them).

To tune pruning, set temporarily log4j.logger.au.csiro.cass.arch.security.BasicPruner to TRACE in conf/log4j.properties file to see what input the pruner gets.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

prune.file.types.after = doc docx pdf
      

See also

• Prune.file.types Directive
• Scan.file.types Directive
• Prune.content.types.after Directive
• Scan.enabled Directive

Remove.duplicates Directive

If switched on, Arch will attempt to remove duplicated entries in index. Such duplicates may be caused by use of URL aliases in indexed sites, or crawling log links with depth higher than one.

Arch will remove entries that have identical hash sums and occur in the same area. Index areas may overlap by design, and entries belonging to different areas are not considered being duplicates even if they are identical.

Removing of duplicates takes considerable time. This time can be reduced if pages have defined canonical URLs in their metadata.

Example:

remove.duplicates = on
      

See also

• Depth.loglinks directive

Root.area Directive

Defines a seed URL for crawling of the area. Area roots (seeds) are used to start area crawling and are included in the area index. The number of roots in an area is limited only by resources.

Example:

root.main = http://www.mycom.com/index.html
root.main = http://www.mycom.com/links.html
      

See also

• Index Area
• Area Directive

Security.enabled Directive

If set to off, this directive switches off security related checks and limitations.

It is recommended to keep security disabled until you get your search working as expected. Else it may get in the way and it will be harder to tell what is causing problems.

Example:

security.enabled = off
      

See also

• Blocked.ip.addresses Directive
• Allowed.ip.addresses Directive
• Allowed.users directive
• Allowed.groups directive
• Allowed.sites directive
• Allowed.areas directive
• Default.users directive
• Default.groups directive
• Default.sites directive
• Default.areas directive
• Admin.ip.address Directive

Scan.alert Directive

Rise an alert of this level if found this string while scanning output or source of the page, or both.

Each alert entry consists of three fields separated by a pipe character. The first field is the string to look for. The second field is the level of alert to rise: SAFE, UNSURE, UNSAFE or THREAT. The third field is what to scan: OUT - only output of the page, SRC - only source of the page, BOTH - both of them.

It is recommended that scanning is switched off during first crawl, else it will generate too many alerts, as every page and link will be new to it.

Example:

scan.alert = mail( | unsafe | src
scan.alert = $_REQUEST | unsafe | src
scan.alert = http://hostile.com | threat | both
      

See also

• Scan.alert.level Directive
• Scan.enabled Directive

Scan.alert.level Directive

The lowest level of alerts is SAFE. If scan.alert.level is set to SAFE, all alerts will be reported.

The highest level of alerts is THREAT. If scan.alert.level is set to THREAT, only alerts of THREAT level will be reported.

Example:

scan.alert.level= UNSURE
      

See also

• Scan.alert Directive
• Scan.enabled Directive

Scan.content.types Directive

Pages with these content types will be scanned.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

scan.content.types = text/html | text/javascript
      

See also

• Scan.file.types Directive
• Scan.src.content.types Directive
• Prune.content.types Directive
• Scan.enabled Directive

Scan.enabled Directive

Enable or disable Arch security scanning and document cleaning related features.

Arch can monitor your site for potential threats, new and changed pages, scripts and links. You can define clues to look for and Arch will notify you when it finds something. However, security scanning has a cost because extra processing is involved. For better protection, it is desirable to scan not only output pages (such as those produced by PHP), but the source (PHP) scripts as well. If you do not want to do security scanning or not ready to configure scanning and prunning parameters, just disable scanning.

It is recommended that scanning is switched off during first crawl, else it will generate too many alerts, as every page and link will be new to it.

Example:

scan.enabled = off
      

See also

• Scan.file.types Directive
• Scan.src.file.types Directive
• Scan.content.types Directive
• Scan.src.content.types Directive
• Prune.file.types Directive
• Prune.content.types Directive

Scan.file.types Directive

Pages with these file types will be scanned.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

scan.file.types = htm html asp aspx do
scan.file.types = php php3 php4 php5
      

See also

• Scan.content.types Directive
• Scan.src.file.types Directive
• Prune.content.types Directive
• Scan.enabled Directive

Scan.ignore.bits Directive

Strings identifying text fragments to ignore. This can be used, for example, to avoid indexing common page fragments, such as headers and footers. Type one pair per line, separate start and end strings with a pipe character.

Each pair must be enclosed in '[' or '{' at the start and ']' or '}' at the end. '[' and ']' mean ignore the fragment, including the boundary string.'{' and '}' mean ignore the fragment, not including the boundary string.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Each one defines one fragment to ignore.

Example:

scan.ignore.bits = [ <div class="header"> | <div class="content"> }
scan.ignore.bits = [ <div class="footer"> | </html> }
      

See also

• Scan.ignore.bits.after Directive
• Scan.ignore.scripts Directive
• Scan.ignore.links Directive
• Scan.alert Directive
• Scan.enabled Directive

Scan.ignore.bits.after Directive

Strings identifying text fragments to ignore in parsed pages. This can be used, for example, to avoid indexing common fragments of MS Word and PDF files. Type one pair per line, separate start and end strings with a pipe character.

Each pair must be enclosed in '[' or '{' at the start and ']' or '}' at the end. '[' and ']' mean ignore the fragment, including the boundary string.'{' and '}' mean ignore the fragment, not including the boundary string.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Each one defines one fragment to ignore.

Example:

scan.ignore.bits.after = [ Contents | page 65 ]
scan.ignore.bits.after = [ Contact | contact@our.com ]
      

See also

• Scan.ignore.bits Directive
• Scan.ignore.scripts Directive
• Scan.alert Directive
• Scan.enabled Directive

Scan.ignore.links Directive

The listed links will be ignored. This can be used, for example, to avoid reporting of common links, such as those occurring in headers and footers.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

scan.ignore.links = http://www.mysite.com/contact.html
scan.ignore.links = http://www.mysite.com/home.html
      

See also

• Scan.ignore.bits Directive
• Scan.ignore.scripts Directive
• Scan.alert Directive
• Scan.enabled Directive

Scan.ignore.scripts Directive

Script fragments containing these strings will be ignored. This can be used, for example, to avoid scanning and reporting common scripts, such as those used to generate headers and footers. Use with care as attackers may include one of such fragments in their script to hide it. Separate strings with a pipe character.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

scan.ignore.scripts = include("header.inc"); | include("footer.inc");
scan.ignore.scripts = include("sidebar.inc"); 
      

See also

• Scan.ignore.bits Directive
• Scan.ignore.scripts Directive
• Scan.alert Directive
• Scan.enabled Directive

Scan.min.script.size Directive

Script fragments of size smaller than this are ignored. Use with care.

Site configuration directives, if present, override root configuration directive.

Example:

scan.min.script.size = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.report.changed.forms Directive

Switches on and off reporting changed pages with forms in them. This notification is of UNSURE level. If your alerts level is set to UNSAFE or THREAT, change reports will not be sent.

Site configuration directives, if present, override root configuration directive.

Example:

scan.report.changed.forms = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.report.changed.pages Directive

Switches on and off reporting changed pages with forms in them. This notification is of UNSURE level. If your alerts level is set to UNSAFE or THREAT, change reports will not be sent.

Site configuration directives, if present, override root configuration directive.

Example:

scan.report.changed.pages = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.report.changed.scripts Directive

Switches on and off reporting changed script files. This notification is of UNSURE level. If your alerts level is set to UNSAFE or THREAT, change reports will not be sent.

Site configuration directives, if present, override root configuration directive.

Example:

scan.report.changed.scripts = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.report.link.changes Directive

Switches on and off reporting changed links in pages. This notification is of UNSURE level. If your alerts level is set to UNSAFE or THREAT, change reports will not be sent.

Site configuration directives, if present, override root configuration directive.

Example:

scan.report.link.changes = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.report.new.forms Directive

Switches on and off reporting changed pages with forms in them. This notification is of UNSURE level. If your alerts level is set to UNSAFE or THREAT, change reports will not be sent.

Site configuration directives, if present, override root configuration directive.

Example:

scan.report.new.forms = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.report.new.pages Directive

Switches on and off reporting new pages. This notification is of UNSURE level. If your alerts level is set to UNSAFE or THREAT, change reports will not be sent.

Site configuration directives, if present, override root configuration directive.

Example:

scan.report.new.pages = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.report.new.scripts Directive

Switches on and off reporting changed pages with forms in them. This notification is of UNSURE level. If your alerts level is set to UNSAFE or THREAT, change reports will not be sent.

Site configuration directives, if present, override root configuration directive.

Example:

scan.report.new.scripts = on
      

See also

• Scan.alert.level
• Scan.alert Directive
• Scan.enabled Directive

Scan.script.edges Directive

Defines strings to use to find starts and ends of script fragments in pages. Type in start and end strings separated by pipe character.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

scan.script.edges = <? | ?> 
scan.script.edges = <script | script> 
      

See also

• Scan.ignore.bits Directive
• Scan.ignore.scripts Directive
• Scan.alert Directive
• Scan.enabled Directive

Scan.src.content.types Directive

Source code of pages with these content types will be scanned. Access to source code of these pages must be provided via scan.source.access.url directive.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

scan.content.types = text/html | text/javascript
      

See also

• Scan.src.file.types Directive
• Scan.content.types Directive
• Scan.source.access.url Directive
• Prune.content.types Directive
• Scan.enabled Directive

Scan.src.file.types Directive

Source code of pages with these file types will be scanned. Access to source code of these pages must be provided via scan.source.access.url directive.

Site configuration directives, if present, override root configuration directives

There can be several directives in one configuration file. Their contents are merged.

Example:

scan.src.file.types = htm html asp aspx do
scan.src.file.types = php php3 php4 php5
      

See also

• Scan.src.content.types Directive
• Scan.file.types Directive
• Prune.content.types Directive
• Scan.enabled Directive

Sitemap.url Directive

A URL of file with pre-processed sitemap data that can be used as a substitute to log processing. See more in Arch deployment manual about generating, encrypting and making available sitemaps of remote sites.

Example:

sitemap.url = http://mySite.base.url/arch/sitemap.dat
      

See also

• Loglinks Area

Solr.url Directive

Address of Solr server that is being used with Arch. Arch comes with its own copy of Jetty engine and Solr. It is also possible to use Solr installed on a remote computer, especially if several Arch crawlers are being used in parallel.

If the Jetty engine installed with Arch is not being used, disable starting it in the Arch crawling script.

Example:

solr.url = http://localhost:8993/arch
      

target.db = jdbc:mysql://localhost/arch?user=myname&password=mypassword
      

temp.dir = /opt/arch/temp
      

threads = 10
      

threads.documentatio. = 10
      

usersread.bookmarks = guest admin
      

watch.mode = off