SANE '98 conference:

For the impatient: instead of reading the proceedings below, view the presentation slides...

UPDATE: the presentation at LinuxWorld conference in Amsterdam, 14-Oct-2004. (PDF)

Abstract

In 1994, a project was started to investigate the options for a world-wide intranet for all Greenpeace staff - either in offices, on ships or on the road with laptops. After one year intensive study, three major contenders remained: AT&T Easylink, MHS/Netware and TCP/IP intranet under Unix. Despite the obvious compatibility advantages of using Internet software on workstations, the main drawback was that office support staff in all our offices needed to learn Unix and TCP/IP for their new servers. Most of them were used to Novell servers of varying sizes until then. Also considering cost reasons, it was decided to install "Black Boxes" on Intel-CPU-based servers in all major offices with all required TCP/IP Intranet server software under Linux O/S. The staff didn't need to learn new software: it was being maintained remotely through telnet and FTP from the Greenpeace International head office. Several offices were interconnected via medium speed Frame Relay circuits, and X.25 dial-in was available for staff in world-wide locations, both methods leased from an international network provider. After several years of operation, it has proven to be a very manageable, cost-effective and highly reliable method. The latest developments concentrated on optimising mini-networks on ships, using Linux-based servers and operator restricted ISDN dial-up via Inmarsat HSD satellite services.

About the speaker:

Sjoerd studied computer technology at HTS 'A', a polytechnic college in Amsterdam, NL, graduating in 1976. He moved to Australia, where he worked for Digital Equipment Corporation for almost 3 years. He was then contracted by the Australian government, Antarctic Division, for 8 years, both in Upper Atmospheric Science and Communications departments. He then joined Greenpeace International in 1987, introducing many new communications methods from Antarctica, ships and offices.

Introduction

This paper describes not so much detailed technical solutions, as these can be covered by specialists much better than myself. What may be interesting however is the process of selecting and implementing an internal global information and communication system for a world-wide non-profit organisation with offices in 30-odd countries, ships, buses and for home workers and travelers, funded by a very limited budget, derived from donations from individual supporters around the world. The local IT support is very limited: only ten of the largest offices can afford only one or two dedicated technical staff. The system we select and implement must be simple to install, operate and support, and be as compatible as possible to existing (software) installations. By adding centrally administered Linux-based servers in each of the LAN-based offices, while introducing an until then unknown TCP/IP-based information and communications system, we reduced the transitory impact to a minimum, and keeping the cost down.

1. Some history and background

1.1 Organisational structure

Despite its public image, Greenpeace's operation is not so much different from other medium sized multi-national organisations. It consists of some 1,000 full-time paid staff and at least as many volunteers, distributed over some 30-odd national offices, more or less held together by an international office: Greenpeace International - my employer. The national offices have a high degree of independence in legal, institutional and operational terms, but the smaller offices in poorer countries are cross-funded by the larger offices in richer countries, with the International office and board in the middle.

The International office performs mainly a coordinating function between the national offices, maintaining continuity and a common face to the outside world, and support the smaller offices where needed. Some typically international tasks are done in the International office, such as: standardizing and monitoring international campaign priorities, coordinating maritime support, acting in countries without a national office, providing press releases and videos to the international media, and of course providing international communications between offices and to the public.

1.2 International communications
see also http://www.greenpeace.org/history.html

In the sphere of NGO's (Non-Governmental Organisations), Greenpeace pioneered global access to email with the introduction of a PC-based BBS-based email system via CompuServe's X.25 network in 1985. Soon it became apparent that in order to provide a truly global system between so many offices and travelers, the email system needed to be contracted out to a commercial Boston-based firm, running DEC RSX-11 based software, interfacing with three international X.25 network providers: Infonet, BT-Tymnet and Sprintnet.

International email communications grew in importance in such an international organisation as Greenpeace, and access facilities to travelers (to conferences, protest action sites, etc.), home workers and ships was developed. All ships were fitted with equipment for PC-based email access via the Inmarsat satellite systems: either modemised access via voice channels, and/or with X.25 messaging via smaller text-only ship-based and mobile terminals.

In 1989, some of the user management, system customising and service provision was taken in-house into the hands of full-time staff (including myself) in the Greenpeace International office, and CompuServe was added as an additional X.25 network provider. In order to save some of the international dial-up charges, a 9,600 baud leased X.25 line was established between the Greenpeace International office and the Boston-based email provider, and the Amsterdam hub became an additional dial-up host in the CompuServe network.

Very soon, an interface to the public Internet seemed inevitable. Most of the NGO counterparts and scientific information sources were on Internet, and we needed direct connectivity with our private email system. This was first attempted in 1993 with varying success rates via our commercial provider in Boston and later their German sister company, and in 1994 it was finally established in our Amsterdam office premises. Not only did we have connectivity for email purposes, but we also experimented with a Linux-based Gopher and WWW server, commencing services in mid August and September 1994, respectively. Funding was a problem initially, so this first server was running on a surplus Intel 386 PC...

At various moments of that year 1994, strong questions were raised whether we should continue with the X.25-connected email system, running on both server and client software that became inadequate for our needs. As other NGO's seem to be overtaking us - particularly in user software - and gaining access via the apparently cheaper Internet, we needed to review our options.

1.3 User surveys and system review

As the first step in this project, several user requirement surveys were performed. We distributed some in-house designed user questionnaires via e-mail, and posted this in the BBS of the central Boston host for about 6 months. As some staff got the impression that we may be going off onto "pet projects", we approached some external consultants who did some preliminary research for us by questioning the Greenpeace International staff - from reception to executives. Additionally - and very opportune -, Bart van den Hooff, a student at the University of Amsterdam did some external research through e-mailed questionnairs, as published in his paper "Incorporating Electronic Mail"; ISDN 90-75727-72-0.

In summary, some of the most important requirements the users wanted, were:

1. A secure system, separate from other organisations;
2. World-wide access, with consistent interface (same username/password);
3. Provide methods to reduce e-mail floods;
4. Better information storage,-search-and-retrieval system;
5. More cost-effective (reduce connect time charges);
6. User-friendlier interface (GUI-based);
7. Multi-lingual;
8. Support for non-text file exchange;
9. Faster accessing and processing (in the background);
10. Compatible with existing intra-office e-mail systems;
11. Affordable to poorer offices.

This investigative process took about 6 months, and the summary of alternatives was presented to a meeting of IT representatives of the 10 largest offices was discussed in November 1994. It took three days to discuss the alternatives, agree on standards and select one of the three main alternative options of a new information system:

1. EasyLink: a turn-key proprietary system with AT&T;
2. An Internet "clone" (the term Intranet was not yet popular);
3. A Novell-based MHS/WAN system, and probably Lotus Notes as information exchange standard.

The second option would require the introduction of an unknown operating system into the organisation: Unix with TCP/IP as transport medium. All existing national office support staff were trained and experienced in MS and Novell IPX/SPX only.

The third option was attractive because it allowed file sharing on a known O/S, and Lotus Notes looked very 'sexy'. However, dial-in e-mail exchange and cross-protocol message exchange was still rather cumbersome, and the offices using DOS and Mac's would be in trouble. The overall cost was pushed up by the requirements to purchase Lotus software licenses and Windows-capable PC's. We thought at that stage that this would rule out acceptance by the poorer offices in countries with more difficult dial-up connections.

1.4 The selection

Finally, the choice was made on the "Internet clone", i.e. Intranet system, as it would allow remote maintenance by staff in the Amsterdam office on Linux/Internet-based and relative cheap and known Intel-based servers. The support staff in national offices did not need to learn Unix, but just the software capable of running Internet-type clients, which was available as shareware or freeware. Because the Linux server needed no local maintenance (besides keeping the hardware up), we dubbed these servers "Black Boxes".

The users needed to learn new client software, but there would be plenty of books, local training courses, computer friends and on-line help available on the Internet - which looked after all exactly like our in-house system.

At the same time, instead of being stuck to one or a few proprietary systems with software we had no control over, we could now base the server software on freely available source code, which we could tune to our particular needs, if required. Even more importantly: new software was being developed at a staggering rate, with new features we wouldn't even dream of when using proprietary sources. Compatibility between various operating systems and platforms was already the highest, and appeared to only increase dramatically.

1.5 Acceptance

A final proposal was distributed to key staff in printed form, but also through e-mail and BBS during early 1995, and a discussion was started within the organisation, with presentations of comparisons with Internet at appropriate occasions. The development of the public WWW site continued, and more and more staff got acquainted with its concept and practicalities. As a result of a special donation, a Sun-clone replaced the 386 as our public WWW server in March 1995. Eventually, the principle and budgetary approval for the new Intranet system arrived in mid 1995.

2 Implementation

The implementation from approval stage to an switch-over stage took about 8 months. First the networks, the main servers and interfacing needed to be brought into place, and then the introduction of new client software and adequate documentation, training and distributed support.

2.1 Network

The new network would consist of:

1. TCP/IP over leased Frame-Relay links between the ten largest offices which already had ethernet LANs;
2. World-wide dial-up facilities via the X.25 network;
3. Two Sun-clone main hosts at the two main hubs: Amsterdam and Boston;
4. Firewalled links to Internet at these hubs;
5. Eight local "Black Box" Linux servers in the offices who required them, fitted with dial-in modems in order to save X.25 dial-up charges.

Various international networks were compared in price, end-to-end installation options, and global coverages. Eventually, CompuServe Network Services was selected for both Frame Relay and X.25 dial-up, which at that time showed the most wide-spread global options at the lowest charges. They would install the Frame-Relay and X.25 lines from end-to-end, including the installation, configuration, around-the-clock monitoring and maintenance of routers in our offices. All we would need to do is plug in an UTP cable into the router, and we had another LAN-based office connected.

New X.25 dial-up connectivity was brought into both the Amsterdam office and the Boston contractor company, in the form of a new or modified X.25 switch and router. New X.121 host names were added for both PPP and Telnet hosting for each of the locations, providing geographic back-up and access cost optimisation. We were given access to the administration engine to maintain userbases on these new CompuServe hosts.Both the Amsterdam and Boston hubs would also be connected to the Internet via firewalls of the dual-homed "bastion" gateway type.
Figure 2 - Networks interconnections

2.2 Servers and clients

The functional concept would comprise of:

1. The international transport medium over the leased lines would be TCP/IP, which would - if required - be relayed and/or buffered through a Black Box Linux server in each of the LAN-based offices. These would be equipped with modems to allow SLIP, PPP, UUCP and Telnet (terminal) dial-in and optionally fax-out;
2. Two Sun-clone main servers were provided in Amsterdam and Boston to maintain the main userbase for staff in smaller offices without LANs and ships, using PPP and Telnet dial-up access via the X.25 network, and direct dial into a Portmaster and a ISDN router. These two main servers and all Linux Black Boxes would share user administration through NIS later on. All servers would provide local WWW, Gopher, WAIS, FTP, NTP, NNTP, Named, Bootp, IRC, Talk, PPP, SLIP, UUCP and Telnet/Lynx services;
3. Two dual-homed BSDI-based TIS firewalls would interface to Internet, which would also translate internal IP addressing (192.168.0.0 and a private top domain) to Internet addressing (xxx.greenpeace.org). Browsing would be made possible in proxy mode (either socks and/or application) and later on also transparent mode, and email addressing would be mapped in fixed translation tables. This method later allowed for all incoming and outgoing email to be scanned for known viruses;
4. The choice of LAN operating systems and client platforms were of lesser relevance, so long as they were TCP/IP-capable, running SMTP, POP3, NNTP, HTTP, etc. Some of the offices either already chose such clients before the introduction of this new Intranet, others followed the suggested standards as agreed in the November '94 meeting, either solely relying on the Linux server, or in addition to other methods;
5. In practice, most of the existing Novell based LAN servers would have a freeware SMTP gateway added, serving clients running both Novell and SMTP/POP3-capable freeware PegasusMail;
6. Installation software kits were created (by intern students) on diskettes with a collection of pre-configured freeware or shareware clients like PegasusMail, Netscape, Trumpet, etc., for DOS, Windows3 and Windows95 based PC's;

2.3 Installation processes

It was imperative that the new system would be developed and run side-by-side with the old system for some time, until all staff was changed over to and comfortable with the new system. Most of the infrastructure of the new Intranet was in place for almost 12 months until we finally decided to pull the plug on the old RSX-11 email system in April 1997 and basically forced everyone to move.

The hardest part turned out to be creating an client installation package (as described in paragraph 2.2.6 above). Without it, only a handful of dare-devils were willing to make the step to the new system. The users needed to have the new system provided to them on the proverbial silver platter. Obviously, the Amsterdam office with Greenpeace Netherlands and Greenpeace International was the first to be fully online - via ethernet, of course.

1. The first Frame Relay link (between Amsterdam and two separate offices in London, UK) was in place in mid November1995. All ten leased lines were in place in May 1996.
2. Both CNS X.25 dial-up hosts were available in July 1996;
3. Development of Linux (Black Box server) installation kit was commenced in October 1995;
4. A selection of standardised hardware for the Linux server was made: a Compaq Prosignia - being available world-wide, the cheapest model in the server range;
5. A the production version of the Linux install kit was installed onto the prototype in January 1996;
6. The user installation kits development was started in August 1996, and the main development was finished by January 1997;
7. The last office had its Linux server remotely installed in November 1996;
8. Shutdown of old system: April 1997.

As the old system basically consisted of hiring a slice of an out-of-house system, we could support the internal userbase and BBS layout with two or three permanent staff in two continents, who dedicated only part of their time to support and administrative tasks. The new Intranet would require a shift of hardware and software development towards "in-house" into the Greenpeace offices, most predominantly into the main hub of the Greenpeace International office in Amsterdam. We found the Boston company willing to mind the required extra equipment and the liaison with the international network companies, but we had to administer its "main" server and firewall remotely via the leased lines from Amsterdam.

During the proposal stage, we anticipated the requirement of:

1. Two full time Helpdesk staff;
2. One part time system admin;
3. One full time development;
4. One-tenth support staff in each of the ten national offices;
5. One full time temporary staff for a nine months installation period.

1. One full time coordinator, designer, system admin and Helpdesk;
2. One part time Helpdesk;
3. One full time user-trainer, handbook writer (from September 1996 onwards);
4. One contractor for system development and Linux support by email: in-house one day a week;
5. One representative from EACH office to act as non-technical "liaison officer", each of whom were called for a one-week in-house training session in February 1997;
6. Existing IT support staff in the ten largest offices to help install client software, and (optionally) maintain their own DNS databases and bootp tables;
7. Two student interns who would develop the user installation kits over six months full-time, followed by a 12 months one-day-a-week support (until end 1997). We are now contracting a software company to design a new version Windows95 and 98 installation and dial-up kit - unfortunately at a relatively high investment cost.

For some, the most interesting topic in this process may be the role that Linux plays, and why it was chosen. The mix of operating systems in the various offices were already creating apparently insurmountable complications, and strong voices were raised which advocated a reduction in the number of them, rather than an expansion.

3.1 The lead-up

Before the start of the project, no-one of the IT support staff had any useful experience with Unix. If there were any Unix systems at all, they were running for specialist financial packages, maintained by their supplier. Traditionally, offices were equipped with DOS (in various tastes) and Novell only, although a couple had introduced Mac's. Internet became available to the wider public through so-called FreeNets, an equivalent of which called De Digitale Stad (The Digital City) was introduced in Amsterdam in early 1994. We brought some of our campaign material to the public in Gopher menus and searchable with WAIS in February 1994. Administrative access was done through FTP and Telnet under SunOS, which was my first acquaintance with any taste of Unix.

After a while it became apparent that is was time to create our own and independent Gopher (and later WWW) server. However, we had no money to buy a Sun like DDS had. I approached the Antenna group, the Netherlands representative of APC (Association of Progressive Networks). They were very keen to help us, and suggested Linux as an affordable means of building an Internet site on a left-over ex-financial database server: a 20MHz PC386 with 20MB RAM and a 200MB hard disk.... It blew my mind that this was possible...

Software support, in the rare occasions when it was needed, was provided mostly remotely from the Antenna group, which indicated to me how this concept could work within our own Intranet. Even this early version of Linux proved highly reliable.

However, this particular installation was done by carrying the PC to Antenna, and have them install and configure the server to our requirements. This solution seems impractical for the servers we required in our offices around the world. We would have to purchase the hardware locally in each country, and hence my idea was to send a tape and installation diskette with individual settings for a particular office from Amsterdam.

3.2 Making the choice

The idea to use freeware and shareware seemed to fit a non-profit organisation well. One of the most frequent complaints from our world-wide staff included that software was deemed costly, particularly in the lower-income offices. We had already received offers from several shareware developers to waive registration charges, and even modify their software specifically to Greenpeace's needs.

We saw as main advantage that we could pick-and-choose such software from the multitude of on-line sources, and that in practice the support was not a problem: the newsgroups provided a good source of help, and in many cases we received quick responses from the developers directly.

Besides it being free of charge, the main advantages of Linux to us were that the sources was open, and could be adapted to our needs as required, and that debugging would be possible without the need to wait for a large distant company to finally do something about it. In practice, it had already shown to be highly reliable as server operating system. The server applications appeared also more configurable than its commercial equivalents.

The main draw-back was that we had insufficient in-house knowledge to administer and develop under Linux efficiently. Even though we had occasional assistance from Antenna, that organisation was in extreme high demand by NGO's and their resources thinly spread.

Fortunately, a consultancy firm X/OS (www.xos.nl) was found in Amsterdam who was willing to produce a menu-driven Linux installation package for us, and provide ongoing contracted support by phone, email and dial-up, as required. Eventually, we agreed on one day a week in-house maintenance and development contract.

As we standardised the hardware of the servers Linux was to be running on, bringing up each of the Black Boxes was a matter of an hour or two in each case - mainly determined by the speed the tape could be read in. The menu-driven installation procedure was so easy that it could be done by a non-technical person, and we logged in remotely and changed the root password within seconds. The local staff (or the hardware supplier) had nothing else to do then to ensure that the jumpers on interface cards didn't produce IRQ conflicts. No Unix knowledge was required on their part, and all of these servers are running untouched for several years, unless required to move the hardware or for other non-software reasons. The uptime of the Swiss Black Box turned out to be over 300 days by now; the date of the last kernel update in order to facilitate a non-standard multi-port serial card...

3.3 Other versions of Unix

The firewall was ordered from a specialist company Tunix (www.tunix.nl), which provided the hardware and the BSDI-based modified TIS software for the Amsterdam host. The Boston firewall was installed on an equivalent PC bought locally, and uploaded and configured via the Frame Relay circuit. Having yet another taste of Unix seemed inevitable, but acceptable, as this is such a specialist application, in which very little software development could be performed by us anyway. If there were rare cases of problems, we called the supplier for help...

The "main" hosts in Amsterdam and Boston are Sun clones, as we thought in 1994 that Linux on a PC may not yet be up to such a heavy duty task. After many years of proven service and fast improvement of Intel hardware, we are starting to change our minds about that. However, luckily there is very little operational and administrative difference between SunOS and Linux, and X/OS is porting many server applications for both systems.

4 Developments

4.1 Recent developments

4.1.1 Small offices

Soon after the infrastructure was more or less in place, several smaller offices expressed a desire to use some kind of shared dial-up connection. The charges for a leased line were too high for offices with less then about 25 staff. The combined dial-up charges of all of the staff together were so high that sharing seemed attractive, particularly if direct phone lines and modems could be saved.

The Linux Black Box concept would in fact be able to provide such facilities, and with an extra dial-in modem could also be maintained from Amsterdam. With a quick international phone call, we could connect and establish a dial-up connection via the CNS X.25 network, after which we could continue to do the maintenance via that lower cost connection.

One particular office (Belgium) has been set up like that, but in fact via the ISDN dial-up connectivity provided by CNS in that country. Because there would also be X.25 connect time charges incurred, and considering the higher cost of a Compaq Prosignia, that office's choice was to not bother with any of the WWW, FTP and other hosting services possible under Linux. That version Black Box, a simple Pentium with a plug-in ISDN card, would act as packet-filtering automatic dial-up router via CNS in case any client would require Web browsing. Email would be delivered from the Novell host to a queue on the Black Box server, which was crontab timed to connect at regular intervals to the Amsterdam main host, where also the incoming email queue would be triggered. No userbase is required on this "mini" version of the Black Box concept.

4.1.2 Ships

Another case would be our ships. They also have small ethernet networks onboard, but no Novell server. Thusfar, one PC in the radio room dialed through an analog modem and Inmarsat voice channels and exchanged email via SMTP and POP3, although we also experimented briefly with UUCP. Messages may be passed on between radio operator and crew via diskettes or more recently Windows95 file sharing.

A recent prototype Black Box server was installed onboard one of the ships with Inmarsat-B with the HSD (High Speed Data) option at 64kbps. A small and relatively cheap interface called S0DA from 7E-Communications converted that into ISDN connectivity with the plug-in ISDN card in the Black Box. Like the Belgium system, the server acted as IP router for all other PC's onboard, but it would not dial automatically. Due to the high Inmarsat charges (about $10-12 per minute for HSD), the dial-up connection is fully under the control of the radio operator.

This Black Box server is fully equipped to provide all Intranet services, such as on-board WWW information services, FTP mirroring, newsgroup feeds, and of course batched SMTP (like the Belgium server). While the radio operator connects for SMTP and newsgroup exchange, the onboard clock is synchronised through NTP and the named tables are updated. During such an SMTP session, the ISDN capacity often allows simultaneous WWW browsing - both on the Intranet and on Internet. An off-line browser package under Linux called WWWOFFLE saves retrieved web pages for onboard viewing by other PC's onboard, and queues page requests for future automatic download during the next dial-up connection.

Email is stored on the Black Box into standard login names (like "captain", "dh1" and "dh2" for deckhands, etc.) which don't need changing after crew changes. Email is retrieved from and posted to the Black Box via SMTP/POP3 with Windows clients on the other PC's, and old mail is saved in private directories on the Linux Black Box using Samba.

Once configured, these Linux-based servers require very little remote attendance. Crontab takes care of house-keeping jobs like logrotate and nightly tape back-ups.

4.2 Future developments

The greatest obstacle to wide-spread use of browsing and other interactive tools like newsgroups in smaller offices remains the X.25 dial-up charges, and the main obstacle to increasing the number of permanently connected offices remains the high lease charges. For a large part, we remain stuck in the email and mailing list culture that has traveled with us from the old centralised X.25-based system. A sharp division remains between "rich" and "poor" offices in the practical availability of on-line search tools, etc., even though technically it is provided.

The obvious solution would be to allow these users to access via their local Internet provider. This was not yet an easy world-wide option in 1994, but now it should be practical for almost all locations where our organisation operates. However, the greatest drawbacks are:

1. The variations in access facilities (modems, ISDN, cable) and charges of ISP's, negotiation of which would have to be done by local staff, who may have troubles understanding the technicalities;
2. Client software provided by these ISP's will interfere with our Intranet preferences;
3. Dial-up access is usually only local or national, and therefore unusable for world-wide travelers. Some ISP's now provide such an optional service, but at a considerably higher charge;
4. It would be impossible for our Intranet Helpdesk to support staff with unknown ISP software;
5. The lack of security on Internet;
6. The lack of control over bandwidth capabilities and response times.

A next generation firewall is being installed to offer the encrypted client connectivity, and installation and configuration of the VPN solution between Black Box servers is in its final stages. In addition, these Linux Black Box servers would be fitted with transparent firewall software, allowing these offices to browse Internet directly through the same local ISP connection, rather than through the remote firewalls in Amsterdam or Boston. The hardest part will be the negotiation with ISP's on adequate permanent connectivity to Internet at each office, without ending up with a fully-fledged and fully-chargeable commercial Internet WWW host service. Dial-up via modems or ISDN would have to remain an option.

5 Conclusion

The choice of an Intranet has proven to be the right one, considering the major shift of the world towards Internet over the last 4-5 years. We flow with and reap the benefits of the hardware and software development stream on The Net. Using a single network provider with end-to-end leased line connectivity made our life a lot easier than having each office try to sort out connectivity on their own. We could offer consistent leased line services to the larger offices, and a standardised login procedure for the smaller offices and stand-alone users. Servers running Linux offered the gateway to and support of TCP/IP-based services into offices which never had to deal with it before. Standardised installation procedures and centralised maintenance has proven highly practicable at relatively low hardware and software charges, and the reliability, applicability and flexibility of Linux-based servers has exceeded our wildest dreams.