Federal Science eLibrary Pilot Project Final Report

Methodology and Results

Project initiation

In spring 2005, the Strategic Alliance approved the Federal Science eLibrary pilot project. NRC-CISTI funded and managed the project on behalf of the Strategic Alliance and assembled a project team that included a project manager and a full-time pilot project coordinator. Team members began work in late summer 2005 to develop the project plan, secure publishers' permission to extend access to their content, prepare and test NRC-CISTI's infostructure and enlist volunteer pilot sites.

The pilot project scope extended access to NRC-CISTI's infostructure to three regional pilot libraries at two Strategic Alliance departments for a period of three months.

Participant selection

The project team chose pilot sites from the Strategic Alliance's member libraries. To properly test the technical infrastructure, the selected sites had to be sufficiently distributed across Canada. The selection criteria included:

• a sizeable but manageable user community of 50 to 500 STM personnel;
• existing modest access to e-journals from users' desktop so that the impact of expanded access could be evaluated; and
• an enthusiastic and involved pilot site librarian.

Strategic Alliance members put forward suggestions for test sites. There was a great deal of interest in the pilot, making the recruitment of volunteer sites relatively straightforward. Three pilot sites, from two Strategic Alliance departments, closely met the selection criteria and were chosen to participate:

1. Northwest Atlantic Fisheries Centre in St. John's, Newfoundland and Labrador (Fisheries and Oceans Canada)
2. CANMET Energy Technology Centre in Devon, Alberta (Natural Resources Canada)
3. Institute of Ocean Sciences/Pacific Geoscience Centre in Sidney, British Columbia (a combined Fisheries and Oceans Canada and Natural Resources Canada site)

The user population totalled 505, including 300 at the Institute of Ocean Sciences/Pacific Geoscience Centre, 55 at the CANMET Energy Technology Centre-Devon, and 150 at the Northwest Atlantic Fisheries Centre Library.

The Institute of Ocean Sciences/Pacific Geoscience Centre pilot site in Sidney, British Columbia, was of particular interest as it houses both Fisheries and Oceans Canada and Natural Resources Canada. Both departments share the same library site and staff; however, library budgets, access rights to e-journals and departmental networks are separate. The Natural Resources Canada section of the library has much better access to electronic resources than the Fisheries and Oceans Canada section, making it an especially interesting location to study the effects on users of unequal access to e-journals.

Access rights to licensed content

NRC-CISTI's infostructure consists of a digital content repository, a secure technology infrastructure, search tools and tools to manage repository content. To offer the pilot sites access to its infostructure, NRC-CISTI negotiated with three publishers (Elsevier, Springer and NRC Research Press) whose content was part of its electronic repository. The successful negotiations resulted in the publishers extending access to their content (over 4 million articles) for the three months of the pilot. NRC-CISTI also successfully negotiated with the vendor Endeavor, who extended access to its Journals OnSite ⁶ system to the pilot sites for the duration of the pilot.

Information infrastructure preparation

At the outset of the pilot, NRC-CISTI's infostructure had already been in development for several years. While it began as an internal NRC resource, its development was directed toward ultimately offering seamless and permanent electronic access to full-text digital content to all Canadians, regardless of their geographic location or affiliation. Given the existence and relatively mature nature of NRC-CISTI's infostructure, it was seen as a natural host platform for the Federal Science eLibrary pilot.

Because the pilot sites used two different types of networks ⁷, investigations were undertaken to determine how users would connect to NRC-CISTI's infostructure and to assess the risks and potential roadblocks to connectivity. These investigations showed that access to the NRC-CISTI infostructure worked equally well from both networks. Connectivity was good from the outset and the few issues that existed were easily and permanently fixed.

Once the basic connectivity was confirmed, access was opened to all pilot users. Access rights to the digital repository were granted based on the originating IP address. Users were given the URL for the Federal Science eLibrary Pilot Project customized homepage to allow them to connect directly to the NRC-CISTI digital repository. The digital repository site was monitored daily for availability and performance. Small periods of downtime occurred but were efficiently handled by alerts to the pilot sites and quick response by supporting IT staff.

As-Is analysis

An As-Is analysis ⁸ was performed to better understand and document the state of the pilot sites at the start of the project. This analysis set up a baseline against which changes could be measured in pilot users' work patterns.

Results of the As-Is analysis are based on findings captured through a Needs Assessment Survey (Appendix A), Pilot User Feedback (Appendix B) and a site visit (Appendix C).

Users and librarians at all three pilot sites expressed similar opinions regarding access to e-content and how it could positively impact their work and productivity.

Remaining current in chosen field

Pilot users indicated they lacked the time to keep up-to-date with the literature in their field. Pilot users were dissatisfied with the number of hours that searching for and obtaining articles took away from their lab work and other important professional activities. They recognized that increased access to electronic journals could mitigate this problem.

• Our facility's library no longer receives paper journals, so access to relevant journals requires a 45-minute drive to the University of Alberta. Requesting photocopies of journal articles through the library system involves a delay of several days. Thus desktop access to e-journals significantly reduces wasted travel time and waiting time.
  Natural Resources Canada - Devon - Survey I
• Because the library does not have a huge budget or space to carry a lot of journals, access to electronic has cut the waiting time for articles, which could be a week or more through interlibrary loans.
  Fisheries and Oceans Canada - Sidney - Survey I

Meeting tight deadlines

Pilot users explained that many of their professional activities, such as the preparation and peer review of manuscripts, demand tight turnaround times. Users felt increased access to e-journals might address the difficulty they have in meeting these deadlines when long waits for information are a factor.

• Since scientists like myself are peer reviewers in our field, quick access to references are very useful for turnaround time.
  Natural Resources Canada - Devon - Survey I

Meeting library client expectations for e-content

Librarians identified difficulties in fully meeting their clients' need for quick and easy desktop access to e-content. Librarians indicated they receive questions regularly from visiting students and researchers, who have high expectations for access based on their previous work experiences in the academic or the private sectors and are disappointed by the comparatively limited access in the government.

• Without easy access to e-journals, at a research level, we are 10 years behind every university and the distance grows every month.
  Natural Resources Canada - Devon - Survey I
• I have worked in US government laboratories and in universities in both the US and Canada. Electronic library resources in the GOC are not as good as enjoyed by our foreign and academic peers. The only reason appears to be money. GOC needs a coordinated strategy to negotiate with publishers so that government scientists across all disciplines have access comparable to our academic peers.
  Fisheries and Oceans Canada - Sidney - Survey I

Librarians and researchers were concerned that students and visiting researchers would not rank the federal government highly as a place to seek permanent employment because they considered lack of access to be a significant liability.

• Students ask about access to journals they are accustomed to using electronically through their universities. When I tell them our access is limited, I notice their surprise and disappointment. It leaves me wondering how interested these talented young people will be in working at federal research facilities such as this one.
  Fisheries and Oceans Canada - Sidney - Telephone interview

Equal access to e-journals for all areas of study

The pilot libraries serve researchers from several disciplines. Pilot users felt journal purchases (electronic and paper) were directed toward the fields with the largest number of researchers. As a result, they felt researchers in less common disciplines had access to very few e-journals in their areas of study. Many users expressed a desire to see more equitable access to e-journals for all government science workers, regardless of area of study or department.

• E-journals allow access to journals that may be of interest to some that work in an area but that may not be the primary focus of the institute overall, and will therefore not be available in their own library.
  Fisheries and Oceans Canada - Sidney - Survey I
• I am a seabird biologist (employed by Environment Canada) but located within a DFO facility. Because of network issues, I am on DFO's network - this unfortunately has prevented me gaining access to certain e-journals (available if I was on the EC network). If the goal of the Federal Science eLibrary is to serve ALL scientists employed within the Federal family, I am hoping that we will be able to access all journals - not just those that are selected 'in house' by scientists within a particular department.
  Fisheries and Oceans Canada - Sidney - Survey I
• Users at the Sidney, British Columbia, site expressed frustration about the inequality of access at the two co-locations (Fisheries and Oceans Canada and Natural Resources Canada). One Fisheries and Oceans user said:

  To need information, and know it is available on the desktop of your colleague at the end of the hall, but not to you because you are in a different department is ridiculous. This problem must be addressed.

Access to information while working in the field/remote locations

Researchers who conduct their research from the field, a small branch office or from home explained the difficulties they had in accessing research materials. They can go for months without access to current literature. They often rely on third parties to conduct literature searches and send them materials. They viewed the lack of first-hand access to current literature as a significant disadvantage to doing quality, competitive research while in a remote location.

• It [e-content] allowed me to improve my ability for working when I am on the go. Typically there is a maximum amount of actual paper that you can carry when traveling. Being able of storing in my laptop the pdfs of many more articles makes [it] much easier to carry with you all you might need to work far from the office (i.e. not only traveling, but working from home during weekends or snow days).
  Fisheries and Oceans Canada - NFLD - Survey I
• I work in the field for long periods and can't access any literature during this time. The most important thing about the Science eLibrary for me would be to have access to it while doing fieldwork. Do you think that will be possible?
  Fisheries and Oceans Canada - Sidney - Site visit discussions with users.

Quality of graphical reproductions

Users needing articles with detailed images appreciated the higher quality reproductions available in full-text PDFs compared to the quality available in scanned or photocopied materials.

• Printed copies of e-journal articles have far better reproduction than photocopied paper journal articles.
  Fisheries and Oceans Canada - Sidney - Survey I

Users also identified two areas where they were concerned e-journals might negatively impact their work:

Usability of computer screens for reading journals

There was some concern expressed that paper versions might disappear altogether leaving users with the less-attractive and only option of reading online.

• I believe that e-journals are grossly over-rated. I do spend time downloading papers now that I then print on a local printer so that I can read them. I do not find it at all effective to read journals on-line; I must print them first. I would say that the impact is small and negative. I will continue to visit the library to read print copies. This is likely more effective and cheaper than printing them on a local network printer and then reading them.
  Fisheries and Oceans Canada - Sidney - Survey I

Preservation of journal articles

Pilot users worried about the long-term preservation of electronic materials and the potential for permanent loss of information in their field.

• While e-journals are very useful, I am deeply worried that they are ephemeral. I use research which is decades-old as much as recent research. If we stop archiving printed journals, what access to today's research will we have in 20 years time?
  Natural Resources Canada - Sidney - Survey I
• While this is an excellent way to do literature searches and stay abreast of recent developments, it is also important to retain the older journals, and not convert completely to an electronic format that may not be accessible in 20 years.
  Fisheries and Oceans Canada - Sidney - Survey I

Communication activities during pilot

During the three months of the pilot, users and librarians had easy access to help, as well as the tools and opportunity to submit feedback at any time.

The pilot coordinator worked to ensure information exchange and follow-up with the pilot site librarians and the user communities. Pilot site users were encouraged to communicate their experiences with the site. Reactions were captured via a "Contact us" email address and informal telephone calls.

The majority of user questions were submitted to the pilot site librarians, which highlighted the importance of having knowledgeable professionals at the service delivery points to facilitate participants' use of the site.

Impact evaluation

Data collection for and the results of the evaluation of the impact of expanded access to digital content on pilot users and librarians' work are described in detail in the following Appendices:

• Appendix B - Pilot User Feedback (includes all general feedback methods)
• Appendix D - Impact Survey
• Appendix E - Pilot Site Librarian Feedback (includes Librarian Questionnaire results)
• Appendix F - Remote User Feedback (includes Remote User Questionnaire results)
• Appendix G - Usage Statistics

Users and librarians at all three sites expressed similar opinions on the impact of the pilot on their work. Close to 80 percent of Impact Survey respondents stated access to the Federal Science eLibrary pilot site had positively impacted their research activities and productivity. As well, 92 percent of the Needs Assessment Survey respondents felt the impact of increased access on their research activities and productivity was positive or very positive.

More specifically, users and librarians identified the following are areas where increased desktop access to e-journals had clearly benefited them.

Improved ability to remain current in chosen field

Users felt desktop access to e-journals facilitated their ability to keep abreast of literature in their field, as well as allowed them to read more widely into related areas of study.

• Instant accessibility to articles saves time; ability to read more widely.
  Geological Survey of Canada - Sidney - Survey II
• I was able to locate and read two articles which I had not been aware of during searches through physical copies of journals.
  Fisheries and Oceans Canada - St. John's - Survey II
• Efficiency improved - saved time - keep up-to date easily.
  NRCan - CETC -Devon - Survey II
• It saved me a trip to the UVic library & provided rapid access to articles needed in preparing reports.
  Fisheries and Oceans Canada - Sidney - Survey II
• Immediate access to journals saves a great deal of time and allows greater efficiency. It also encourages me to check a wider amount of the pertinent literature.
  Natural Resources Canada - Sidney - Survey II

Improved ability to meet tight deadlines

Users found increased access to e-journals helped them carry out professional activities such as lab work, manuscript preparation and peer reviews with fewer interruptions to work flow. Increased desktop access allowed users to find materials immediately without leaving the office.

• When looking up some references for a manuscript it was very useful to be able to find them on line rather than go up to the library where they may or may not be and copy or ask for a copy from some other library.
  Fisheries and Oceans Canada - Sidney - Survey II
• Instant access to new and old journal articles has resulted in less time searching and more time for writing papers and reports
  Fisheries and Oceans Canada - Sidney - Survey II
• A problem arose in a research project: I went to e-journals, searched for relevant papers and in minutes had what I needed, as opposed to the couple of days (minimum) it would normally take.
  Natural Resources Canada - Sidney - Survey II
• I needed to update some information for an internal presentation and was able to retrieve the data from the eLibrary.
  Natural Resources Canada - Devon - Survey II

Improved ability to meet library client expectations for e-content

Pilot site librarians observed positive reactions to increased desktop access from their clients. Many library clients asked if the service could be continued past the pilot end date.

• They [pilot users] were very happy to be able to access much needed science journals to assist with their work. In fact, they were unhappy to see it end.
  Fisheries and Oceans Canada - St. John's - Librarian Questionnaire

Improved access to information while working in the field/remote locations

Remote pilot users found increased access to e-journals had a positive impact on their ability to keep current and find materials needed during their fieldwork.

• When I started working here I could not believe the government did not make it easy for researchers to obtain scientific papers. It seemed very dark ages. We are a science institute that is not as up to date [as] the universities we had to go through to get here.
  IOS - Sidney - Remote User Questionnaire

Improved equality of access

The pilot site librarian in Sidney, British Columbia, reported that the Fisheries and Oceans Canada users, who prior to the pilot had less access to electronic journals than the co-located Natural Resources Canada users, had been especially pleased with the increased access offered during the pilot. The major users of the pilot site in Sidney were from the Fisheries and Oceans group. During the site visit, Fisheries and Oceans users stated they saw the potential for a Federal Science eLibrary to address the problems of unequal access within the Institute.

Environmental Benefits

Several users expressed their preference for reading online and commented on the environmental benefits (reduced paper usage) of electronic access.

• Literature reviews in which rapid elimination of papers saves much photocopying and paper wastage. Also, saving papers electronically is much more efficient in terms of storage and paper usage.
  Fisheries and Oceans Canada - Sidney - Survey II

Pilot users registered many general comments, expressing appreciation of the pilot site and interest in continuing to have access.

• Please extend the pilot for the year!
  Fisheries and Oceans Canada - Sidney - Survey II
• This is a great initiative. It will be extremely useful in the long run.
  Fisheries and Oceans Canada - St. John's -Survey II
• This e-library electronic journal access service is great. Very useful in finding info & research, especially from a regional DFO office perspective. Also saves substantial time in searching for information while providing highly relevant search results. I have used this service over the past number of days and I think it is a great work tool & departmental asset. Only issue is that the service times out after 20 minutes or so.[*] Not a real problem. Hope this service will be widely available for future continued use. Thanks.
  Fisheries and Oceans Canada - NFLD - "Contact Us" email

  *The infostructure interface times out after 30 minutes of inactivity. Once this occurs, users must log back onto the infostructure to start searching again.

---

⁶ "Endeavor's Journals OnSite is an enterprise-level system that enables libraries to locally store, search and browse electronic journals from any publisher in one location. With Journals Onsite, patrons can search and browse through all of the electronic journal content licensed by a library and be taken directly to the full-text article. Journals Onsite also enables library staff to centralize the management of all their full-text electronic journal collections." (From an online brochure at http://www.endinfosys.com/prods/brochures/RB/full_jonsite_FINAL.pdf ; accessed March 2006.)

⁷

• Northwest Atlantic Fisheries Centre in St.John's, Newfoundland and Labrador - public Internet
• CANMET Energy Technology Centre in Devon, Alberta - SCnet
• Institute of Ocean Sciences in Sidney, British Columbia - public Internet
• Pacific Geoscience Centre in Sidney, British Columbia - SCnet

⁸ "As-Is" analysis is done to study and document the current state of a business environment and/or process. This snapshot of the environment at a certain point in time can be used either for activities such as performing business process engineering or, as in this case, as a point of comparison to measure change in the environment after the introduction of new processes.