Final Reports


2010 Final Report

NDNP Awardee Final Report
(July 1, 2008 – June 30, 2010)
NEH Award Number: PJ50038-08
Project Name: Washington State Digital Newspaper Project
Awardee Institution: Washington State Library
NDNP State: Washington
Project Director: Marlys Rudeen
Project Manager: Laura Robinson
Report Date: September 30, 2010

Project Activities

Project Administration

Upon notice of the National Digital Newspaper Program (NDNP) award, an internal project team at the Washington State Library (WSL) was created, a search started for a project manager, and a request for proposal submitted to select a digitization and conversion vendor. By November 2008 a project manager was hired, an advisory committee convened, space for the project was arranged at the University of Washington (UW) and WSL, a vendor was selected, and equipment was purchased.

Throughout the grant period the project director and project manager attended the annual awardee meetings and turned in interim project and fiscal reports. A production schedule was created and batches delivered throughout the project. After communication from the Library of Congress (LC) regarding the upload and availability of newspaper pages, news was announced via the Washington State Library blog and press releases for each title were sent to media, the advisory committee, museums, schools, and other interested parties to announce the availability of new content on Chronicling America.

Due to state budget reductions, the project team lost two Digital Project Librarians (one at the beginning of the grant and one half way through), both of whom had a total of 13 hours a week devoted to the NDNP grant. While this likely affected the speed of QA and some coordination with CONSER catalogers, QA time was made up by help from other WSL staff, and the refining and automating of much of the QA workflow with batch scripts.

The titles were extensively researched before and during digitization. A list of titles, their lccns, and dates of coverage were delivered to NDNP staff at LC. Information about each title, including the condition of the film, where the film was stored, which regions the film covered and an estimated number of pages were compiled on the project wiki: http://wiki.sos.wa.gov/ndnp.

WSL and UW worked with their film service bureaus to check availability of master negatives and perform a technical analysis of the film, testing density and resolution. The content of the film was also analyzed to ensure proper coverage of dates and minimize running into large and unexpected issue gaps.

A schedule for production was created around three major milestones of the project (July 31, 2009; Dec. 31, 2009, and June 2010) estimating shipments 1 batch (18-20 reels) every 6-8 weeks. The first sample reel was delivered to WSL and was tested and validated using the DVV. On preliminary QA, some issues were found with the sample reel but resolved in time to send the sample reel to the Library of Congress by February 1, 2009.

Some production delay was caused by the adoption of new technical specifications for metadata and changes to the Digital Viewer and Validator (DVV). With each change of the DVV, the scripts used to create the metadata had to be changed and tested by our conversion vendor, Content Conversion Specialists (CCS).

Communication with the scanning and text conversion vendors was handled via an MS Sharepoint portal hosted by Online Computer Library Center (OCLC), the vendor hired to scan the microfilm. The project manager stayed in communication with the vendors and participating partners through the details of the project such as testing, duplication, and scanning of film; the delivery of the images and conversion of the images to text; the delivery of batches and rework reports; and finally communication of accepted batches. Because of the number of organizations involved and number of time zones and countries, communication and delivery often felt cumbersome.

A large part of project administration was a basic understanding of the technical specifications of the grant. This understanding was essential in making communication between the partners possible as well as understanding why there were delays in production. It was also much easier to create a more efficient workflow with a better understanding of the validation process and the technical specifications.

Title Selection

A large selection of titles were originally researched and vetted by a Selection Advisory Committee. The preliminary list consisted of 94 titles researched and submitted as possibilities for scanning. The committee reviewed the list for intellectual content such as the coverage of political, economic, cultural history, religious, and ethnic events from 1900-1910. Other considerations were geographic coverage and titles that served as the “paper of record” for their regions. Final ranking resulted in a final list of 23 titles. The final list reflected titles of historical importance, geographically diverse coverage, and titles that had a complete (or nearly complete) run of issues filmed for the time period selected.

In the case that some film would have to be passed over because of bad film found during technical analysis, more titles were picked as substitutes. Four related titles; Ranch, Ranche and Range, Ranch (Seattle), and Washington Farmer; were added to the final list that did not appear on the preliminary list. These titles were filmed as part of another grant after the preliminary list was made. They added important content about the business of agriculture and life in the Northwest during the time period.

Microfilm Evaluation

While it was beneficial to first check the positives for technical targets, reduction ratios, etc. it was not always possible to confirm the quality of the negatives this way. Since the large majority of the master negatives of the selected reels were housed by service bureaus, technical analysis was conducted after our initial selection and during the creation of the second generation masters. During technical analysis, some film was deemed of lesser quality, but no titles were completely omitted from the final list because of too high reduction ratios or bad resolution.

As a project partner, UW was charged with ordering its own film while WSL ordered the creation and testing of its film through Proquest and the Washington’s Office of the Secretary of State’s Imaging and Preservation Division. The master negatives of the titles selected for review were tested to determine film density, resolution, and reduction ratio. This testing cost between $20-$40 per reel depending on the vendor. WSL staff compiled any technical data within their abilities (e.g. identifying reduction ratios from the positive before title selection).

In 2009 the NDNP Technical Specifications changed making the capture of much of the reel technical analysis (e.g. density readings, resolution, etc.) optional. WSL determined that the readings coming from different vendors on film that didn’t have targets were too subjective to be very helpful or meaningful. So much time was lost trying to get meaningful readings and educating vendors about the specifications that we didn’t feel it was worth the time and money involved in capturing data from non-targeted film.

The project manager, working with UW, hired a student from the Information School and together they evaluated the microfilm and captured title, reel and issue data on a spreadsheet using Google Docs. Google Docs was chosen because of its use of versioning and the easy access between locations and institutions. The data was imported into an Access database and delivered to OCLC. The data was also used for WSL’s QA processes which will be discussed in further detail in the Quality Review section below. The Google Docs spreadsheets were available to view online and were used to calculate and display reel page totals and project progress from the project wiki.

Scanning and Text Conversion

A vendor was selected to do the scanning and text conversion through a competitive Request for Proposal process that followed the rules and guidelines of the State of Washington. OCLC Preservation Service Center, which later became Back Stage Library Works (BSLW), was selected as the vendor with the most experience and proven record of working with microfilm, preservation quality scanning, and producing digital files to the NDNP specifications.

Negatives duplicated by other vendors were shipped to OCLC, after which the film was scanned. The images were then shipped to CCS, the sub-contractor hired by OCLC to process the images, create the OCR files in the ALTO format, create the derivative images and the embedded metadata in each image file using the DocWorks software, a workflow management product. The batches were validated and shipped back to WSL where validation was verified and data and image quality were checked again. Error reports were shared with CCS and often data and images were either reworked and shipped to WSL via a second hard drive or delivered over ftp. When possible, WSL staff then reintegrated the reworked data and images into the batches, re-validated them, and shipped them to LC.

Quality Review – After a general assessment of the condition of the film, the images and data went through a three part screening process. Each newspaper page was first viewed as positive microfilm (a.k.a. service copies) by the project assistant at UW. Much of the reel and issue data was captured during this process. The data was then reviewed by the project manager and delivered to OCLC/CCS. Each frame was again reviewed during scanning and creation of derivatives, while the data was used to collate the digital images and populate parts of the xml and image headers. Once the batches were delivered to WSL, their validation was verified. The images were again reviewed by staff at WSL (each JP2 and PDF is viewed in thumbnail form) and the output data tested against the input data by batch scripts written by the project manager.

Delivery

At this point 100% of the data and images have been delivered and accepted by LC. All essays have been researched, written, delivered to, and accepted by NEH staff.

Accomplishments

Washington’s participation in the National Digital Newspaper Program accomplished goals and initiated further plans for future development of digital collections. 1) It produced files for over 100,000 pages of newspaper content that met the NDNP standards and specifications for deposit with the Library of Congress. 2) It initiated a move to create access to a state repository of digital newspaper content for Washington papers. 3) It has helped create a set of technical best practices to convert newspaper microfilm to digital formats that will serve as a model for future digital newspaper projects in the state. 4) It has successfully demonstrated how an organized, collaborative statewide approach achieves more consistent, sustainable, and cost effective output of digital files for historical research.

The project has made available pages of newspapers around the state from a significant period in Washington’s history. NDNP has also helped extend the access and efforts of the US Newspaper Project in Washington to microfilm and catalog newspaper collections. It created an Advisory Committee of representatives and experts to help achieve the long term goals of the project. So far the Committee has accomplished an extensive survey of extant historical newspaper hard copy held by heritage organizations and libraries statewide as well as reviewed a preliminary list of Washington newspaper titles that met the criteria for inclusion in NDNP.

Another development that has been initiated but remains ongoing is the use of the Washington State Digital Archive’s methodology for long term preservation of digital images and data. Staff has begun the first steps of creating a digital archive of newspaper resources that will be maintained and housed by the Digital Archives, with oversight and user interface development provided by WSL and agency web staff.

Audiences

The main audiences for this project are the historical research professionals, students, teachers, and family researchers. While our goal was to cover major population centers of the state we were also especially aware of regional researchers interested in areas of the state where the newspaper comprises one of the only records of community life available. This is especially true in more rural populations of our state. During this grant period, we worked diligently to include papers from different geographical regions of the State as well as papers that provided a recognized source for special audiences and interest groups. For example, papers that covered labor organizations, agriculture and farming, and the first successful African American newspaper in Seattle were selected.

Evaluation

WSL has been happy with the success of the grant and the learning and experience gained. Over 100,000 newspaper pages were digitized from microfilm and nearly all of those pages have gone live on Chronicling America. Researchers are able to view newspaper pages online for free that they would have otherwise had to travel to WSL or UW to research. There has been a steady increase in page views in the year our newspapers have been live. We’ve started to combine usage statistics to track of our local newspaper products (Pioneer Newspaper Collection) and the newspaper titles for NDNP. With the issuing of targeted press releases for each featured region, staff have seen some press coverage and received positive feedback from local community members and organizations, including a desire from several rural areas to have their local papers digitized.

Evaluation of the project workflow for the past two years led to changes in the work plan for the second grant proposal. Since the vendor used a sub-contractor, project staff was often trying to convey complex questions to, and transfer data between, staff in OCLC/BSLW’s office in Pennsylvania, CCS’ office in Germany, and the actual work unit in Romania. While everyone did their best, it was a frustrating and time consuming process that left project staff feeling that they did not have sufficient control over the schedule and workflow. It is anticipated that, even with the steep learning curve the software will require, bringing the image processing and metadata creation in-house will address these concerns.

An assessment of project strengths would include: 1) the establishment of a strong, collaborative working relationship with the University of Washington, 2) the creation of an adaptable workflow for largescale newspaper digitization, and 3) the start of a plan for implementing steps to carry out newspaper digitization beyond our NDNP grant.

Some assessed weaknesses have been 1) often underestimating timelines due to workflow or shipping constraints, 2) mishaps in communication between the vendors and project staff that have caused delays, and 3) not utilizing local libraries and project partners as much as desired when planning outreach and promotion to local researchers and patrons. This will be a priority in the second round of the project.

Continuation and Long Term Impact

WSL has received another two-year NDNP grant to continue the work started in 2008. Staff has also begun to plan and put infrastructure and equipment in place to integrate the output of the NDNP grants with existing digital newspaper content created at WSL. WSL has begun brainstorming ways to cover more areas of the state, especially less populated areas that will likely not be covered in the NDNP grant. Because of the popularity of the project and the need to cover more areas of the state, staff has planned to license the workflow software docWorks and bring the NDNP text conversion and derivative creation in house. The hope is to build up staff expertise and expand newspaper digitization to other areas of the state and plan for continuation after NDNP.

This project has strengthened our collaboration with the University of Washington in preservation and access of the state’s newspapers. It has also helped us initiate a move to digitize our assets in standard formats and prioritize the use of standard data formats in seeing the first hand benefits of sharing a research repository with other states. We hope to further our collaborative partnerships in the area of outreach and use of the NDNP collections in the next grant round.

Grant Products

Nearly all the output created during the grant period is available for public use and research. Products of the NDNP grant include approximately 450,000 digital files (100,000 pages scanned as tifs, two forms of image derivatives per page, one xml OCR text derivative per page, and one xml METS file per issue and reel to package the files), over 100,000 newspaper pages searchable online, essays for each title, updated bibliographic MARC records, press releases for each title, a project wiki that includes information and links to research about newspapers, and scripts and workflow processes to make post digitization QA more efficient.

Washington 2008-2010 Title List

Title	City	Dates	Reels	Pages
Cayton's Weekly	Seattle, WA	1917-1921	1	738
Colfax Gazette	Colfax, WA	1900-1912	8	4814
Colville Examiner	Colville, WA	1907-1922	5	6422
Commonwealth	Everett, WA	1911-1914	1	322
Industrial Freedom	Edison, WA	1898-1901	1	250
Leavenworth Echo	Leavenworth, WA	1904-1922	6	5596
Pullman Herald	Pullman, WA	1888-1922	11	13650
Ranch	Yakima, WA	1894-1912	8	10090
San Juan Islander	Friday Harbor, WA	1894-1914	11	6818
Seattle Republican	Seattle, WA	1896-1902	5	4668
Seattle Star	Seattle, WA	1899-1910	22	24325
Tacoma Times	Tacoma, WA	1894-1922	24	25023
Washington Socialist	Everett, WA	1914-1915	1	601
Yakima Herald	Yakima, WA	1893-1912	15	8539
Totals 119 111856

Other products include the following presentations, wiki, posts and articles:
  1. Blog posts: http://blogs.secstate.wa.gov/library/?tag=NDNP
  2. Wiki: http://wiki.secstate.wa.gov/ndnp/
  3. Presentations:
    1. Aug 2010 - Washington and Pacific Northwest Library Association, Victoria, BC
    2. Aug 2010 - NDNP Annual Meeting, Washington, DC
    3. Nov 2009 - Museum Computer Network, Portland, OR
    4. Oct 2009 - Washington Library Media Association, Yakima, WA
    5. Oct 2009 - LITA National Forum, Salt Lake City, UT
    6. Aug 2009 - NDNP Annual Meeting, Washington, DC
    7. April 2009 - Washington Library Association, Spokane, WA
    8. Mar 2009 - Washington State Council on Social Studies, Chelan, WA
  4. Article: Mar. 2010 - Microform & Imaging Review article: The Evolution of Newspaper Digitization at the Washington State Library

Template

COVER PAGE

Provide the following information in the order requested:


NARRATIVE DESCRIPTION

The items listed are provided as guidance to the project director in developing the narrative description of project activities. Because projects vary considerably, not all items will be relevant to a particular project. Please feel free to organize this portion of the report in the way that most clearly presents what has taken place during the grant period.

Final Performance Report

Using the project description and plan of work that were approved by NEH as a point of departure, the final performance report should address the following subjects:

1. Project Activities
2. Accomplishments
3. Audiences
4. Evaluation
5. Continuation of the Project
6. Long Term Impact
7. Grant Products
Normally, the information that is to be included in a final narrative description can adequately be covered in a report that does not exceed ten typewritten pages.

APPENDICES

Enclose with the report one copy of any supporting material that would contribute to an understanding of the project and its accomplishments to date. This would include:


It is not necessary to append work in progress, such as draft chapters of a book or other manuscript materials. However, unless otherwise specified in the conditions of the grant award, two copies of any publication, film, videotape, or slide presentation resulting from the grant should be forwarded to the Endowment with the final report.

Top

Resources


Top