> DataConversion

• FrontPage
• RecentChanges
• FindPage
• HelpContents

• Edit
• View
• Diffs
• Info
• Subscribe
• Raw
• Print

• AttachFile
• DSP2XML
• DeletePage
• LikePages
• LocalSiteMap
• SpellCheck

LOC Metadata

Using MARC data from: http://www.archive.org/details/marc_records_scriblio_net

For details of the RDF conversion work done by Alistair Miles in 2008/09, see http://code.google.com/p/code4rda/wiki/MilestoneOne

Statistics

MODS XML

Below are statistics on the MODS XML representation of the LOC dataset.

MARC XML

Below are statistics on the MARC XML representation of the LOC dataset. (These stats would be better if you could see the count of subfield codes in the context of specific datafield tags.)

Sample Record (Part 29, Record 1)

Record 1 from part 29.

MARC XML

Generated by yaz-marcdump:

<record xmlns="http://www.loc.gov/MARC21/slim">
  <leader>00519cam a22002171 4500</leader>
  <controlfield tag="001">sa 64009056 </controlfield>
  <controlfield tag="003">DLC</controlfield>
  <controlfield tag="005">20050711190518.0</controlfield>
  <controlfield tag="008">941008s1962 ii b 000 0 orio </controlfield>
  <datafield tag="010" ind1=" " ind2=" ">
    <subfield code="a">sa 64009056 </subfield>
  </datafield>
  <datafield tag="025" ind1=" " ind2=" ">
    <subfield code="a">PL480:I-O-205</subfield>
  </datafield>
  <datafield tag="035" ind1=" " ind2=" ">
    <subfield code="a">(OCoLC)31249133</subfield>
  </datafield>
  <datafield tag="040" ind1=" " ind2=" ">
    <subfield code="a">DLC</subfield>
    <subfield code="c">CU</subfield>
    <subfield code="d">CU</subfield>
    <subfield code="d">DLC</subfield>
  </datafield>
  <datafield tag="042" ind1=" " ind2=" ">
    <subfield code="a">premarc</subfield>
  </datafield>
  <datafield tag="050" ind1="0" ind2="0">
    <subfield code="a">PK2579.R255</subfield>
    <subfield code="b">K3</subfield>
  </datafield>
  <datafield tag="100" ind1="1" ind2=" ">
    <subfield code="a">Rautroy, Sachi,</subfield>
    <subfield code="d">1916-</subfield>
  </datafield>
  <datafield tag="245" ind1="1" ind2="0">
    <subfield code="a">Kabita&#772;.</subfield>
  </datafield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">[1962]</subfield>
  </datafield>
  <datafield tag="300" ind1=" " ind2=" ">
    <subfield code="a">8, 304 p.</subfield>
    <subfield code="c">25 cm.</subfield>
  </datafield>
  <datafield tag="500" ind1=" " ind2=" ">
    <subfield code="a">In Oriya.</subfield>
  </datafield>
  <datafield tag="504" ind1=" " ind2=" ">
    <subfield code="a">Bibliographical footnotes.</subfield>
  </datafield>
</record>

MODS XML

Generated from MARC XML by MARC21slim2MODS3.xsl:

<?xml version="1.0" encoding="UTF-8"?>
<mods xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.loc.gov/mods/v3" version="3.3" xsi:schemaLocation="http://www.loc.gov/mods/v3 http://www.loc.gov/standards/mods/v3/mods-3-3.xsd">
  <titleInfo>
    <title>Kabita&#772;</title>
  </titleInfo>
  <name type="personal">
    <namePart>Rautroy, Sachi</namePart>
    <namePart type="date">1916-</namePart>
    <role>
      <roleTerm authority="marcrelator" type="text">creator</roleTerm>
    </role>
  </name>
  <typeOfResource>text</typeOfResource>
  <genre authority="marcgt">bibliography</genre>
  <originInfo>
    <place>
      <placeTerm type="code" authority="marccountry">ii</placeTerm>
    </place>
    <dateIssued>[1962]</dateIssued>
    <dateIssued encoding="marc">1962</dateIssued>
    <issuance>monographic</issuance>
  </originInfo>
  <language>
    <languageTerm authority="iso639-2b" type="code">ori</languageTerm>
  </language>
  <physicalDescription>
    <form authority="marcform">print</form>
    <extent>8, 304 p. 25 cm.</extent>
  </physicalDescription>
  <note>In Oriya.</note>
  <note>Bibliographical footnotes.</note>
  <classification authority="lcc">PK2579.R255 K3</classification>
  <identifier type="lccn">sa 64009056</identifier>
  <recordInfo>
    <recordContentSource authority="marcorg">DLC</recordContentSource>
    <recordCreationDate encoding="marc">941008</recordCreationDate>
    <recordChangeDate encoding="iso8601">20050711190518.0</recordChangeDate>
    <recordIdentifier source="DLC">sa 64009056 </recordIdentifier>
  </recordInfo>
</mods>

MODS RDF

Generated from MODS XML by Simile marcmods2rdf/stylesheets/mods2rdf.xslt then tidied up by hand because the stylesheet expects a modsCollection root element:

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:modsrdf="http://simile.mit.edu/2006/01/ontologies/mods3#"
  xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:dcterms="http://purl.org/dc/terms/" xmlns:role="http://simile.mit.edu/2006/01/roles#">

  <modsrdf:Record rdf:about="http://libraries.mit.edu/barton/DLC/sa_64009056_">
    <modsrdf:records rdf:resource="info:lccn/sa_64009056"/>
    <modsrdf:origin rdf:resource="info:marcorg/DLC"/>
    <modsrdf:created>
      <modsrdf:Date modsrdf:value="941008" modsrdf:encoding="marc"/>
    </modsrdf:created>
    <modsrdf:changed>
      <modsrdf:Date modsrdf:value="20050711190518.0" modsrdf:encoding="iso8601"/>
    </modsrdf:changed>
  </modsrdf:Record>
  <rdf:Description rdf:about="info:lccn/sa 64009056">
    <rdf:type rdf:resource="http://simile.mit.edu/2006/01/ontologies/mods3#Text"/>
    <modsrdf:title>
      <rdf:Description>
        <rdf:type rdf:resource="http://simile.mit.edu/2006/01/ontologies/mods3#Title"/>
        <modsrdf:value>Kabita&#772;</modsrdf:value>
      </rdf:Description>
    </modsrdf:title>
    <role:creator rdf:resource="http://simile.mit.edu/2006/01/Entity#Rautroy_Sachi_1916"/>
    <modsrdf:genre>
      <modsrdf:Genre rdf:about="http://simile.mit.edu/2006/01/genre/marcgt/bibliography"
        modsrdf:authority="marcgt" modsrdf:value="bibliography"/>
    </modsrdf:genre>

    <modsrdf:dateIssued>
      <modsrdf:Date modsrdf:value="[1962]"/>
    </modsrdf:dateIssued>
    <modsrdf:dateIssued>
      <modsrdf:Date modsrdf:value="1962" modsrdf:encoding="marc"/>
    </modsrdf:dateIssued>
    <modsrdf:issuance>monographic</modsrdf:issuance>

    <modsrdf:language>
      <modsrdf:Language rdf:about="http://simile.mit.edu/2006/01/language/iso639-2b/ori"
        modsrdf:authority="iso639-2b" modsrdf:value="ori"/>
    </modsrdf:language>
    <modsrdf:physicalDescription>
      <modsrdf:Description>
        <modsrdf:form>
          <modsrdf:Form rdf:about="http://simile.mit.edu/2006/01/form/marcform/print"
            modsrdf:value="print" modsrdf:authority="marcform"/>
        </modsrdf:form>
        <modsrdf:extent>8, 304 p. 25 cm.</modsrdf:extent>
      </modsrdf:Description>
    </modsrdf:physicalDescription>
    <modsrdf:note>In Oriya.</modsrdf:note>
    <modsrdf:note>Bibliographical footnotes.</modsrdf:note>
    <modsrdf:classification>
      <modsrdf:Classification modsrdf:value="PK2579.R255 K3" modsrdf:authority="lcc"/>
    </modsrdf:classification>
  </rdf:Description>
  <rdf:Description rdf:about="http://simile.mit.edu/2006/01/Entity#Rautroy_Sachi_1916">
    <rdf:type rdf:resource="http://simile.mit.edu/2006/01/ontologies/mods3#Person"/>
    <modsrdf:fullName>Rautroy, Sachi</modsrdf:fullName>
    <modsrdf:dates>1916-</modsrdf:dates>

  </rdf:Description>
</rdf:RDF>

Downloads

Downloads of this dataset, MODS XML and MARC XML representations..

• http://dcmi-rda.s3.amazonaws.com/locdata/part01-marcxml.tar.gz

• http://dcmi-rda.s3.amazonaws.com/locdata/part01-modsxml.tar.gz

• http://dcmi-rda.s3.amazonaws.com/locdata/part02-marcxml.tar.gz

• http://dcmi-rda.s3.amazonaws.com/locdata/part02-modsxml.tar.gz

• [...]

• http://dcmi-rda.s3.amazonaws.com/locdata/part29-marcxml.tar.gz

• http://dcmi-rda.s3.amazonaws.com/locdata/part29-modsxml.tar.gz

Each download is a gzipped tar containing a *set* of up to 25 xml files. Each of these files is a 10,000 record split of the data in the corresponding part. I broke each part into 10,000 record splits so I could process the transformations more easily.

N.B. there is a bug in part 13 split 25, for some reason the marc xml output was incomplete so up to 10,000 records could be missing.

Processing Notes

The statistics were generated using Hadoop 0.17.2 running in pseudo-distributed mode on a single EC2 c1.xlarge instance. The job took 50 minutes to complete.

Assuming hadoop binaries are on the path...

$ hadoop jar /usr/local/hadoop/contrib/streaming/hadoop-0.17.2.1-streaming.jar -input input -output output -mapper mapper7.py -reducer aggregate -file mapper7.py

Contents of mapper7.py ...

  1 
  2 
  3 
  4 
  5 
  6 
  7 
  8 
  9 
 10 
 11 
 12 
 13 
 14 
 15 
 16 
 17 
 18 
 19 
 20 
 21 
 22 
 23 
 24 
 25 
 26 
 27 
 28 
 29 
 30 
 31 
 32 
 33 
 34 
 35 
 36 
 37 
 38 
 39 
 40 
 41 
 42 
 43 
 44 
 45 
 46 
 47 
 48 
 49 
 50 
 51 
 52 
#!/usr/bin/python

import sys
import xml.sax
import string

def printElementPathCount(stack):
    print "LongValueSum:" + string.join(stack, "/") + "\t" + "1"

def printAttrPathCount(stack, attr):
    print "LongValueSum:" + string.join(stack, "/") + "@" + attr + "\t" + "1"

def printAttrValPathCount(stack, attr, val):
    print "LongValueSum:" + string.join(stack, "/") + "@" + attr + "='" + val + "'\t" + "1"

class ModsHandler(xml.sax.ContentHandler):

    def __init__(self):
        self.stack = []

    def startElement(self, name, attrs):
        self.stack.append(name)
        printElementPathCount(self.stack)
        for (k,v) in attrs.items():
            printAttrPathCount(self.stack, k)
            if (k == u"type" or k == u"encoding" or k == u"authority"):
                printAttrValPathCount(self.stack, k, v)

    def endElement(self, name):
        self.stack.pop()

def main(argv):
    parser = xml.sax.make_parser(["xml.sax.xmlreader.IncrementalParser"])
    parser.setContentHandler(ModsHandler())
    lines = []
    line = sys.stdin.readline()
    try:
        while line:
            lines.append(line)
            line = sys.stdin.readline()
    except "end of file":
        print "end of file hit"
        return None
    except:
        print "Unexpected error attempting to parse line: "+line+"\n", sys.exc_info()[0]
        return None
    for line in lines:
        parser.feed(line)
    parser.close()

if __name__ == "__main__":
    main(sys.argv)

N.B. mapper7.py caches lines for each file before feeding them to the SAX parser. I tried a direct feed of stdin to the SAX parser, but got broken pipe errors when running hadoop in distributed mode (standalone works fine).

Contents of hadoop-site.xml ...

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>localhost:9000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>mapred.tasktracker.map.tasks.maximum</name>
    <value>8</value>
  </property>
  <property>
    <name>mapred.tasktracker.reduce.tasks.maximum</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>/mnt/tmp/hadoop/data</value>
  </property>
</configuration>

Immutable page (last edited 2009-10-13 06:10:01 by AlistairMiles)