存储带有POS标签的语料库

Thagor 发表于 Dev

塔戈尔

我使用NLTK并使用POS标记德国维基百科。结构非常简单，一个大列表包含每个句子作为单词列表，POS标记元组示例：

[[(Word1,POS),(Word2,POS),...],[(Word1,POS),(Word2,POS),...],...]

因为维基百科很大，所以我显然无法将整个大列表存储在内存中，因此我需要一种将部分列表保存到磁盘的方法。这样做的一种好方法是什么，以便以后我可以轻松地从磁盘上遍历所有句子和单词？

睡觉

使用pickle，请参见https://wiki.python.org/moin/UsingPickle：

import io
import cPickle as pickle

from nltk import pos_tag
from nltk.corpus import brown

print brown.sents()
print 

# Let's tag the first 10 sentences.
tagged_corpus = [pos_tag(i) for i in brown.sents()[:10]]

with io.open('brown.pos', 'wb') as fout:
    pickle.dump(tagged_corpus, fout)

with io.open('brown.pos', 'rb') as fin:
    loaded_corpus = pickle.load(fin)

for sent in loaded_corpus:
    print sent
    break

[出去]：

[[u'The', u'Fulton', u'County', u'Grand', u'Jury', u'said', u'Friday', u'an', u'investigation', u'of', u"Atlanta's", u'recent', u'primary', u'election', u'produced', u'``', u'no', u'evidence', u"''", u'that', u'any', u'irregularities', u'took', u'place', u'.'], [u'The', u'jury', u'further', u'said', u'in', u'term-end', u'presentments', u'that', u'the', u'City', u'Executive', u'Committee', u',', u'which', u'had', u'over-all', u'charge', u'of', u'the', u'election', u',', u'``', u'deserves', u'the', u'praise', u'and', u'thanks', u'of', u'the', u'City', u'of', u'Atlanta', u"''", u'for', u'the', u'manner', u'in', u'which', u'the', u'election', u'was', u'conducted', u'.'], ...]

[(u'The', 'DT'), (u'Fulton', 'NNP'), (u'County', 'NNP'), (u'Grand', 'NNP'), (u'Jury', 'NNP'), (u'said', 'VBD'), (u'Friday', 'NNP'), (u'an', 'DT'), (u'investigation', 'NN'), (u'of', 'IN'), (u"Atlanta's", 'JJ'), (u'recent', 'JJ'), (u'primary', 'JJ'), (u'election', 'NN'), (u'produced', 'VBN'), (u'``', '``'), (u'no', 'DT'), (u'evidence', 'NN'), (u"''", "''"), (u'that', 'WDT'), (u'any', 'DT'), (u'irregularities', 'NNS'), (u'took', 'VBD'), (u'place', 'NN'), (u'.', '.')]

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。