I’ll create the getting_started.md file for the tutorials section:
docs/tutorials/getting_started.md
# Getting Started with Qubee NLP
This tutorial will guide you through installing and using the Qubee NLP library for Afaan Oromoo text processing.
## Prerequisites
Before you begin, make sure you have:
- **Python 3.7 or higher** installed
- **Basic knowledge of Python** programming
- **Familiarity with Afaan Oromoo** (Oromo language) is helpful but not required
- **Text editor or IDE** (VS Code, PyCharm, or any Python IDE)
## Installation
### Method 1: Install from PyPI (Recommended)
```bash
pip install qubee-nlp
Method 2: Install from Source
# Clone the repository
git clone https://github.com/yourusername/qubee-nlp.git
cd qubee-nlp
# Install in development mode
pip install -e .
# Install development dependencies (optional)
pip install -e .[dev]
Verify Installation
import qubee_nlp
print(f"Qubee NLP version: {qubee_nlp.__version__}")
Understanding Qubee Script
Qubee is the Latin-based alphabet used for writing Afaan Oromoo. Here are the basics:
from qubee_nlp.alphabet import QubeeAlphabet
# Qubee has 5 vowels
print("Vowels:", QubeeAlphabet.VOWELS)
# Output: {'A', 'E', 'I', 'O', 'U'}
# And 21 consonants
print("Number of consonants:", len(QubeeAlphabet.CONSONANTS))
# Output: 21
# Diacritics are sometimes used for tone marking (optional)
print("Diacritic mapping (first 3):", dict(list(QubeeAlphabet.DIACRITICS.items())[:3]))
# Output: {'Á': 'A', 'É': 'E', 'Í': 'I'}
Qubee Alphabet Overview
Category |
Letters |
Description |
|---|---|---|
Vowels |
A, E, I, O, U |
Basic vowels (with optional diacritics: á, é, í, ó, ú) |
Consonants |
B, C, D, F, G, H, J, K, L, M, N, P, Q, R, S, T, V, W, X, Y, Z |
21 consonants |
Digraphs |
CH, DH, NY, PH, SH |
Common consonant combinations |
Special |
‘ (apostrophe), - (hyphen) |
Used in words like “waa’ee” (about) |
Your First Qubee NLP Program
Create a file first_program.py:
#!/usr/bin/env python3
"""First program with Qubee NLP."""
from qubee_nlp import word_tokenize, sentence_tokenize
def main():
# Sample Afaan Oromoo text
text = "Afaan Oromoo afaan jalqaba Oromiyaati. Waa'ee isaa dubbachuun barbaachisaa dha."
# Tokenize into words
words = word_tokenize(text)
print(f"Words: {words}")
# Tokenize into sentences
sentences = sentence_tokenize(text)
print(f"\nSentences: {sentences}")
# Basic statistics
print(f"\nStatistics:")
print(f" Text length: {len(text)} characters")
print(f" Number of words: {len(words)}")
print(f" Number of sentences: {len(sentences)}")
print(f" Average words per sentence: {len(words)/len(sentences):.1f}")
if __name__ == "__main__":
main()
Run it:
python first_program.py
Expected Output:
Words: ['AFAAN', 'OROMOO', 'AFAAN', 'JALQABA', 'OROMIYAATI', 'WAA\'EE', 'ISAA', 'DUBBACHUUN', 'BARBAACHISAA', 'DHA']
Sentences: ['AFAAN OROMOO AFAAN JALQABA OROMIYAATI.', 'WAA\'EE ISAA DUBBACHUUN BARBAACHISAA DHA.']
Statistics:
Text length: 70 characters
Number of words: 10
Number of sentences: 2
Average words per sentence: 5.0
Core Concepts
1. Text Validation
Always validate your text before processing:
from qubee_nlp import validate_qubee_text
# Valid Afaan Oromoo text
text1 = "Afaan Oromoo afaan guddaa dha."
is_valid1, invalid1 = validate_qubee_text(text1)
print(f"Text 1 valid: {is_valid1}, Invalid chars: {invalid1}")
# Output: Text 1 valid: True, Invalid chars: []
# Text with invalid characters
text2 = "Afaan Oromoo 123 test!"
is_valid2, invalid2 = validate_qubee_text(text2)
print(f"Text 2 valid: {is_valid2}, Invalid chars: {invalid2}")
# Output: Text 2 valid: False, Invalid chars: ['1', '2', '3', '!']
2. Text Normalization
Normalize text to consistent format:
from qubee_nlp import normalize_qubee
# Text with diacritics and extra spaces
text = " Áfáan Oromoo gúddáa dha. "
normalized = normalize_qubee(text)
print(f"Normalized: '{normalized}'")
# Output: Normalized: 'AFAAN OROMOO GUDDAA DHA.'
# Preserve case
normalized_preserve = normalize_qubee("Áfáan Oromoo", preserve_case=True)
print(f"Preserved case: '{normalized_preserve}'")
# Output: Preserved case: 'Afaan Oromoo'
3. Tokenization
Tokenize text into words and sentences:
from qubee_nlp import QubeeTokenizer
# Create tokenizer with options
tokenizer = QubeeTokenizer(
preserve_case=True, # Keep original case
strict=False # Use lenient validation
)
text = "Afaan Oromoo afaan jalqaba Oromiyaati."
tokens = tokenizer.tokenize(text)
print(f"Tokens: {tokens}")
# Output: Tokens: ['Afaan', 'Oromoo', 'afaan', 'jalqaba', 'Oromiyaati']
sentences = tokenizer.sentence_tokenize(text + " Kunis dubbii dha.")
print(f"Sentences: {sentences}")
# Output: Sentences: ['Afaan Oromoo afaan jalqaba Oromiyaati.', 'Kunis dubbii dha.']
4. Character Analysis
Analyze text at character level:
from qubee_nlp.alphabet import QubeeAlphabet
def analyze_characters(text):
"""Analyze character distribution in text."""
vowels = consonants = others = 0
vowel_list = []
consonant_list = []
for char in text:
if char.isspace():
continue
if QubeeAlphabet.is_vowel(char):
vowels += 1
if char.upper() not in vowel_list:
vowel_list.append(char.upper())
elif QubeeAlphabet.is_consonant(char):
consonants += 1
if char.upper() not in consonant_list:
consonant_list.append(char.upper())
else:
others += 1
total = vowels + consonants + others
return {
'vowels': vowels,
'consonants': consonants,
'others': others,
'total': total,
'vowel_percent': (vowels / total * 100) if total > 0 else 0,
'consonant_percent': (consonants / total * 100) if total > 0 else 0,
'vowel_list': sorted(vowel_list),
'consonant_list': sorted(consonant_list)
}
text = "Afaan Oromoo"
stats = analyze_characters(text)
print(f"Vowels: {stats['vowels']} ({stats['vowel_percent']:.1f}%)")
print(f"Consonants: {stats['consonants']} ({stats['consonant_percent']:.1f}%)")
print(f"Unique vowels: {stats['vowel_list']}")
print(f"Unique consonants: {stats['consonant_list']}")
Step-by-Step Tutorial
Step 1: Setting Up Your Project
Create a new directory for your project:
mkdir my_qubee_project
cd my_qubee_project
Create a virtual environment:
python -m venv venv
Activate the virtual environment:
Windows:
venv\Scripts\activate
macOS/Linux:
source venv/bin/activate
Install Qubee NLP:
pip install qubee-nlp
Step 2: Create a Simple Text Analyzer
Create analyzer.py:
#!/usr/bin/env python3
"""Simple Afaan Oromoo text analyzer."""
from qubee_nlp import (
word_tokenize,
sentence_tokenize,
validate_qubee_text,
normalize_qubee
)
from qubee_nlp.alphabet import QubeeAlphabet
class SimpleTextAnalyzer:
def __init__(self):
self.results = {}
def analyze(self, text):
"""Analyze Afaan Oromoo text."""
# Store original text
self.results['original'] = text
# Validate
is_valid, invalid_chars = validate_qubee_text(text)
self.results['is_valid'] = is_valid
self.results['invalid_chars'] = invalid_chars
if not is_valid:
return self.results
# Normalize
normalized = normalize_qubee(text, preserve_case=True)
self.results['normalized'] = normalized
# Tokenize
words = word_tokenize(normalized, preserve_case=True)
sentences = sentence_tokenize(normalized, preserve_case=True)
self.results['words'] = words
self.results['sentences'] = sentences
self.results['word_count'] = len(words)
self.results['sentence_count'] = len(sentences)
self.results['avg_words_per_sentence'] = len(words) / len(sentences) if sentences else 0
# Character analysis
self._analyze_characters(text)
return self.results
def _analyze_characters(self, text):
"""Analyze character distribution."""
vowels = consonants = 0
for char in text:
if char.isspace():
continue
if QubeeAlphabet.is_vowel(char):
vowels += 1
elif QubeeAlphabet.is_consonant(char):
consonants += 1
total_chars = vowels + consonants
self.results['character_stats'] = {
'vowels': vowels,
'consonants': consonants,
'total': total_chars,
'vowel_percent': (vowels / total_chars * 100) if total_chars > 0 else 0,
'consonant_percent': (consonants / total_chars * 100) if total_chars > 0 else 0
}
def print_report(self):
"""Print analysis report."""
print("=" * 60)
print("AFAN OROMOO TEXT ANALYSIS REPORT")
print("=" * 60)
print(f"\nOriginal Text: {self.results['original'][:50]}..."
if len(self.results['original']) > 50 else self.results['original'])
print(f"\nVALIDATION:")
print(f" Valid: {'✓ Yes' if self.results['is_valid'] else '✗ No'}")
if not self.results['is_valid']:
print(f" Invalid characters: {self.results['invalid_chars']}")
if self.results['is_valid']:
print(f"\nNORMALIZED TEXT: {self.results['normalized']}")
print(f"\nTOKENIZATION:")
print(f" Words: {self.results['word_count']}")
print(f" Sentences: {self.results['sentence_count']}")
print(f" Average words/sentence: {self.results['avg_words_per_sentence']:.1f}")
print(f"\nCHARACTER ANALYSIS:")
stats = self.results['character_stats']
print(f" Vowels: {stats['vowels']} ({stats['vowel_percent']:.1f}%)")
print(f" Consonants: {stats['consonants']} ({stats['consonant_percent']:.1f}%)")
print(f" Total characters: {stats['total']}")
print(f"\nSAMPLE WORDS (first 5):")
for word in self.results['words'][:5]:
print(f" - {word}")
# Example usage
if __name__ == "__main__":
analyzer = SimpleTextAnalyzer()
sample_texts = [
"Afaan Oromoo afaan guddaa dha.",
"Oromiyaan biyya guddaa Afriikaa keessatti argamti.",
"Qubee sirna barreeffama Afaan Oromoo dha.",
]
for i, text in enumerate(sample_texts, 1):
print(f"\n\n{'='*60}")
print(f"ANALYSIS {i}")
print(f"{'='*60}")
analyzer.analyze(text)
analyzer.print_report()
Run the analyzer:
python analyzer.py
Step 3: Working with Files
Create file_processor.py:
#!/usr/bin/env python3
"""Process Afaan Oromoo text files."""
import os
from pathlib import Path
from qubee_nlp import word_tokenize, sentence_tokenize
from qubee_nlp.alphabet import validate_qubee_text
from collections import Counter
class FileProcessor:
def __init__(self):
self.results = {}
def process_file(self, filepath):
"""Process a text file containing Afaan Oromoo text."""
if not os.path.exists(filepath):
print(f"Error: File not found: {filepath}")
return None
# Read the file
with open(filepath, 'r', encoding='utf-8') as f:
text = f.read()
print(f"\nProcessing file: {filepath}")
print(f"File size: {len(text):,} characters")
# Validate
is_valid, invalid_chars = validate_qubee_text(text)
if not is_valid:
print(f"Warning: Invalid characters found: {invalid_chars}")
# Tokenize
words = word_tokenize(text)
sentences = sentence_tokenize(text)
# Statistics
word_counts = Counter(words)
unique_words = set(words)
result = {
'filepath': filepath,
'characters': len(text),
'words': len(words),
'sentences': len(sentences),
'unique_words': len(unique_words),
'is_valid': is_valid,
'invalid_chars': invalid_chars,
'most_common_words': word_counts.most_common(10),
'lexical_diversity': len(unique_words) / len(words) if words else 0
}
self._print_summary(result)
return result
def _print_summary(self, result):
"""Print processing summary."""
print(f"\nSUMMARY:")
print(f" Characters: {result['characters']:,}")
print(f" Words: {result['words']:,}")
print(f" Sentences: {result['sentences']:,}")
print(f" Unique words: {result['unique_words']:,}")
print(f" Lexical diversity: {result['lexical_diversity']:.3f}")
print(f" Valid Qubee: {'✓ Yes' if result['is_valid'] else '✗ No'}")
if result['most_common_words']:
print(f"\nTOP 10 MOST COMMON WORDS:")
for word, count in result['most_common_words']:
percentage = (count / result['words']) * 100
print(f" {word:15} {count:5} ({percentage:.1f}%)")
def batch_process(self, directory):
"""Process all text files in a directory."""
directory = Path(directory)
if not directory.exists() or not directory.is_dir():
print(f"Error: Directory not found: {directory}")
return []
results = []
for filepath in directory.glob('*.txt'):
result = self.process_file(filepath)
if result:
results.append(result)
self._print_batch_summary(results)
return results
def _print_batch_summary(self, results):
"""Print batch processing summary."""
if not results:
return
print(f"\n{'='*60}")
print(f"BATCH PROCESSING SUMMARY")
print(f"{'='*60}")
total_files = len(results)
total_chars = sum(r['characters'] for r in results)
total_words = sum(r['words'] for r in results)
total_sentences = sum(r['sentences'] for r in results)
print(f"\nProcessed {total_files} files")
print(f"Total characters: {total_chars:,}")
print(f"Total words: {total_words:,}")
print(f"Total sentences: {total_sentences:,}")
print(f"Average words per file: {total_words/total_files:,.0f}")
# Example: Create and process a sample file
if __name__ == "__main__":
# Create sample directory
sample_dir = Path("sample_texts")
sample_dir.mkdir(exist_ok=True)
# Create sample text files
sample_texts = {
"sample1.txt": """Afaan Oromoo afaan Kushitikii kan dubbatamu Oromiyaa fi naannawa ishee keessatti dha.
Afaanichi afaan baayyinaan dubbatamu Afriikaa keessatti, Afrikaa Kibbaa fi Kaabaati.
Qubee sirna barreeffama Afaan Oromoo sirna Laatin irratti hundaa'e dha.""",
"sample2.txt": """Oromiyaan biyya guddaa Afriikaa keessatti argamti.
Biyyichi baayyina ummataa fi ballina lafaatiin Afriikaa keessatti lammaffaa dha.
Oromiyaan naannoo biyyoo adda addaa wajjin dhadhaabdi.""",
"sample3.txt": """Gadaa sirna haaraa hawaasaa Oromoo dha.
Sirnichii mooraa siyaasaa, dinagdee, hawaasaa fi aadaa of keessaa qaba.
Gadaa sirna demokraatawaa kan ture yeroo dheeraaf."""
}
# Write sample files
for filename, content in sample_texts.items():
filepath = sample_dir / filename
with open(filepath, 'w', encoding='utf-8') as f:
f.write(content)
print(f"Created: {filepath}")
# Process files
processor = FileProcessor()
print(f"\n{'='*60}")
print("PROCESSING FILES")
print(f"{'='*60}")
results = processor.batch_process(sample_dir)
Run the file processor:
python file_processor.py
Step 4: Interactive Exploration
Create interactive_explorer.py:
#!/usr/bin/env python3
"""Interactive exploration of Qubee NLP."""
def explore_text():
"""Interactive text exploration."""
print("=== Qubee NLP Interactive Explorer ===\n")
print("Type Afaan Oromoo text to analyze it.")
print("Commands: 'quit', 'help', 'examples'\n")
while True:
user_input = input("Enter text or command: ").strip()
if user_input.lower() in ['quit', 'exit', 'q']:
print("Goodbye!")
break
if user_input.lower() == 'help':
print("\nAvailable commands:")
print(" quit, exit, q - Exit the program")
print(" help - Show this help message")
print(" examples - Show example texts")
print("\nOr enter Afaan Oromoo text to analyze it.")
continue
if user_input.lower() == 'examples':
print("\nExample texts:")
print("1. Afaan Oromoo afaan guddaa dha.")
print("2. Oromiyaan biyya guddaa Afriikaa keessatti argamti.")
print("3. Qubee sirna barreeffama Afaan Oromoo dha.")
continue
if not user_input:
continue
# Process the text
from qubee_nlp import (
validate_qubee_text,
normalize_qubee,
word_tokenize,
sentence_tokenize
)
from qubee_nlp.alphabet import QubeeAlphabet, split_into_syllables
print(f"\nAnalyzing: '{user_input}'")
print("-" * 40)
# Validate
is_valid, invalid = validate_qubee_text(user_input)
if not is_valid:
print(f"✗ Invalid characters: {invalid}")
continue
print("✓ Text is valid Qubee")
# Normalize
normalized = normalize_qubee(user_input, preserve_case=True)
print(f"✓ Normalized: '{normalized}'")
# Tokenize
words = word_tokenize(user_input, preserve_case=True)
sentences = sentence_tokenize(user_input, preserve_case=True)
print(f"✓ Words ({len(words)}): {words}")
print(f"✓ Sentences ({len(sentences)}):")
for i, sentence in enumerate(sentences, 1):
print(f" {i}. {sentence}")
# Character analysis
print("\nCharacter Analysis:")
char_counts = {}
for char in user_input.lower():
if char.isalpha():
char_counts[char] = char_counts.get(char, 0) + 1
for char, count in sorted(char_counts.items()):
is_vowel = QubeeAlphabet.is_vowel(char)
is_consonant = QubeeAlphabet.is_consonant(char)
char_type = "vowel" if is_vowel else "consonant" if is_consonant else "other"
print(f" '{char}': {count:2} times ({char_type})")
# Syllable analysis for first word
if words:
first_word = words[0]
syllables = split_into_syllables(first_word)
print(f"\nSyllable analysis for '{first_word}': {syllables}")
print() # Empty line for readability
if __name__ == "__main__":
explore_text()
Run the interactive explorer:
python interactive_explorer.py
Common Tasks
Task 1: Text Cleaning
from qubee_nlp import normalize_qubee
from qubee_nlp.alphabet import validate_qubee_text
def clean_text(text):
"""Clean and normalize Afaan Oromoo text."""
# Validate
is_valid, invalid = validate_qubee_text(text)
if not is_valid:
# Remove invalid characters
for char in invalid:
text = text.replace(char, '')
# Normalize
cleaned = normalize_qubee(text)
return cleaned
text = " Áfáan Oromoo gúddáa dha. "
cleaned = clean_text(text)
print(f"Cleaned: '{cleaned}'")
Task 2: Word Frequency Analysis
from qubee_nlp import word_tokenize
from collections import Counter
def analyze_frequencies(text):
"""Analyze word frequencies in Afaan Oromoo text."""
tokens = word_tokenize(text)
freq = Counter(tokens)
print(f"Total words: {len(tokens)}")
print(f"Unique words: {len(freq)}")
print("\nMost frequent words:")
for word, count in freq.most_common(10):
percentage = (count / len(tokens)) * 100
print(f" {word}: {count} ({percentage:.1f}%)")
return freq
text = "Afaan Oromoo afaan guddaa dha. Oromiyaan biyya guddaa dha."
freq = analyze_frequencies(text)
Task 3: Text Comparison
from qubee_nlp import word_tokenize
def compare_texts(text1, text2):
"""Compare two Afaan Oromoo texts."""
tokens1 = set(word_tokenize(text1))
tokens2 = set(word_tokenize(text2))
common = tokens1.intersection(tokens2)
unique1 = tokens1 - tokens2
unique2 = tokens2 - tokens1
print(f"Text 1 unique words: {len(unique1)}")
print(f"Text 2 unique words: {len(unique2)}")
print(f"Common words: {len(common)}")
similarity = len(common) / (len(tokens1) + len(tokens2) - len(common))
print(f"Jaccard similarity: {similarity:.2f}")
return {
'common': sorted(common),
'unique1': sorted(unique1),
'unique2': sorted(unique2),
'similarity': similarity
}
text1 = "Afaan Oromoo afaan guddaa dha."
text2 = "Oromiyaan biyya guddaa dha."
compare_texts(text1, text2)
Troubleshooting
Common Issues
ImportError: No module named ‘qubee_nlp’
# Make sure you installed the package pip install qubee-nlp # Or if installed in development mode pip install -e .
UnicodeDecodeError when reading files
# Always specify encoding with open('file.txt', 'r', encoding='utf-8') as f: text = f.read()
Invalid characters error
# Use validate_qubee_text to check from qubee_nlp.alphabet import validate_qubee_text is_valid, invalid = validate_qubee_text(your_text) print(f"Invalid characters: {invalid}") # Clean the text for char in invalid: your_text = your_text.replace(char, '')
Performance issues with large texts
# Process in chunks chunk_size = 10000 # characters for i in range(0, len(text), chunk_size): chunk = text[i:i+chunk_size] # Process chunk
Next Steps
Now that you’ve completed this getting started guide:
Explore Advanced Features: Check out the other tutorials:
Afaan Oromoo NLP - Advanced language-specific processing
Read the API Documentation:
Try the Example Scripts:
Work with Real Data: Try processing real Afaan Oromoo texts from:
Oromo news websites
Wikipedia articles in Afaan Oromoo
Oromo literature
Social media posts in Afaan Oromoo
Contribute to the Project:
Report issues on GitHub
Suggest new features
Contribute code improvements
Getting Help
GitHub Issues: Report bugs or request features
Documentation: Check the API documentation for detailed information
Examples: Look at the examples directory for working code
Email: guutatesema@gmail.com
Happy coding with Afaan Oromoo NLP! 🌍📚
This `getting_started.md` file provides a comprehensive tutorial covering:
1. **Installation** - Multiple installation methods
2. **Basic Concepts** - Understanding Qubee script and alphabet
3. **Step-by-Step Tutorials** - From simple scripts to file processing
4. **Interactive Tools** - For hands-on exploration
5. **Common Tasks** - Practical examples for real-world use
6. **Troubleshooting** - Solutions to common problems
7. **Next Steps** - Guidance for further learning
The tutorial is designed to be beginner-friendly while covering all essential aspects of using Qubee NLP for Afaan Oromoo text processing.