SimString

Base type for all custom db collections.

Abstract base type for all string similarity measures.

Feature extraction on character-level ngrams

Cosine Similarity Measure.

Dice Similarity Measure.

Custom DB collection for storing SimString data using base Dictionary Dict

DictDB(x::CharacterNGrams)

Initialize a dict DB with additional containers and Metadata for CharacterNGrams

Arguments

• x: CharacterNGrams object

Example

db = DictDB(CharacterNGrams(2, " "))

Returns

• DictDB: A DictDB object with additional containers and Metadata for CharacterNGrams

DictDB(x::MecabNGrams)

Initialize a dict DB with additional containers and Metadata for MecabNGrams

Arguments

• x: MecabNGrams object

Example

db = DictDB(MecabNGrams(2, " ", Mecab()))

Returns

• DictDB: A DictDB object with additional containers and Metadata for MecabNGrams

DictDB(x::WordNGrams)

Initialize a dict DB with additional containers and Metadata for WordNGrams

Arguments

• x: WordNGrams object

Example

db = DictDB(WordNGrams(2, " ", " "))

Returns

• DictDB: A DictDB object with additional containers and Metadata for WordNGrams

Exact Match Similarity Measure.

Abstract type for feature extraction structs

Jaccard Similarity Measure.

Feature extraction based on mecab word-level ngrams

Overlap Similarity Measure.

Feature extraction based on word-level ngrams

append!(db::AbstractSimStringDB, file::AbstractString)

Add bulk items to a new or existing collection of strings using from a file using the custom AbstractSimStringDB type.

Arguments:

• db`: AbstractSimStringDB - The database to add the items to
• file: AbstractString - Path to the file to read from

Example:

db = DictDB(CharacterNGrams(2, " "));
append!(db, "./data/test.txt")

Returns:

• db: AbstractSimStringDB - The database with the items added

append!(db::AbstractSimStringDB, str::Vector)

Add bulk items to a new or existing collection of strings using the custom AbstractSimStringDB type.

Arguments:

• db: AbstractSimStringDB - The database to add the strings to
• str: Vector of AbstractString - Vector/Array of strings to add to the database

Example:

db = DictDB(CharacterNGrams(2, " "));
append!(db, ["foo", "foo", "fooo"]);

Returns:

• db: AbstractSimStringDB - The database with the new strings added

push!(db::AbstractSimStringDB, str::AbstractString)

Add a new item to a new or existing collection of strings using the custom AbstractSimStringDB type.

Arguments:

• db: AbstractSimStringDB - The collection of strings to add to
• str: AbstractString - The string to add to the collection

Example:

julia db = DictDB(CharacterNGrams(2, " ")); push!(db, "foo") push!(db, "bar") push!(db, "fooo")`

Returns:

• db: AbstractSimStringDB - The collection of strings with the new string added

Pretty print summary stats for the DB

Internal function to count and pad generated character-level ngrams (including duplicates)

describe_collection(db::DictDB)

Basic summary stats for the DB

Arguments

• db: DictDB object

Example

db = DictDB(CharacterNGrams(2, " "));
append!(db, ["foo", "bar", "fooo"]);
describe_collection(db)
(total_collection = 3, avg_size_ngrams = 4.5, total_ngrams = 13)

# Returns
* NamedTuples: Summary stats for the DB

Internal function to generate character-level ngrams features from an AbstractString

Internal function to generate Mecab word-level ngrams features from an AbstractString

Internal function to generate word-level ngrams features from an AbstractString

Internal function for generating a base DictDB object for WordNGrams and MecabNGrams

Internal function to generate intial uncounted word ngrams on a word level

Internal function to generate intial uncounted ngrams on a character level

Internal function to lookup feature sets by size and feature

Internal function to make zero indexed circular arrays

Calculate maximum feature size for Cosine similarity measure.

Calculate maximum feature size for Dice similarity measure.

Calculate maximum feature size for ExactMatch similarity measure.

Calculate maximum feature size for Jaccard similarity measure.

Calculate maximum feature size for Overlap similarity measure.

Calculate minimum feature size for Cosine similarity measure.

Calculate minimum feature size for Dice similarity measure.

Calculate minimum feature size for ExactMatch similarity measure.

Calculate minimum feature size for Jaccard similarity measure.

Calculate minimum feature size for Overlap similarity measure.

Calculate the minimum overlap (τ) for a query size, candidate size, and α using Cosine similarity measure.

Calculate the minimum overlap (τ) for a query size, candidate size, and α using Dice similarity measure.

Calculate the minimum overlap (τ) for a query size, candidate size, and α using ExactMatch similarity measure.

Calculate the minimum overlap (τ) for a query size, candidate size, and α using Jaccard similarity measure.

Calculate the minimum overlap (τ) for a query size, candidate size, and α using Overlap similarity measure.

Internal function to create counted ngrams

Internal function which performs the overlap join

Internal function to pad AbstractString types with specified padder

Internal function to pad AbstractVector types with specified padder

Internal function which ranks the results of a search using the specified similarity measure.

Search for strings in custom DictDB string collection using the SimString algorithm and a similarity measure.

search(measure::AbstractSimilarityMeasure, db_collection::AbstractSimStringDB, query::AbstractString;
    α=0.7, ranked=true)

Search for strings in a string collection using the SimString algorithm and a similarity measure.

Arguments:

• measure::AbstractSimilarityMeasure - The similarity measure to use.
• db_collection::AbstractSimStringDB - The database collection to search.
• query::AbstractString - The query string to search for.
• α::float - The α parameter for the SimString algorithm.
• ranked::Boolean - Whether to return the results in ranked order.

Example

db = DictDB(CharacterNGrams(2, " "));
append!(db, ["foo", "bar", "fooo"]);

search(Dice(), db, "foo"; α=0.8, ranked=true)
# 2-element Vector{Tuple{String, Float64}}:
#  ("foo", 1.0)
#  ("fooo", 0.8888888888888888)

Returns

• A Vector of results, where each element is a Tuple of the form (string, similarity measure score).

Calculate similarity score between X and Y using Cosine similarity measure.

Calculate similarity score between X and Y using Dice similarity measure.

Calculate similarity score between X and Y using ExactMatch similarity measure.

Calculate similarity score between X and Y using Jaccard similarity measure.

Calculate similarity score between X and Y using Overlap similarity measure.

Internal function to tokenize a string using Mecab